DPDK usage discussions
 help / color / mirror / Atom feed
* Enable RX interrupts
@ 2021-11-01  5:50 Jack Humphries
  2021-11-01  7:46 ` Dmitry Kozlyuk
  2021-11-01 15:20 ` Stephen Hemminger
  0 siblings, 2 replies; 7+ messages in thread
From: Jack Humphries @ 2021-11-01  5:50 UTC (permalink / raw)
  To: users

Hi folks,

Hope all are well. I’m trying to enable interrupts in DPDK so that my network receive thread can sleep on an epoll until packets arrive. I am using the ixgbe kernel driver and igb_uio userspace driver with an Intel 82599ES 10Gbps NIC.

I'm doing roughly the following to enable the interrupts, but the epoll never indicates that packets have arrived. The thread only handles packets when the epoll times out. I don't even see interrupts arrive from the device when monitoring /proc/interrupts.

port_conf.intr_conf.rxq = 1;

...

CHECK_EQ(rte_eth_dev_rx_intr_ctl_q(kPort, kQueue, RTE_EPOLL_PER_THREAD,
                                   RTE_INTR_EVENT_ADD, nullptr),
         0); 
CHECK_EQ(rte_eth_dev_rx_intr_enable(kPort, kQueue), 0);

...

rte_epoll_event event;
while (true) {
  int n = rte_epoll_wait(RTE_EPOLL_PER_THREAD, &event, /*maxevents=*/1,
                         /*timeout=*/1000);
  if (n == 0) {
    // Timeout expired.
  } else {
    // Received RX interrupt.
  }
}

Given that I don't see anything coming through in /proc/interrupts, I am going to start digging through the ixgbe driver. However, I wanted to ask here first to see if my setup is missing anything obvious. I based it closely on the l3fwd-power example, though I haven't been able to get that example up and running yet since my NIC only has one port plugged in right now and the example requires two (COVID building restrictions).

Thanks,
Jack Humphries

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Enable RX interrupts
  2021-11-01  5:50 Enable RX interrupts Jack Humphries
@ 2021-11-01  7:46 ` Dmitry Kozlyuk
  2021-11-01  8:56   ` Jack Humphries
  2021-11-01 15:20 ` Stephen Hemminger
  1 sibling, 1 reply; 7+ messages in thread
From: Dmitry Kozlyuk @ 2021-11-01  7:46 UTC (permalink / raw)
  To: Jack Humphries; +Cc: users, Haiyue Wang

2021-10-31 22:50 (UTC-0700), Jack Humphries:
> Hi folks,
> 
> Hope all are well. I’m trying to enable interrupts in DPDK so that my network receive thread can sleep on an epoll until packets arrive. I am using the ixgbe kernel driver and igb_uio userspace driver with an Intel 82599ES 10Gbps NIC.
> 
> I'm doing roughly the following to enable the interrupts, but the epoll never indicates that packets have arrived. The thread only handles packets when the epoll times out. I don't even see interrupts arrive from the device when monitoring /proc/interrupts.
> 
> port_conf.intr_conf.rxq = 1;
> 
> ...
> 
> CHECK_EQ(rte_eth_dev_rx_intr_ctl_q(kPort, kQueue, RTE_EPOLL_PER_THREAD,
>                                    RTE_INTR_EVENT_ADD, nullptr),
>          0); 
> CHECK_EQ(rte_eth_dev_rx_intr_enable(kPort, kQueue), 0);
> 
> ...
> 
> rte_epoll_event event;
> while (true) {
>   int n = rte_epoll_wait(RTE_EPOLL_PER_THREAD, &event, /*maxevents=*/1,
>                          /*timeout=*/1000);
>   if (n == 0) {
>     // Timeout expired.
>   } else {
>     // Received RX interrupt.
>   }
> }
> 
> Given that I don't see anything coming through in /proc/interrupts, I am going to start digging through the ixgbe driver. However, I wanted to ask here first to see if my setup is missing anything obvious. I based it closely on the l3fwd-power example, though I haven't been able to get that example up and running yet since my NIC only has one port plugged in right now and the example requires two (COVID building restrictions).
> 
> Thanks,
> Jack Humphries

Hi Jack,

Have you called rte_eth_dev_rx_intr_enable(port_id, queue_id)?

Not sure about ixgbe, but vmxnet3 also requires rearming interrupts with it
on each iteration before rte_epoll_wait().
This is what l3fwd-power does by the way (see turn_on_off_intr() function).

+ ixgbe maintainer just in case.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Enable RX interrupts
  2021-11-01  7:46 ` Dmitry Kozlyuk
@ 2021-11-01  8:56   ` Jack Humphries
  2021-11-06 15:12     ` Thomas Monjalon
  0 siblings, 1 reply; 7+ messages in thread
From: Jack Humphries @ 2021-11-01  8:56 UTC (permalink / raw)
  To: Dmitry Kozlyuk; +Cc: users, Haiyue Wang

Hi Dmitry,

Thanks for the quick response! Yes, I was calling rte_eth_dev_rx_intr_enable — I just tried re-arming interrupts before each call to epoll and my initial experiment shows that fixes the issue.

You are right that l3fwd-power does the re-arming as well. If possible, it would be helpful to have a comment in that example code about the re-arming since the impression I got from quickly looking at the l3fwd-power code is that interrupts are turned off because the app wants to poll for a threshold of time before “giving up” and turning interrupts back on again. I’m happy to open a pull request to add this comment, too, if you prefer.

Thanks again for your help!
Jack

> On Nov 1, 2021, at 12:46 AM, Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
> 
> 2021-10-31 22:50 (UTC-0700), Jack Humphries:
>> Hi folks,
>> 
>> Hope all are well. I’m trying to enable interrupts in DPDK so that my network receive thread can sleep on an epoll until packets arrive. I am using the ixgbe kernel driver and igb_uio userspace driver with an Intel 82599ES 10Gbps NIC.
>> 
>> I'm doing roughly the following to enable the interrupts, but the epoll never indicates that packets have arrived. The thread only handles packets when the epoll times out. I don't even see interrupts arrive from the device when monitoring /proc/interrupts.
>> 
>> port_conf.intr_conf.rxq = 1;
>> 
>> ...
>> 
>> CHECK_EQ(rte_eth_dev_rx_intr_ctl_q(kPort, kQueue, RTE_EPOLL_PER_THREAD,
>>                                   RTE_INTR_EVENT_ADD, nullptr),
>>         0); 
>> CHECK_EQ(rte_eth_dev_rx_intr_enable(kPort, kQueue), 0);
>> 
>> ...
>> 
>> rte_epoll_event event;
>> while (true) {
>>  int n = rte_epoll_wait(RTE_EPOLL_PER_THREAD, &event, /*maxevents=*/1,
>>                         /*timeout=*/1000);
>>  if (n == 0) {
>>    // Timeout expired.
>>  } else {
>>    // Received RX interrupt.
>>  }
>> }
>> 
>> Given that I don't see anything coming through in /proc/interrupts, I am going to start digging through the ixgbe driver. However, I wanted to ask here first to see if my setup is missing anything obvious. I based it closely on the l3fwd-power example, though I haven't been able to get that example up and running yet since my NIC only has one port plugged in right now and the example requires two (COVID building restrictions).
>> 
>> Thanks,
>> Jack Humphries
> 
> Hi Jack,
> 
> Have you called rte_eth_dev_rx_intr_enable(port_id, queue_id)?
> 
> Not sure about ixgbe, but vmxnet3 also requires rearming interrupts with it
> on each iteration before rte_epoll_wait().
> This is what l3fwd-power does by the way (see turn_on_off_intr() function).
> 
> + ixgbe maintainer just in case.


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Enable RX interrupts
  2021-11-01  5:50 Enable RX interrupts Jack Humphries
  2021-11-01  7:46 ` Dmitry Kozlyuk
@ 2021-11-01 15:20 ` Stephen Hemminger
  2021-11-01 19:16   ` Jack Humphries
  1 sibling, 1 reply; 7+ messages in thread
From: Stephen Hemminger @ 2021-11-01 15:20 UTC (permalink / raw)
  To: Jack Humphries; +Cc: users

On Sun, 31 Oct 2021 22:50:40 -0700
Jack Humphries <jack@chillysky.com> wrote:

> Hi folks,
> 
> Hope all are well. I’m trying to enable interrupts in DPDK so that my network receive thread can sleep on an epoll until packets arrive. I am using the ixgbe kernel driver and igb_uio userspace driver with an Intel 82599ES 10Gbps NIC.
> 
> I'm doing roughly the following to enable the interrupts, but the epoll never indicates that packets have arrived. The thread only handles packets when the epoll times out. I don't even see interrupts arrive from the device when monitoring /proc/interrupts.
> 
> port_conf.intr_conf.rxq = 1;
> 
> ...
> 
> CHECK_EQ(rte_eth_dev_rx_intr_ctl_q(kPort, kQueue, RTE_EPOLL_PER_THREAD,
>                                    RTE_INTR_EVENT_ADD, nullptr),
>          0); 
> CHECK_EQ(rte_eth_dev_rx_intr_enable(kPort, kQueue), 0);
> 
> ...
> 
> rte_epoll_event event;
> while (true) {
>   int n = rte_epoll_wait(RTE_EPOLL_PER_THREAD, &event, /*maxevents=*/1,
>                          /*timeout=*/1000);
>   if (n == 0) {
>     // Timeout expired.
>   } else {
>     // Received RX interrupt.
>   }
> }
> 
> Given that I don't see anything coming through in /proc/interrupts, I am going to start digging through the ixgbe driver. However, I wanted to ask here first to see if my setup is missing anything obvious. I based it closely on the l3fwd-power example, though I haven't been able to get that example up and running yet since my NIC only has one port plugged in right now and the example requires two (COVID building restrictions).

If you are going to use interrupts you need to have MSI-X to get per-queue interrupts.
The igb_uio driver doesn't support this; you need to use vfio-pci in kernel to get queue interrupts.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Enable RX interrupts
  2021-11-01 15:20 ` Stephen Hemminger
@ 2021-11-01 19:16   ` Jack Humphries
  2021-11-01 19:51     ` Stephen Hemminger
  0 siblings, 1 reply; 7+ messages in thread
From: Jack Humphries @ 2021-11-01 19:16 UTC (permalink / raw)
  To: Stephen Hemminger; +Cc: users

Thanks Stephen. I am just using one RX queue from the NIC. I assume that since I don’t need per-queue interrupts here since there is only one queue, I should be good using ixgbe+igb_uio?

Thanks,
Jack

> On Nov 1, 2021, at 8:20 AM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> 
> On Sun, 31 Oct 2021 22:50:40 -0700
> Jack Humphries <jack@chillysky.com> wrote:
> 
>> Hi folks,
>> 
>> Hope all are well. I’m trying to enable interrupts in DPDK so that my network receive thread can sleep on an epoll until packets arrive. I am using the ixgbe kernel driver and igb_uio userspace driver with an Intel 82599ES 10Gbps NIC.
>> 
>> I'm doing roughly the following to enable the interrupts, but the epoll never indicates that packets have arrived. The thread only handles packets when the epoll times out. I don't even see interrupts arrive from the device when monitoring /proc/interrupts.
>> 
>> port_conf.intr_conf.rxq = 1;
>> 
>> ...
>> 
>> CHECK_EQ(rte_eth_dev_rx_intr_ctl_q(kPort, kQueue, RTE_EPOLL_PER_THREAD,
>>                                   RTE_INTR_EVENT_ADD, nullptr),
>>         0); 
>> CHECK_EQ(rte_eth_dev_rx_intr_enable(kPort, kQueue), 0);
>> 
>> ...
>> 
>> rte_epoll_event event;
>> while (true) {
>>  int n = rte_epoll_wait(RTE_EPOLL_PER_THREAD, &event, /*maxevents=*/1,
>>                         /*timeout=*/1000);
>>  if (n == 0) {
>>    // Timeout expired.
>>  } else {
>>    // Received RX interrupt.
>>  }
>> }
>> 
>> Given that I don't see anything coming through in /proc/interrupts, I am going to start digging through the ixgbe driver. However, I wanted to ask here first to see if my setup is missing anything obvious. I based it closely on the l3fwd-power example, though I haven't been able to get that example up and running yet since my NIC only has one port plugged in right now and the example requires two (COVID building restrictions).
> 
> If you are going to use interrupts you need to have MSI-X to get per-queue interrupts.
> The igb_uio driver doesn't support this; you need to use vfio-pci in kernel to get queue interrupts.


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Enable RX interrupts
  2021-11-01 19:16   ` Jack Humphries
@ 2021-11-01 19:51     ` Stephen Hemminger
  0 siblings, 0 replies; 7+ messages in thread
From: Stephen Hemminger @ 2021-11-01 19:51 UTC (permalink / raw)
  To: Jack Humphries; +Cc: users

On Mon, 1 Nov 2021 12:16:14 -0700
Jack Humphries <jack@chillysky.com> wrote:

> Thanks Stephen. I am just using one RX queue from the NIC. I assume that since I don’t need per-queue interrupts here since there is only one queue, I should be good using ixgbe+igb_uio?
> 
> Thanks,
> Jack
> 

Even with one queue, it won't work because the interrupt is dedicated to link status.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Enable RX interrupts
  2021-11-01  8:56   ` Jack Humphries
@ 2021-11-06 15:12     ` Thomas Monjalon
  0 siblings, 0 replies; 7+ messages in thread
From: Thomas Monjalon @ 2021-11-06 15:12 UTC (permalink / raw)
  To: Jack Humphries; +Cc: Dmitry Kozlyuk, users, Haiyue Wang, david.hunt

01/11/2021 09:56, Jack Humphries:
> Hi Dmitry,
> 
> Thanks for the quick response! Yes, I was calling rte_eth_dev_rx_intr_enable — I just tried re-arming interrupts before each call to epoll and my initial experiment shows that fixes the issue.
> 
> You are right that l3fwd-power does the re-arming as well. If possible, it would be helpful to have a comment in that example code about the re-arming since the impression I got from quickly looking at the l3fwd-power code is that interrupts are turned off because the app wants to poll for a threshold of time before “giving up” and turning interrupts back on again. I’m happy to open a pull request to add this comment, too, if you prefer.

The best is to submit a patch on the mailing list, Cc'in the relevant maintainers.
Thanks




^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2021-11-06 15:12 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-01  5:50 Enable RX interrupts Jack Humphries
2021-11-01  7:46 ` Dmitry Kozlyuk
2021-11-01  8:56   ` Jack Humphries
2021-11-06 15:12     ` Thomas Monjalon
2021-11-01 15:20 ` Stephen Hemminger
2021-11-01 19:16   ` Jack Humphries
2021-11-01 19:51     ` Stephen Hemminger

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).