[dpdk-users] Windows Interrupts with XL710/i40e

[dpdk-users] Windows Interrupts with XL710/i40e

[Mark Cheatham]

Hello All,

We have a very demanding project that requires capturing GVSP (GigE Vision Streaming Protocol) packets at ~2.5GB/s burst rates into Windows. For the past year or so, we have been building and using the draft DPDK repository located here: https://git.dpdk.org/draft/dpdk-draft-windows/

Overall, this has worked extremely well for our needs. We were also able to modify the source to include the e1000 (Intel I350) and store into externally allocated buffers.

Using the poll mode drivers for both e1000 and i40e, we see some interesting behavior that I believe is due to the cores spinning during times of low/no activity. These packets are fed into our application that has a lot of concurrent work happening. We were able to put in delays of a few milliseconds at points during the receive when we know no data should be coming in. This showed noticeable improvement; however, it would be great to have interrupt support.

We modified the netUIO Windows kernel driver and added WdfInterruptCreate() calls and added MSI support via the INF file. Brought in the same register sets into the driver into the ISR callback that was created to enable/disable interrupts. This works fine on the e1000, but for the i40e, there are some strange lags in getting the interrupt processed. We were not able to get an "interrupt 0 only" mode working for the i40e and had to enable the MSI-X vectors, but we see multiple milliseconds between when the ISR call occurs for an Rx packet and when we actually get our event set in the DPC. We have an FPGA that is generating the GVSP packets, and for a given image frame, it will always take ~18ms between the leader and trailer to be transmitted, and we are seeing cases of our Rx burst read taking less than this amount, which means that we begin reading well after the image frame has started to arrive. Also, as soon as an interrupt happens, we disable them and don't enable them until the read finishes. We are looking at dealing with 50 interrupts per second, max.

It should be noted that we are not using any of the interrupt skeleton code for Windows in the latest DPDK version. We are using an event handle (CreateEvent()) that's passed in to the netUIO driver, and then it's checked for being set with a separate DeviceIoControl() call. Once the event is set in the DPC, we kick off the Rx burst reads. I also tried changing some of the EITR indexes but no luck there. Again, this approach works fine in the e1000 driver with no delays.

Could it be the firmware/NVM on the 710? What else could be missing? From what I have read it should be on the order of <100 microseconds to issue the interrupt. I did not see anything obviously different from the interrupt register values on the draft repository vs. 21.05 (latest as of this email).

Thank you all for the input!

Regards,
Mark

Re: [dpdk-users] Windows Interrupts with XL710/i40e

[Dmitry Kozlyuk]

Hello, Mark!

+ Windows maintainers from Intel and Microsoft
to answer your questions regarding XL710 and kernel.

First of all, sounds like you've done a great job, congratulations!

Windows draft repository is not developed anymore.

DPDK has its own subsystem for handling interrupts.
In upstream Windows DPDK I'm implementing it currently, no patches sent yet.
The plan is to use IOCP and not events, but the flow will be similar:
the app subscribes to an interrupt event and gets its callback executed.
If you have any requirements for DPDK beyond that, please share them.

There is a bi-weekly Windows DPDK Community Call, if you want to participate
in discussions and to stay in the loop. Harini (cc'd) can send an invitation.

2021-06-28 22:17 (UTC+0000), Mark Cheatham: 
> We have a very demanding project that requires capturing GVSP (GigE Vision
> Streaming Protocol) packets at ~2.5GB/s burst rates into Windows. For the
> past year or so, we have been building and using the draft DPDK repository
> located here: https://git.dpdk.org/draft/dpdk-draft-windows/
>
> Overall, this has worked extremely well for our needs. We were also able to
> modify the source to include the e1000 (Intel I350) and store into
> externally allocated buffers. Using the poll mode drivers for both e1000
> and i40e, we see some interesting behavior that I believe is due to the
> cores spinning during times of low/no activity. These packets are fed into
> our application that has a lot of concurrent work happening. We were able
> to put in delays of a few milliseconds at points during the receive when we
> know no data should be coming in. This showed noticeable improvement;
> however, it would be great to have interrupt support.
> 
> We modified the netUIO Windows kernel driver and added WdfInterruptCreate()
> calls and added MSI support via the INF file. Brought in the same register
> sets into the driver into the ISR callback that was created to
> enable/disable interrupts. This works fine on the e1000, but for the i40e,
> there are some strange lags in getting the interrupt processed. We were not
> able to get an "interrupt 0 only" mode working for the i40e and had to
> enable the MSI-X vectors, but we see multiple milliseconds between when the
> ISR call occurs for an Rx packet and when we actually get our event set in
> the DPC. We have an FPGA that is generating the GVSP packets, and for a
> given image frame, it will always take ~18ms between the leader and trailer
> to be transmitted, and we are seeing cases of our Rx burst read taking less
> than this amount, which means that we begin reading well after the image
> frame has started to arrive. Also, as soon as an interrupt happens, we
> disable them and don't enable them until the read finishes. We are looking
> at dealing with 50 interrupts per second, max.
>
> It should be noted that we are not using any of the interrupt skeleton code
> for Windows in the latest DPDK version. We are using an event handle
> (CreateEvent()) that's passed in to the netUIO driver, and then it's
> checked for being set with a separate DeviceIoControl() call. Once the
> event is set in the DPC, we kick off the Rx burst reads. I also tried
> changing some of the EITR indexes but no luck there. Again, this approach
> works fine in the e1000 driver with no delays.
> 
> Could it be the firmware/NVM on the 710? What else could be missing? From
> what I have read it should be on the order of <100 microseconds to issue
> the interrupt. I did not see anything obviously different from the
> interrupt register values on the draft repository vs. 21.05 (latest as of
> this email).
> 
> Thank you all for the input!
> 
> Regards,
> Mark

Re: [dpdk-users] Windows Interrupts with XL710/i40e

[Mark Cheatham]

Hi Dmitry,

Thanks for getting back to me! Yes, I would love to be included on discussions for Windows DPDK. It sounds like using IOCP would work fine for the interrupt approach, and those would suit our needs vs. using events.

I am happy to provide further diagnostics with regards to the XL710 interrupts. If anyone has any ideas of things to test or try out, please let me know.

Regards,
Mark

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> 
Sent: Tuesday, June 29, 2021 1:51 PM
To: Mark Cheatham <mcheatham@boulderimaging.com>
Cc: users@dpdk.org; Kadam, Pallavi <pallavi.kadam@intel.com>; Menon, Ranjit <ranjit.menon@intel.com>; Dmitry Malloy (MESHCHANINOV) <dmitrym@microsoft.com>; Narcisa Ana Maria Vasile <Narcisa.Vasile@microsoft.com>; Harini Ramakrishnan <Harini.Ramakrishnan@microsoft.com>
Subject: Re: [dpdk-users] Windows Interrupts with XL710/i40e

CAUTION: This email originated from outside your organization. Exercise caution when opening attachments or clicking links, especially from unknown senders.

Hello, Mark!

+ Windows maintainers from Intel and Microsoft
to answer your questions regarding XL710 and kernel.

First of all, sounds like you've done a great job, congratulations!

Windows draft repository is not developed anymore.

DPDK has its own subsystem for handling interrupts.
In upstream Windows DPDK I'm implementing it currently, no patches sent yet.
The plan is to use IOCP and not events, but the flow will be similar:
the app subscribes to an interrupt event and gets its callback executed.
If you have any requirements for DPDK beyond that, please share them.

There is a bi-weekly Windows DPDK Community Call, if you want to participate in discussions and to stay in the loop. Harini (cc'd) can send an invitation.

2021-06-28 22:17 (UTC+0000), Mark Cheatham: 
> We have a very demanding project that requires capturing GVSP (GigE 
> Vision Streaming Protocol) packets at ~2.5GB/s burst rates into 
> Windows. For the past year or so, we have been building and using the 
> draft DPDK repository located here: 
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fgit.dpdk.org%2fdra
> ft%2fdpdk-draft-windows%2f&c=E,1,40vTC7l0vx8uUep1S8crsQwilJpRITxwnFKhi
> pYissOi7ruG02RQRt3crcKcUzJhJ3Qa73vi_s0gHE9RvKpx9eq9-HdhTIwIMEPBk-k2S25
> j9TRDgg,,&typo=1
>
> Overall, this has worked extremely well for our needs. We were also 
> able to modify the source to include the e1000 (Intel I350) and store 
> into externally allocated buffers. Using the poll mode drivers for 
> both e1000 and i40e, we see some interesting behavior that I believe 
> is due to the cores spinning during times of low/no activity. These 
> packets are fed into our application that has a lot of concurrent work 
> happening. We were able to put in delays of a few milliseconds at 
> points during the receive when we know no data should be coming in. 
> This showed noticeable improvement; however, it would be great to have interrupt support.
> 
> We modified the netUIO Windows kernel driver and added 
> WdfInterruptCreate() calls and added MSI support via the INF file. 
> Brought in the same register sets into the driver into the ISR 
> callback that was created to enable/disable interrupts. This works 
> fine on the e1000, but for the i40e, there are some strange lags in 
> getting the interrupt processed. We were not able to get an "interrupt 
> 0 only" mode working for the i40e and had to enable the MSI-X vectors, 
> but we see multiple milliseconds between when the ISR call occurs for 
> an Rx packet and when we actually get our event set in the DPC. We 
> have an FPGA that is generating the GVSP packets, and for a given 
> image frame, it will always take ~18ms between the leader and trailer 
> to be transmitted, and we are seeing cases of our Rx burst read taking 
> less than this amount, which means that we begin reading well after 
> the image frame has started to arrive. Also, as soon as an interrupt 
> happens, we disable them and don't enable them until the read finishes. We are looking at dealing with 50 interrupts per second, max.
>
> It should be noted that we are not using any of the interrupt skeleton 
> code for Windows in the latest DPDK version. We are using an event 
> handle
> (CreateEvent()) that's passed in to the netUIO driver, and then it's 
> checked for being set with a separate DeviceIoControl() call. Once the 
> event is set in the DPC, we kick off the Rx burst reads. I also tried 
> changing some of the EITR indexes but no luck there. Again, this 
> approach works fine in the e1000 driver with no delays.
> 
> Could it be the firmware/NVM on the 710? What else could be missing? 
> From what I have read it should be on the order of <100 microseconds 
> to issue the interrupt. I did not see anything obviously different 
> from the interrupt register values on the draft repository vs. 21.05 
> (latest as of this email).
> 
> Thank you all for the input!
> 
> Regards,
> Mark