DPDK usage discussions
 help / color / mirror / Atom feed
* [dpdk-users] Windows Interrupts with XL710/i40e
@ 2021-06-28 22:17 Mark Cheatham
  2021-06-29 19:50 ` Dmitry Kozlyuk
  0 siblings, 1 reply; 3+ messages in thread
From: Mark Cheatham @ 2021-06-28 22:17 UTC (permalink / raw)
  To: users

Hello All,

We have a very demanding project that requires capturing GVSP (GigE Vision Streaming Protocol) packets at ~2.5GB/s burst rates into Windows. For the past year or so, we have been building and using the draft DPDK repository located here: https://git.dpdk.org/draft/dpdk-draft-windows/

Overall, this has worked extremely well for our needs. We were also able to modify the source to include the e1000 (Intel I350) and store into externally allocated buffers.

Using the poll mode drivers for both e1000 and i40e, we see some interesting behavior that I believe is due to the cores spinning during times of low/no activity. These packets are fed into our application that has a lot of concurrent work happening. We were able to put in delays of a few milliseconds at points during the receive when we know no data should be coming in. This showed noticeable improvement; however, it would be great to have interrupt support.

We modified the netUIO Windows kernel driver and added WdfInterruptCreate() calls and added MSI support via the INF file. Brought in the same register sets into the driver into the ISR callback that was created to enable/disable interrupts. This works fine on the e1000, but for the i40e, there are some strange lags in getting the interrupt processed. We were not able to get an "interrupt 0 only" mode working for the i40e and had to enable the MSI-X vectors, but we see multiple milliseconds between when the ISR call occurs for an Rx packet and when we actually get our event set in the DPC. We have an FPGA that is generating the GVSP packets, and for a given image frame, it will always take ~18ms between the leader and trailer to be transmitted, and we are seeing cases of our Rx burst read taking less than this amount, which means that we begin reading well after the image frame has started to arrive. Also, as soon as an interrupt happens, we disable them and don't enable them until the read finishes. We are looking at dealing with 50 interrupts per second, max.

It should be noted that we are not using any of the interrupt skeleton code for Windows in the latest DPDK version. We are using an event handle (CreateEvent()) that's passed in to the netUIO driver, and then it's checked for being set with a separate DeviceIoControl() call. Once the event is set in the DPC, we kick off the Rx burst reads. I also tried changing some of the EITR indexes but no luck there. Again, this approach works fine in the e1000 driver with no delays.

Could it be the firmware/NVM on the 710? What else could be missing? From what I have read it should be on the order of <100 microseconds to issue the interrupt. I did not see anything obviously different from the interrupt register values on the draft repository vs. 21.05 (latest as of this email).

Thank you all for the input!

Regards,
Mark

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [dpdk-users] Windows Interrupts with XL710/i40e
  2021-06-28 22:17 [dpdk-users] Windows Interrupts with XL710/i40e Mark Cheatham
@ 2021-06-29 19:50 ` Dmitry Kozlyuk
  2021-07-01 15:58   ` Mark Cheatham
  0 siblings, 1 reply; 3+ messages in thread
From: Dmitry Kozlyuk @ 2021-06-29 19:50 UTC (permalink / raw)
  To: Mark Cheatham
  Cc: users, Kadam, Pallavi, Menon, Ranjit,
	Dmitry Malloy (MESHCHANINOV),
	Narcisa Ana Maria Vasile, Harini Ramakrishnan

Hello, Mark!

+ Windows maintainers from Intel and Microsoft
to answer your questions regarding XL710 and kernel.

First of all, sounds like you've done a great job, congratulations!

Windows draft repository is not developed anymore.

DPDK has its own subsystem for handling interrupts.
In upstream Windows DPDK I'm implementing it currently, no patches sent yet.
The plan is to use IOCP and not events, but the flow will be similar:
the app subscribes to an interrupt event and gets its callback executed.
If you have any requirements for DPDK beyond that, please share them.

There is a bi-weekly Windows DPDK Community Call, if you want to participate
in discussions and to stay in the loop. Harini (cc'd) can send an invitation.

2021-06-28 22:17 (UTC+0000), Mark Cheatham: 
> We have a very demanding project that requires capturing GVSP (GigE Vision
> Streaming Protocol) packets at ~2.5GB/s burst rates into Windows. For the
> past year or so, we have been building and using the draft DPDK repository
> located here: https://git.dpdk.org/draft/dpdk-draft-windows/
>
> Overall, this has worked extremely well for our needs. We were also able to
> modify the source to include the e1000 (Intel I350) and store into
> externally allocated buffers. Using the poll mode drivers for both e1000
> and i40e, we see some interesting behavior that I believe is due to the
> cores spinning during times of low/no activity. These packets are fed into
> our application that has a lot of concurrent work happening. We were able
> to put in delays of a few milliseconds at points during the receive when we
> know no data should be coming in. This showed noticeable improvement;
> however, it would be great to have interrupt support.
> 
> We modified the netUIO Windows kernel driver and added WdfInterruptCreate()
> calls and added MSI support via the INF file. Brought in the same register
> sets into the driver into the ISR callback that was created to
> enable/disable interrupts. This works fine on the e1000, but for the i40e,
> there are some strange lags in getting the interrupt processed. We were not
> able to get an "interrupt 0 only" mode working for the i40e and had to
> enable the MSI-X vectors, but we see multiple milliseconds between when the
> ISR call occurs for an Rx packet and when we actually get our event set in
> the DPC. We have an FPGA that is generating the GVSP packets, and for a
> given image frame, it will always take ~18ms between the leader and trailer
> to be transmitted, and we are seeing cases of our Rx burst read taking less
> than this amount, which means that we begin reading well after the image
> frame has started to arrive. Also, as soon as an interrupt happens, we
> disable them and don't enable them until the read finishes. We are looking
> at dealing with 50 interrupts per second, max.
>
> It should be noted that we are not using any of the interrupt skeleton code
> for Windows in the latest DPDK version. We are using an event handle
> (CreateEvent()) that's passed in to the netUIO driver, and then it's
> checked for being set with a separate DeviceIoControl() call. Once the
> event is set in the DPC, we kick off the Rx burst reads. I also tried
> changing some of the EITR indexes but no luck there. Again, this approach
> works fine in the e1000 driver with no delays.
> 
> Could it be the firmware/NVM on the 710? What else could be missing? From
> what I have read it should be on the order of <100 microseconds to issue
> the interrupt. I did not see anything obviously different from the
> interrupt register values on the draft repository vs. 21.05 (latest as of
> this email).
> 
> Thank you all for the input!
> 
> Regards,
> Mark


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [dpdk-users] Windows Interrupts with XL710/i40e
  2021-06-29 19:50 ` Dmitry Kozlyuk
@ 2021-07-01 15:58   ` Mark Cheatham
  0 siblings, 0 replies; 3+ messages in thread
From: Mark Cheatham @ 2021-07-01 15:58 UTC (permalink / raw)
  To: Dmitry Kozlyuk
  Cc: users, Kadam, Pallavi, Menon, Ranjit,
	Dmitry Malloy (MESHCHANINOV),
	Narcisa Ana Maria Vasile, Harini Ramakrishnan

Hi Dmitry,

Thanks for getting back to me! Yes, I would love to be included on discussions for Windows DPDK. It sounds like using IOCP would work fine for the interrupt approach, and those would suit our needs vs. using events.

I am happy to provide further diagnostics with regards to the XL710 interrupts. If anyone has any ideas of things to test or try out, please let me know.

Regards,
Mark

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> 
Sent: Tuesday, June 29, 2021 1:51 PM
To: Mark Cheatham <mcheatham@boulderimaging.com>
Cc: users@dpdk.org; Kadam, Pallavi <pallavi.kadam@intel.com>; Menon, Ranjit <ranjit.menon@intel.com>; Dmitry Malloy (MESHCHANINOV) <dmitrym@microsoft.com>; Narcisa Ana Maria Vasile <Narcisa.Vasile@microsoft.com>; Harini Ramakrishnan <Harini.Ramakrishnan@microsoft.com>
Subject: Re: [dpdk-users] Windows Interrupts with XL710/i40e

CAUTION: This email originated from outside your organization. Exercise caution when opening attachments or clicking links, especially from unknown senders.

Hello, Mark!

+ Windows maintainers from Intel and Microsoft
to answer your questions regarding XL710 and kernel.

First of all, sounds like you've done a great job, congratulations!

Windows draft repository is not developed anymore.

DPDK has its own subsystem for handling interrupts.
In upstream Windows DPDK I'm implementing it currently, no patches sent yet.
The plan is to use IOCP and not events, but the flow will be similar:
the app subscribes to an interrupt event and gets its callback executed.
If you have any requirements for DPDK beyond that, please share them.

There is a bi-weekly Windows DPDK Community Call, if you want to participate in discussions and to stay in the loop. Harini (cc'd) can send an invitation.

2021-06-28 22:17 (UTC+0000), Mark Cheatham: 
> We have a very demanding project that requires capturing GVSP (GigE 
> Vision Streaming Protocol) packets at ~2.5GB/s burst rates into 
> Windows. For the past year or so, we have been building and using the 
> draft DPDK repository located here: 
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fgit.dpdk.org%2fdra
> ft%2fdpdk-draft-windows%2f&c=E,1,40vTC7l0vx8uUep1S8crsQwilJpRITxwnFKhi
> pYissOi7ruG02RQRt3crcKcUzJhJ3Qa73vi_s0gHE9RvKpx9eq9-HdhTIwIMEPBk-k2S25
> j9TRDgg,,&typo=1
>
> Overall, this has worked extremely well for our needs. We were also 
> able to modify the source to include the e1000 (Intel I350) and store 
> into externally allocated buffers. Using the poll mode drivers for 
> both e1000 and i40e, we see some interesting behavior that I believe 
> is due to the cores spinning during times of low/no activity. These 
> packets are fed into our application that has a lot of concurrent work 
> happening. We were able to put in delays of a few milliseconds at 
> points during the receive when we know no data should be coming in. 
> This showed noticeable improvement; however, it would be great to have interrupt support.
> 
> We modified the netUIO Windows kernel driver and added 
> WdfInterruptCreate() calls and added MSI support via the INF file. 
> Brought in the same register sets into the driver into the ISR 
> callback that was created to enable/disable interrupts. This works 
> fine on the e1000, but for the i40e, there are some strange lags in 
> getting the interrupt processed. We were not able to get an "interrupt 
> 0 only" mode working for the i40e and had to enable the MSI-X vectors, 
> but we see multiple milliseconds between when the ISR call occurs for 
> an Rx packet and when we actually get our event set in the DPC. We 
> have an FPGA that is generating the GVSP packets, and for a given 
> image frame, it will always take ~18ms between the leader and trailer 
> to be transmitted, and we are seeing cases of our Rx burst read taking 
> less than this amount, which means that we begin reading well after 
> the image frame has started to arrive. Also, as soon as an interrupt 
> happens, we disable them and don't enable them until the read finishes. We are looking at dealing with 50 interrupts per second, max.
>
> It should be noted that we are not using any of the interrupt skeleton 
> code for Windows in the latest DPDK version. We are using an event 
> handle
> (CreateEvent()) that's passed in to the netUIO driver, and then it's 
> checked for being set with a separate DeviceIoControl() call. Once the 
> event is set in the DPC, we kick off the Rx burst reads. I also tried 
> changing some of the EITR indexes but no luck there. Again, this 
> approach works fine in the e1000 driver with no delays.
> 
> Could it be the firmware/NVM on the 710? What else could be missing? 
> From what I have read it should be on the order of <100 microseconds 
> to issue the interrupt. I did not see anything obviously different 
> from the interrupt register values on the draft repository vs. 21.05 
> (latest as of this email).
> 
> Thank you all for the input!
> 
> Regards,
> Mark


^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2021-07-05  6:58 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-28 22:17 [dpdk-users] Windows Interrupts with XL710/i40e Mark Cheatham
2021-06-29 19:50 ` Dmitry Kozlyuk
2021-07-01 15:58   ` Mark Cheatham

DPDK usage discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.dpdk.org/users/0 users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 users users/ http://inbox.dpdk.org/users \
		users@dpdk.org
	public-inbox-index users

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git