From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 34656A0A0F for ; Mon, 5 Jul 2021 08:58:24 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C211A41156; Mon, 5 Jul 2021 08:58:23 +0200 (CEST) Received: from outbound-ip17a.ess.barracuda.com (outbound-ip17a.ess.barracuda.com [209.222.82.199]) by mails.dpdk.org (Postfix) with ESMTP id 7421240141 for ; Thu, 1 Jul 2021 17:58:07 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11lp2174.outbound.protection.outlook.com [104.47.56.174]) by mx-outbound10-141.us-east-2a.ess.aws.cudaops.com (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 01 Jul 2021 15:58:03 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PgqgwJqyIB+IjwSEYNtcG/dnVKoaVJE+noAKNnwk92EoGwt1yeYmNN4DKfYhTdkHRLYVt4LeL7g0Xwpf6GR8o7OgrnQ4EjPxKn/lM/OMJR2rk/pbk+RIBgUggt0pcvS/uAvNuipqBJIMuuc/AbbvoJUW2ceFyFdGAjW7kDXbYkvjRGQIP/RTuibYMH3YBegorDL4t/7kw2YA0CCPaflhnjCIqcMeA0VZU6w4kZHRPJTIhfSGCI3WQ8kHCoclIfe4hgc+8jWrXkSWFBk8/nLV3WY6j5TMlXUCRdV3V/90NSM2eH0HiHJwenT9e+CF4Pof+g9Tdz2hV9mR77ZwailTvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8Ig6lKwW1B7opxW44vUgkboUiRdevFN+l7JFlZk1aqk=; b=c5qJusHkFRsGts6H9jAyCh6fK5TzCrKUsubsKHxPcwuxzQH7tgSPirD4aDgDwdD0IshZfJZgRYFPEVqcOb0wUgABs/ACphl7YoHgzGv0YSA4pQBfMxdmShbi/LPqySxNeXWeKfFJcfnZRyqTUtn2FvqbvWUUlTjoN43LFqJET0KsTzIfj+ReG1pBWgDh+AH+4qjyAd05EzZwbubzr72TGvPNLyvxBAFOEvzw5xPFHFtF8RWDS+6M+QYzrC8vJFLtSiWC152P3hnVACz0tFd1J/jnMSc5LDeHJs7H87wsMCD3trHUmX2lInd4Dslpdu3xDtG4T8c2BWgXbymuQVQs8Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=boulderimaging.com; dmarc=pass action=none header.from=boulderimaging.com; dkim=pass header.d=boulderimaging.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=boulderimaging.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8Ig6lKwW1B7opxW44vUgkboUiRdevFN+l7JFlZk1aqk=; b=YZaAnhhMyXlB/aaQcubvhxDXI0y7EYZAdrwwGcLrmaAo2N+CHd3EGQGjmqVed+tSZlo6BOIPunBI0Y0lYtvK89NYToWm0Oo1NGqgCuC73yoPkImX9V2B21aR6DxN5VM/HumKNduPo5neniYBxI4Rt3me/sdctD9JblHJWf/qd8k= Received: from CY4PR08MB2566.namprd08.prod.outlook.com (2603:10b6:903:100::8) by CY4PR08MB2469.namprd08.prod.outlook.com (2603:10b6:903:43::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4264.23; Thu, 1 Jul 2021 15:58:01 +0000 Received: from CY4PR08MB2566.namprd08.prod.outlook.com ([fe80::ac73:fc3:8894:d3e4]) by CY4PR08MB2566.namprd08.prod.outlook.com ([fe80::ac73:fc3:8894:d3e4%9]) with mapi id 15.20.4264.028; Thu, 1 Jul 2021 15:58:01 +0000 From: Mark Cheatham To: Dmitry Kozlyuk CC: "users@dpdk.org" , "Kadam, Pallavi" , "Menon, Ranjit" , "Dmitry Malloy (MESHCHANINOV)" , Narcisa Ana Maria Vasile , Harini Ramakrishnan Thread-Topic: [dpdk-users] Windows Interrupts with XL710/i40e Thread-Index: AddsZ2W0lR3+Vit5T4aXVYdILsf5YgAuKsoAAFxJ1gA= Date: Thu, 1 Jul 2021 15:58:01 +0000 Message-ID: References: <20210629224929.668f9ba9@sovereign> In-Reply-To: <20210629224929.668f9ba9@sovereign> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=boulderimaging.com; x-originating-ip: [173.8.238.209] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 409a6233-8a82-4018-4491-08d93ca9010e x-ms-traffictypediagnostic: CY4PR08MB2469: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Q6YvB0fw3wFTMulQl/47Dk6KYcR2Fu9sMMMpr0zuWLQxDPP3WFeltZMTeIQsa1QOcNlwDg/lS4vZxUYDhSoI5I8Uwxg1Sb+KGlkfIMh8xQMXRY0NeanszGSUMItnDZR0rDfIjNUwF9fQ+F95+uK5Yp/Qio4jRNcNAA2bV0D/SYhZ4RCEkGJZcaxzVFVAWuk9q81AaoPIFMhg1aQr6ezqk/TlINbNGWh4B0ebXjcL/djnQRuFtKR/EQ56Cr4FvcZuOjHX4Q4ykStKnh9qsM0uoHwxzDalR7GT69VC9FkNtlIEfSKrSfByU3j3bbBaOST6Sw7zlAIlvQlwNyNofMnzlxCM0E/I0xVDYkCXIE38TWqItAN0YyWozjtHni89Qv06eCF+Z4WttwgpNI7A8V4Y2fhbOKRfsQfCFiuNb6Vm+tAqoj063j3eQDNw1xgndCQH7qyhVaktqDab2isiUbgc/J+DiCxaCobXqXv/2zjA7FUxaBO6pM7i9KYG6MNlk411ZgGZhmCqsADBnZyYRWmiK1IzovnGOvVVtnFbRCmBKifERDj9haOLNJ0zY9TMAXe3h4HwStRsQdDcK3+HFzq0sR5Tz5mckQhz4s2k5XBde33RGxlSGibpy5/InhCNEMc1Eo/eDIODyS80PN/JrVxrvnAHK1iynXqqL444pL06opPsrdTb/2gqI9cYsyJTOysJ1RPL9NcuyC27nsKbn6iqQw017PU2LDcRHAbmokt6tcM7wQQgoLviT3wUA4bLSXfif7uwMdf9HgeQwxtk+vS3QA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CY4PR08MB2566.namprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(54906003)(8676002)(64756008)(6916009)(66946007)(66446008)(66556008)(4326008)(9686003)(66476007)(52536014)(5660300002)(2906002)(55016002)(86362001)(33656002)(26005)(7696005)(83380400001)(186003)(53546011)(6506007)(71200400001)(122000001)(45080400002)(498600001)(8936002)(38100700002)(76116006)(966005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?10qBo6Zp7oYcr1zE8feZxRDusszoW8oRHw1L9cNegiqbfmATqGfRjP+p5oti?= =?us-ascii?Q?5+YvNOQ9OXNdSYJrKsZK32j4Dcv/DpY5xCQEJcNL8KB1GElgvjD49FQViKPC?= =?us-ascii?Q?mIkYzm5EgcTcZ/IBMa9ImDSVLpA1I1LRy8JB1ZhRdKGAZqHdqky+sycE2hd9?= =?us-ascii?Q?XyPiLUAhKW89vPdpfgEN9TrPjwAqgMr/2EyTx/aT8eeYkdN2jtF2dWxxE+n4?= =?us-ascii?Q?Zdu+I5elcOC54uECt1lU5NzzF1LQwDXVX6/q4ZTrqU2AeGrrooRJj20/m9xb?= =?us-ascii?Q?mmhqGNX+rDVIkX4+1Er3T1oUswEvCnjqzMH53tRn9KqHlNwDGe3XR5q6NvBW?= =?us-ascii?Q?XoPKjF4BpJKfrk+JaelKmDltnL4Jiz8JppD0kEVTb2kKchiVwMBU8SW07OrX?= =?us-ascii?Q?7MioSTqeIxP7xpSls9xnRnsrKbcG4YXl6UnQLPuqxyC9D+Zi53ltDdadZDsS?= =?us-ascii?Q?nGvcaGPJttxf4qv1yy1YGvjItf7XkUMTzSLiTG15lNqcW6KVhYFYtcJ+Jshv?= =?us-ascii?Q?rVsvulNUh85yIQXY1WVfGTzcpsDczMn5OsS4wvA1bIyojUF6uGOA3Ul741YM?= =?us-ascii?Q?8/irdSsyw+yO5HF3izd5/ZYvfBlTGy8BXzj6cXZRnuhYlaWeuodqFXifKYlq?= =?us-ascii?Q?DXf9o/2+qqruOYckjkBX9xnpx38KN6y+Fsw8No+fdufROkyeeJFDwmiGOPtF?= =?us-ascii?Q?plInp6rCi8xAgpIzNxP8fhYARenx5jbEKoEl1WBwqklJ6H0ZsXufxjROCiWO?= =?us-ascii?Q?lxK0BqHLZvvx31HUuXXDs1gfS+EO9uV1/Fu538rQh8mjp2c3Yvp10SzJHWy4?= =?us-ascii?Q?5G0AX6QuPAJE2TUfMZWYCUkcIiP/8NGpgmI6GQr4APNWnVnMfJjDJz5WjsJ7?= =?us-ascii?Q?U7LaKfOSzyLp/tg96gzb6X5ZdZmqMashBROv14PA000kpBMblUk4yPc5oZvX?= =?us-ascii?Q?pLBZPdSTHr26xnAFLTU5ku2tgTGQ22C5mXsLF8kLSL/k5m6Pb+VB+ddguk58?= =?us-ascii?Q?ERsZl3SBYv2YLAwUawr3YheDh/d4pcnKyspNhiqmcK+iiAVYvozKHxTOk03B?= =?us-ascii?Q?gX1BXDSX3zXARsqPd2I4mCliMYjnFMtrbn+bWaxmcTspRRBwntcqYqf/4YFo?= =?us-ascii?Q?quiqKxelQQs5Xvx7eRGPgo0UaMjEKLz0HRi2mbAw2GVizTvStvsiESZforyX?= =?us-ascii?Q?G2rOkOgCH5hpkaX+wmwyQ9NkBhVWyUBxVpiPn2jc/jFkHIa1HO/Nl97+pD8W?= =?us-ascii?Q?WsGG2a1Y1sDTRxOVHRFQL0jgaFthS30kwLTGIHUHn06+f+V24gG+al6JRK6B?= =?us-ascii?Q?lBeZFrIBOyLNyV6yl2ZZ3KvF?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: boulderimaging.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CY4PR08MB2566.namprd08.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 409a6233-8a82-4018-4491-08d93ca9010e X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Jul 2021 15:58:01.3798 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 2f618655-ddf8-44dc-9f89-a72274cd6b97 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: m+hy38ed9mjpbBQ1zQyDbpwxVmh9xwkG/hNqTzBt4J3ojSz6pJBt84LwQyOX4iGZj4U44wuzLU+SZ+DhAyPoRRyDJS67ClEWbUF5c0vf56Y= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR08MB2469 X-BESS-ID: 1625155083-102701-5293-34790-1 X-BESS-VER: 2019.1_20210629.1607 X-BESS-Apparent-Source-IP: 104.47.56.174 X-BESS-Outbound-Spam-Score: 1.00 X-BESS-Outbound-Spam-Report: Code version 3.2, rules version 3.2.2.233315 [from cloudscan9-47.us-east-2a.ess.aws.cudaops.com] Rule breakdown below pts rule name description ---- ---------------------- -------------------------------- 0.50 BSF_RULE7568M META: Custom Rule 7568M 0.50 BSF_RULE_7582B META: Custom Rule 7582B 0.00 BSF_BESS_OUTBOUND META: BESS Outbound X-BESS-Outbound-Spam-Status: SCORE=1.00 using account:ESS111980 scores of KILL_LEVEL=7.0 tests=BSF_RULE7568M, BSF_RULE_7582B, BSF_BESS_OUTBOUND X-BESS-BRTS-Status: 1 X-Mailman-Approved-At: Mon, 05 Jul 2021 08:58:23 +0200 Subject: Re: [dpdk-users] Windows Interrupts with XL710/i40e X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi Dmitry, Thanks for getting back to me! Yes, I would love to be included on discussi= ons for Windows DPDK. It sounds like using IOCP would work fine for the int= errupt approach, and those would suit our needs vs. using events. I am happy to provide further diagnostics with regards to the XL710 interru= pts. If anyone has any ideas of things to test or try out, please let me kn= ow. Regards, Mark -----Original Message----- From: Dmitry Kozlyuk =20 Sent: Tuesday, June 29, 2021 1:51 PM To: Mark Cheatham Cc: users@dpdk.org; Kadam, Pallavi ; Menon, Ranjit= ; Dmitry Malloy (MESHCHANINOV) ; Narcisa Ana Maria Vasile ; Harini Ramak= rishnan Subject: Re: [dpdk-users] Windows Interrupts with XL710/i40e CAUTION: This email originated from outside your organization. Exercise cau= tion when opening attachments or clicking links, especially from unknown se= nders. Hello, Mark! + Windows maintainers from Intel and Microsoft to answer your questions regarding XL710 and kernel. First of all, sounds like you've done a great job, congratulations! Windows draft repository is not developed anymore. DPDK has its own subsystem for handling interrupts. In upstream Windows DPDK I'm implementing it currently, no patches sent yet= . The plan is to use IOCP and not events, but the flow will be similar: the app subscribes to an interrupt event and gets its callback executed. If you have any requirements for DPDK beyond that, please share them. There is a bi-weekly Windows DPDK Community Call, if you want to participat= e in discussions and to stay in the loop. Harini (cc'd) can send an invitat= ion. 2021-06-28 22:17 (UTC+0000), Mark Cheatham:=20 > We have a very demanding project that requires capturing GVSP (GigE=20 > Vision Streaming Protocol) packets at ~2.5GB/s burst rates into=20 > Windows. For the past year or so, we have been building and using the=20 > draft DPDK repository located here:=20 > https://linkprotect.cudasvc.com/url?a=3Dhttps%3a%2f%2fgit.dpdk.org%2fdra > ft%2fdpdk-draft-windows%2f&c=3DE,1,40vTC7l0vx8uUep1S8crsQwilJpRITxwnFKhi > pYissOi7ruG02RQRt3crcKcUzJhJ3Qa73vi_s0gHE9RvKpx9eq9-HdhTIwIMEPBk-k2S25 > j9TRDgg,,&typo=3D1 > > Overall, this has worked extremely well for our needs. We were also=20 > able to modify the source to include the e1000 (Intel I350) and store=20 > into externally allocated buffers. Using the poll mode drivers for=20 > both e1000 and i40e, we see some interesting behavior that I believe=20 > is due to the cores spinning during times of low/no activity. These=20 > packets are fed into our application that has a lot of concurrent work=20 > happening. We were able to put in delays of a few milliseconds at=20 > points during the receive when we know no data should be coming in.=20 > This showed noticeable improvement; however, it would be great to have in= terrupt support. >=20 > We modified the netUIO Windows kernel driver and added=20 > WdfInterruptCreate() calls and added MSI support via the INF file.=20 > Brought in the same register sets into the driver into the ISR=20 > callback that was created to enable/disable interrupts. This works=20 > fine on the e1000, but for the i40e, there are some strange lags in=20 > getting the interrupt processed. We were not able to get an "interrupt=20 > 0 only" mode working for the i40e and had to enable the MSI-X vectors,=20 > but we see multiple milliseconds between when the ISR call occurs for=20 > an Rx packet and when we actually get our event set in the DPC. We=20 > have an FPGA that is generating the GVSP packets, and for a given=20 > image frame, it will always take ~18ms between the leader and trailer=20 > to be transmitted, and we are seeing cases of our Rx burst read taking=20 > less than this amount, which means that we begin reading well after=20 > the image frame has started to arrive. Also, as soon as an interrupt=20 > happens, we disable them and don't enable them until the read finishes. W= e are looking at dealing with 50 interrupts per second, max. > > It should be noted that we are not using any of the interrupt skeleton=20 > code for Windows in the latest DPDK version. We are using an event=20 > handle > (CreateEvent()) that's passed in to the netUIO driver, and then it's=20 > checked for being set with a separate DeviceIoControl() call. Once the=20 > event is set in the DPC, we kick off the Rx burst reads. I also tried=20 > changing some of the EITR indexes but no luck there. Again, this=20 > approach works fine in the e1000 driver with no delays. >=20 > Could it be the firmware/NVM on the 710? What else could be missing?=20 > From what I have read it should be on the order of <100 microseconds=20 > to issue the interrupt. I did not see anything obviously different=20 > from the interrupt register values on the draft repository vs. 21.05=20 > (latest as of this email). >=20 > Thank you all for the input! >=20 > Regards, > Mark