From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 41A8EA0A0C for ; Tue, 29 Jun 2021 09:11:10 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 999714117E; Tue, 29 Jun 2021 09:11:01 +0200 (CEST) Received: from outbound-ip17b.ess.barracuda.com (outbound-ip17b.ess.barracuda.com [209.222.82.214]) by mails.dpdk.org (Postfix) with ESMTP id C69FF4068A for ; Tue, 29 Jun 2021 00:17:56 +0200 (CEST) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2100.outbound.protection.outlook.com [104.47.70.100]) by mx-outbound19-252.us-east-2b.ess.aws.cudaops.com (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 28 Jun 2021 22:17:54 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AEI+vY4fhZM2LJx7j2Dm0h5odLtI8TY2cvdQfgYwKS1v40D4d43uG3waAYr0DXEfmRw3K84e2boyZTrSn6NL5EDAQhmepeygmw/JnCmxE3CpiUfbp1aHTj6xZWatUGb3Q/xAMCfGqg+wazwjCIu9cHpqUYwZwZFkHipSnfQhzzTNjlPOakvC+u3bUH4IeLfZYpGAxlevwLAsuLqN/0yZBtbijuyNLpvYMQsmLhkfr3RLNdKHhXe/qfDUdLlb9y1d/mLwjxHadUgYwoUDxR9vYqNCmOYPzrgi8THOk2knpNnzdzDMeua+Xa141kKGYgnaOrcoCPfmSz5PdK+IBr2GjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MUFhp/S6snRZPcoWi2RL5BLaWSgX1OcZRSJQijQJpwg=; b=YwLFsFtrc13irbzUD8ZfvX32rPm8crqZSTf8Q6YQuNKhbbvI2bHjJNud7Vxnnm6F1+pUlBYqQL+CQJbIbNDwlmXI8EdrMfvglrMYMt7Zc54E2foIValpdsj7qdMKQa3AWyHxm6Khyq1ETHA2mgh/n2AmoqAzL+PTNZDDpEYItN3AHo5kBXfd6bIXz8LdmLT0+NHIXk8YrLWV/xS/IPfhdrTR0LfKcLRPRW9Ts6pmuRIrXQ0NIRuYGdAipCXCCgLIZqF2XuYiBHDYpNbsdaKaTk8zPEWUZqrgYuCKjCfERV5imrO6b2dEug8NSBCTF40LOESSmyh4i0RtegoyKe/B8Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=boulderimaging.com; dmarc=pass action=none header.from=boulderimaging.com; dkim=pass header.d=boulderimaging.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=boulderimaging.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MUFhp/S6snRZPcoWi2RL5BLaWSgX1OcZRSJQijQJpwg=; b=oBpREKng0SpewL8J9Xvs3s+lGl7yX/CR24K1zonyZNKcuo8n5bIn5zmMpDjCYngbOariNFf9eYWDN+ojB4ifpM7Oqv9qy5NvEQRMYGpx4hJ+glw7Jl8Mi8mDv8IFxH3YHwwHdeqyQ/p3Azes+HfE1ZUkgJYAhjXs92hf5jVMqwU= Received: from CY4PR08MB2566.namprd08.prod.outlook.com (2603:10b6:903:100::8) by CY1PR0801MB2185.namprd08.prod.outlook.com (2a01:111:e400:c611::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4264.20; Mon, 28 Jun 2021 22:17:52 +0000 Received: from CY4PR08MB2566.namprd08.prod.outlook.com ([fe80::ac73:fc3:8894:d3e4]) by CY4PR08MB2566.namprd08.prod.outlook.com ([fe80::ac73:fc3:8894:d3e4%9]) with mapi id 15.20.4264.026; Mon, 28 Jun 2021 22:17:52 +0000 From: Mark Cheatham To: "users@dpdk.org" Thread-Topic: Windows Interrupts with XL710/i40e Thread-Index: AddsZ2W0lR3+Vit5T4aXVYdILsf5Yg== Date: Mon, 28 Jun 2021 22:17:52 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=boulderimaging.com; x-originating-ip: [173.8.238.209] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a70efde8-6b42-427c-fe4a-08d93a829290 x-ms-traffictypediagnostic: CY1PR0801MB2185: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: XGEuK1LOUwV6AR+bLPlElCMUfWB+vLe01kjHbmWX5G5Ej+yCptoQ2i9906AAzwwqPjtc7FBiAkDP0ebvKv04ld4wlJwZTp294qVkwZ/BE6S7W2XDqzYH6KHsWi164StGOajZciKSdJS1Nr29absp4h5pxxMNded5phiu82VkjY7Bf8y2eSJ94gIfXnxrP8fr+8VTEMiJrn1vUIujiQpDrvRSc7UXhTpS2F8YgHkUdW9fW4Cfozp0Z4iaJ4/2RnCtjBw7Zfz/xUXZJEZFZBHahWueSTALYcQ6uwjZU6z7wcI6a53UDTuzegMpVgYC3POOI8GQIQf+rReFY/152dv0OWlWRdS3eqHQcOABm/52N+cMSEWozaPEOhz7JXz63UKQF3gyNvfqSFMNwGoppX09wFohw7KO83ZqFJQ3qIY/PeInphVgYB+ueYSTGoyC1SgpXM9BgzJ+wMLcuESMbhmOi7NVIMSsqf0Y+RicYVUWROZQLeoMu6+bGLMDurl0CcL9JJIsU26/IZQkUcqnG4WuCmjHfkRy05h7Jqqqk2NM5UHQ+Do4yaEbLR2x9AGGWSf3nzZ06NvpPUPjsMnt+ikjzuq44qeG8axD5y4O8s2pEKLmByIGh2Wp0sQePM6CmBdC/3Ni4pDDf++a2LJF2KAcItpZOsLLVg8mKA/3SuBMr4184XSoX/ng84lVfIkMw/do6ijSdaVgeJEog4JlEl/SQKAx44zqHxZy3qvLFMKwt2HSwtX8DH+LxkJCQqqidUR50qwybiDMeCoJvlwjH/eTMA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CY4PR08MB2566.namprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(136003)(346002)(396003)(39830400003)(376002)(478600001)(76116006)(66946007)(966005)(316002)(38100700002)(2906002)(7696005)(8936002)(122000001)(9686003)(5660300002)(55016002)(6916009)(66476007)(8676002)(83380400001)(186003)(9326002)(166002)(86362001)(64756008)(66556008)(66446008)(6506007)(26005)(33656002)(71200400001)(52536014); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?w3QnvuFD5LmU/svSo/MuJFmVMUpcAGx3O/pQvU03kfD6zNa5fTXyvE04DAdc?= =?us-ascii?Q?+u3SibqYj5hwVbj3cn95fr77j83RnQadU3aoM7Spxbc5brjL7/uL7K8n8+w3?= =?us-ascii?Q?YkncLqF65SETrJ/LgI5WHd8LXewaoW1ZVzcP/i53c9xwbIwF1ihMTzrOdQUQ?= =?us-ascii?Q?og4NH6v5eBFQ9maVGcOQ/V+PMShXu5OQQyRoviKm2VKxU3s6uC62tw5WWyab?= =?us-ascii?Q?IYhtfOxdA3wxRLhwBRzPf5lFiz4j40I1gsYwzF95WYFjuyMEIYwuQqzMHTEc?= =?us-ascii?Q?3medFUxs+YNqt36D1JQ8ZbFFiMingVneh5seIQHJXsuqmVxU8ID1qwCZWh/y?= =?us-ascii?Q?WgzPjOpiQpSBvMiudQYiPxCRCfkgWPQnpaG6Cd1Hs8cspu0xOMAxJzKiAIYh?= =?us-ascii?Q?zb5WhMVgTa67ZPuqfNIBy7llpQoAmr9DxHzQpwj8zM4P2v5Ao6uRprgopnim?= =?us-ascii?Q?bIfX0pP3CN23VSqRDTFoAyBgBl0hKFKiaTRWtsPxqsSJkQvB/ys63KWeW1DL?= =?us-ascii?Q?z3Q2Jv0PJEd94PvWSvt8euqpXfd38a//PposF1mWkrdj17NVaEPAesUBgsIb?= =?us-ascii?Q?OyYkKWRKlXJGM40LuN/w91P/y9oZM3criYDVpcxUBiiVVIWkRbpmBQXh8D/V?= =?us-ascii?Q?w9o4Im1pU5pe8wzVn2EaKsUEKjGxnJH0vNDDNS5mC8P71e5XkM9jf73SImt+?= =?us-ascii?Q?nJ3rO/QXfHK5gAjPe6JMY/uSqpy8x0zPOc4+6qHohNaPkNWr5pH1M6HQUAaU?= =?us-ascii?Q?roI3+sE7ndbsOccpGCsFwkuixXHPw12tORhWxR0/GEixEygk2tR/AX1NZJwV?= =?us-ascii?Q?dykQOS8KyfBQUmc8jpPA1mztdSaMjLeE5lXpEe0gbMbxOlT42OAIUXNQL0U5?= =?us-ascii?Q?vHB8nN3ex7YaDGZW/mO46Hv0r1Orv+Wb9sjepcLFNWjwbkPQE4wXamL2EtSS?= =?us-ascii?Q?0n97pF2UaU0ILoZ7ym5Lxh9VXbU1H2ZKT0Mn8VEnAscmgk3b0ix4YqBY9KQ/?= =?us-ascii?Q?QLOEDy5i83M7b6tFDq9EDr/qMuOkrDS4emOnRe7BTusxuwlFcJWgtZVbwPf5?= =?us-ascii?Q?7oxETzLJ6thCx5MreN0ujSFltU/wDg4FszhtT+/X6ffqnIaH+ekMPC9KtHzx?= =?us-ascii?Q?l0l1p3g5PbwjPPdxkm+T0CJm5VleEw3aa8UGKzUpiwTRTppUsC9I025saDjv?= =?us-ascii?Q?MAWkXglSjHoZiZNZBoQW/TPqGonc0o9OriuwF1+NHUd/RQSw4iQHjKNzmMwa?= =?us-ascii?Q?bt81CzIi1HMjrVOlIRAHP2z56Z17TOfClYxf4fHWRfkr6uK2+gBR2BrzaPxp?= =?us-ascii?Q?9HYgM9uhrzjknnAn0+HUFi+L?= x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: boulderimaging.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CY4PR08MB2566.namprd08.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: a70efde8-6b42-427c-fe4a-08d93a829290 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Jun 2021 22:17:52.7801 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 2f618655-ddf8-44dc-9f89-a72274cd6b97 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: DfalUzdx32EqDPQT/t8xoapM0nIyLoZlFU/+C7lRjU3lphjHwZqH+NTNIyAUC/mkPnks+wxH1DL66AmrS32HmtB2gn1cpWxv04raOwb2Ebg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0801MB2185 X-BESS-ID: 1624918674-105116-5292-58075-1 X-BESS-VER: 2019.1_20210623.1648 X-BESS-Apparent-Source-IP: 104.47.70.100 X-BESS-Outbound-Spam-Score: 0.00 X-BESS-Outbound-Spam-Report: Code version 3.2, rules version 3.2.2.233251 [from cloudscan9-114.us-east-2a.ess.aws.cudaops.com] Rule breakdown below pts rule name description ---- ---------------------- -------------------------------- 0.00 HTML_MESSAGE BODY: HTML included in message 0.00 BSF_BESS_OUTBOUND META: BESS Outbound X-BESS-Outbound-Spam-Status: SCORE=0.00 using account:ESS111980 scores of KILL_LEVEL=7.0 tests=HTML_MESSAGE, BSF_BESS_OUTBOUND X-BESS-BRTS-Status: 1 X-Mailman-Approved-At: Tue, 29 Jun 2021 09:10:59 +0200 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: [dpdk-users] Windows Interrupts with XL710/i40e X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hello All, We have a very demanding project that requires capturing GVSP (GigE Vision = Streaming Protocol) packets at ~2.5GB/s burst rates into Windows. For the p= ast year or so, we have been building and using the draft DPDK repository l= ocated here: https://git.dpdk.org/draft/dpdk-draft-windows/ Overall, this has worked extremely well for our needs. We were also able to= modify the source to include the e1000 (Intel I350) and store into externa= lly allocated buffers. Using the poll mode drivers for both e1000 and i40e, we see some interestin= g behavior that I believe is due to the cores spinning during times of low/= no activity. These packets are fed into our application that has a lot of c= oncurrent work happening. We were able to put in delays of a few millisecon= ds at points during the receive when we know no data should be coming in. T= his showed noticeable improvement; however, it would be great to have inter= rupt support. We modified the netUIO Windows kernel driver and added WdfInterruptCreate()= calls and added MSI support via the INF file. Brought in the same register= sets into the driver into the ISR callback that was created to enable/disa= ble interrupts. This works fine on the e1000, but for the i40e, there are s= ome strange lags in getting the interrupt processed. We were not able to ge= t an "interrupt 0 only" mode working for the i40e and had to enable the MSI= -X vectors, but we see multiple milliseconds between when the ISR call occu= rs for an Rx packet and when we actually get our event set in the DPC. We h= ave an FPGA that is generating the GVSP packets, and for a given image fram= e, it will always take ~18ms between the leader and trailer to be transmitt= ed, and we are seeing cases of our Rx burst read taking less than this amou= nt, which means that we begin reading well after the image frame has starte= d to arrive. Also, as soon as an interrupt happens, we disable them and don= 't enable them until the read finishes. We are looking at dealing with 50 i= nterrupts per second, max. It should be noted that we are not using any of the interrupt skeleton code= for Windows in the latest DPDK version. We are using an event handle (Crea= teEvent()) that's passed in to the netUIO driver, and then it's checked for= being set with a separate DeviceIoControl() call. Once the event is set in= the DPC, we kick off the Rx burst reads. I also tried changing some of the= EITR indexes but no luck there. Again, this approach works fine in the e10= 00 driver with no delays. Could it be the firmware/NVM on the 710? What else could be missing? From w= hat I have read it should be on the order of <100 microseconds to issue the= interrupt. I did not see anything obviously different from the interrupt r= egister values on the draft repository vs. 21.05 (latest as of this email). Thank you all for the input! Regards, Mark