From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3FCE6A0613 for ; Mon, 26 Aug 2019 08:11:28 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7E5DB1BF33; Mon, 26 Aug 2019 08:11:27 +0200 (CEST) Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-eopbgr20087.outbound.protection.outlook.com [40.107.2.87]) by dpdk.org (Postfix) with ESMTP id 513891BF32 for ; Mon, 26 Aug 2019 08:11:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=h+szHqMiC95j9vUXvl4S32BVwbmFm7xDiZKb5datqnI=; b=NEnESa6TMkRhNsaL+NOB9uo3IifaoKhmuTMlX9Oq3NNBxL8+jWlXtfHluZxwQGrtwq9y2rz9/k6qQdbVULZlNsN20sIgJ/gOUVAthzWdfGzigXW1rMCDOEwTQGYoQ/rRFAoS4vfgEjsxvt9fSJCipFx16NRaZOG+Y29MPmZzdSk= Received: from VI1PR0801CA0072.eurprd08.prod.outlook.com (2603:10a6:800:7d::16) by DB6PR0802MB2599.eurprd08.prod.outlook.com (2603:10a6:4:a2::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2178.16; Mon, 26 Aug 2019 06:11:22 +0000 Received: from AM5EUR03FT045.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e08::200) by VI1PR0801CA0072.outlook.office365.com (2603:10a6:800:7d::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2199.15 via Frontend Transport; Mon, 26 Aug 2019 06:11:21 +0000 Authentication-Results: spf=temperror (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=temperror action=none header.from=arm.com; Received-SPF: TempError (protection.outlook.com: error in processing during lookup of arm.com: DNS Timeout) Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT045.mail.protection.outlook.com (10.152.17.105) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2199.13 via Frontend Transport; Mon, 26 Aug 2019 06:11:20 +0000 Received: ("Tessian outbound df6366865adf:v27"); Mon, 26 Aug 2019 06:11:18 +0000 X-CR-MTA-TID: 64aa7808 Received: from 1ddf7489228f.1 (ip-172-16-0-2.eu-west-1.compute.internal [104.47.12.59]) by 64aa7808-outbound-1.mta.getcheckrecipient.com id D9CBF53D-3093-48F2-911C-0C24D768E7A3.1; Mon, 26 Aug 2019 06:11:12 +0000 Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-db3eur04lp2059.outbound.protection.outlook.com [104.47.12.59]) by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 1ddf7489228f.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Mon, 26 Aug 2019 06:11:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dONQIaaOPWJssAM31IN0lNE5QPT+sghpUaq9lwfX5wH7JYnSmiWM7nzLNZ8Ro2gsoDXkZM+Io5z+elPE9blCuxCqSZCs8kdBRj6NZsZ4yhJWixOS+rwv1qk3IVI82xAJEEF7SQ7p3VEWBeb1XIL9P+AbxjPW7qj0pntq2zuMZNoeZUBxAlG8+4Kj5QGKJ5JVeR5D5E+5RW31QJfbaV2AU6jCYYw+CyaXXNHl0236fGHI9v8jhJOHC+lzxVAV2SUzMr3albZ4gOYXVFqPeHl5Oo2vTJSLEy86O278RnY34de0tXCggbMSvbaotfdHSqP4bVF7tXRq7qSZ9RRmECQJHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=h+szHqMiC95j9vUXvl4S32BVwbmFm7xDiZKb5datqnI=; b=ccYWpsSJjNC+Gaapv7Q4H5Aj6nj+LVmuNHsxXKPLJ8jB7SH3pCBP5XpGzJaRHbEmISy/c8XI5b1EuAd9/Qb2eKNOJtwci5HBcBUubODglOhgjb+lAgNzkXCOZfLGp2PAMEhTemRtW2Jr0yw4/w6vQ30xNqjm06vC41uUJAebWsTtCvxZ9aeR6h0C3tMZfv0x/E7d4z3OPzY1CY5vj0F0gkrqqGYkrAaV2DpYlObnSQ+Bj6R2qs/nuj3bdDBu/+Qy0RhoRjvILFcMAvTLgBRK4BYwEJ5mx2qZXfLISA/4GlPeajOTS6Pho3onB2rFZAxrzK7Bzzl50MR02kd8B5h8xA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=h+szHqMiC95j9vUXvl4S32BVwbmFm7xDiZKb5datqnI=; b=NEnESa6TMkRhNsaL+NOB9uo3IifaoKhmuTMlX9Oq3NNBxL8+jWlXtfHluZxwQGrtwq9y2rz9/k6qQdbVULZlNsN20sIgJ/gOUVAthzWdfGzigXW1rMCDOEwTQGYoQ/rRFAoS4vfgEjsxvt9fSJCipFx16NRaZOG+Y29MPmZzdSk= Received: from VE1PR08MB5149.eurprd08.prod.outlook.com (20.179.30.152) by VE1PR08MB4974.eurprd08.prod.outlook.com (10.255.158.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2178.20; Mon, 26 Aug 2019 06:11:09 +0000 Received: from VE1PR08MB5149.eurprd08.prod.outlook.com ([fe80::a8af:a9b8:4597:4128]) by VE1PR08MB5149.eurprd08.prod.outlook.com ([fe80::a8af:a9b8:4597:4128%3]) with mapi id 15.20.2178.020; Mon, 26 Aug 2019 06:11:09 +0000 From: Honnappa Nagarahalli To: Ye Xiaolong , "Zhang, Qi Z" CC: "Gavin Hu (Arm Technology China)" , "Richardson, Bruce" , "Ananyev, Konstantin" , "users@dpdk.org" , "Phil Yang (Arm Technology China)" , "Ruifeng Wang (Arm Technology China)" , nd Thread-Topic: i40e vPMD fix out of order Rx read issue Thread-Index: AdVCzJhDNbgBl9FATkWplfpJPawhKgAEGCLwAC59RaAAIKjSIAOMKbFQAAGo47AAYyTYgAH9EtTw Date: Mon, 26 Aug 2019 06:11:08 +0000 Message-ID: References: <039ED4275CED7440929022BC67E7061153D71548@SHSMSX105.ccr.corp.intel.com> <039ED4275CED7440929022BC67E7061153D7485E@SHSMSX105.ccr.corp.intel.com> <039ED4275CED7440929022BC67E7061153D7D657@SHSMSX105.ccr.corp.intel.com> <20190816025058.GA53345@intel.com> In-Reply-To: <20190816025058.GA53345@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: f931b255-537e-431e-a1b9-867dd40e6418.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; x-originating-ip: [217.140.111.135] x-ms-publictraffictype: Email X-MS-Office365-Filtering-Correlation-Id: 07420e04-2e7c-40cd-3cfe-08d729ec36aa X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam-Untrusted: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600166)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020); SRVR:VE1PR08MB4974; X-MS-TrafficTypeDiagnostic: VE1PR08MB4974:|DB6PR0802MB2599: X-MS-Exchange-PUrlCount: 5 x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true x-ms-oob-tlc-oobclassifiers: OLM:1728;OLM:1728; x-forefront-prvs: 01415BB535 X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(136003)(396003)(346002)(366004)(376002)(39850400004)(13464003)(199004)(189003)(4326008)(7736002)(229853002)(316002)(99286004)(478600001)(476003)(66066001)(102836004)(74316002)(8936002)(26005)(66946007)(446003)(11346002)(76176011)(25786009)(71200400001)(71190400001)(186003)(53546011)(2906002)(256004)(9686003)(53936002)(81166006)(14454004)(6506007)(55016002)(52536014)(14444005)(6246003)(6116002)(54906003)(486006)(5660300002)(66446008)(66476007)(64756008)(66556008)(86362001)(305945005)(6436002)(33656002)(7696005)(81156014)(6306002)(110136005)(8676002)(76116006)(3846002)(966005); DIR:OUT; SFP:1101; SCL:1; SRVR:VE1PR08MB4974; H:VE1PR08MB5149.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Message-Info-Original: wvl+h4JFgmAM2RJ1St2bWnlgH6C7kgLg23NdWZciAOXFsCLyzbI8MHqNPuoEuGAIpASsgoR3PNNeXzZ72NDj/aABsu1d+MOaBlnKM4crxl6+5W4JfD7zImD/KVJh04cP7pxzplouylM2pD1blMlkgMRYXaIt8xgubsiC+ZmKVQLM/vvWASsuS92gKWDxB9238VLpQgItYlF4H+TLsP6whyUFaOVLaAKb5sFxrETeoT04B8sMyjBza2vXojJCeqVVaq696zNF6ZQtEa1wTG32X3Hy8Q2IILJZYkZOTRYMEf88Qt+ePLe0gmc5CjhJXtfMw6clWThJm9/oLFwnokjpBHgbdWJkneN3DlzpWjaOMWO1kxZhnduv6oPhijNEqOAD6kV2wGMd3YAEttv/ddSwyE+NW/T4or/WgAdv4e5k+MY= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1PR08MB4974 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT045.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; IPV:CAL; SCL:-1; CTRY:IE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(376002)(396003)(136003)(39850400004)(346002)(2980300002)(13464003)(199004)(189003)(11346002)(52536014)(47776003)(26826003)(25786009)(229853002)(126002)(33656002)(97756001)(76130400001)(55016002)(102836004)(478600001)(6246003)(23726003)(66066001)(6306002)(9686003)(81156014)(966005)(186003)(14454004)(7736002)(336012)(50466002)(316002)(2906002)(356004)(7696005)(76176011)(74316002)(53546011)(486006)(22756006)(110136005)(54906003)(305945005)(99286004)(36906005)(3846002)(446003)(6116002)(70206006)(8676002)(46406003)(476003)(4326008)(70586007)(5660300002)(8936002)(63370400001)(63350400001)(81166006)(14444005)(86362001)(26005)(6506007)(8746002); DIR:OUT; SFP:1101; SCL:1; SRVR:DB6PR0802MB2599; H:64aa7808-outbound-1.mta.getcheckrecipient.com; FPR:; SPF:TempError; LANG:en; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; A:1; MX:1; X-MS-Office365-Filtering-Correlation-Id-Prvs: 43c357f7-3797-43f3-f295-08d729ec3045 X-Microsoft-Antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600166)(710020)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020); SRVR:DB6PR0802MB2599; NoDisclaimer: True X-Forefront-PRVS: 01415BB535 X-Microsoft-Antispam-Message-Info: dDBfKZ3xEiwJy09RL2ZlCfq0kMGBaXfp2x6WkLIx4X9YSx4Et5YSinNwFIawzLYJ8fk4IR0zq5ZblkmB5/YeW026oa8qajgOguUEglOmsx4jAssV3NG+SQVCYQbNqKP8KSuIcIXNMThWImqz7Kkzv5yxWVXx+E+mUHRUgSvXWfToPSTAfiTJBzIAaEy6dULWyONXjRSvLVMkYPWIDBGLAggw1wcyVaN24WdALrXaRRA3WO+HE0HIW9j2osGQttaANcuHK4kMcv0ARINLIl8zeuOwfVr30W7i/x8pCVWLmswv9AZJA9ogCJb4U8SyjLQqGYm+8Y8UlUItwIMNMCaqb5F5/V08ybV2Td/iEax90nRbmy8Cw6cgFnoMdMtJhrazuTMkdFUBPdfV3AMFGYZ+vq9sl21Kn3GGdYvf9x3Mhw4= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2019 06:11:20.1631 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 07420e04-2e7c-40cd-3cfe-08d729ec36aa X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR0802MB2599 Subject: Re: [dpdk-users] i40e vPMD fix out of order Rx read issue X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi Qi, Thank you so much for your explanation. I am trying to review this patch a= nd I have further questions on the NIC behavior. > >> > > Subject: RE: i40e vPMD fix out of order Rx read issue > >> > > > >> > > Hi Qi, > >> > > > >> > > Thanks for your explanation! > >> > > I did some testing and found the barriers caused a big drop in > >> > > RFC2544 NDR performance on aarch64, how about it on X86? > >> > For x86, I think we don't have performance drop, the memory barrier > >> > just change the compiler's behavior to avoid generate out of order > >> > read instructions, and x86 guarantee no out of read execution, so > >> > it does not add new instructions that cost CPU cycles. > >> > > >> > > Is it possible to count DD bits in a way of surviving across the > >> > > out-of-order descriptors reading? > >> > > >> > I think it is possible, but this will impact performance on x86, > >> > but for aarch64, you can try out to see if that benefit and do > >> > proper optimization on related vPMD implementation. > >> The patches for aarch64 are in community review, and performances > >> gain were measured both for ixgbe and i40e vPMD. Very appreciate if > >> you can further review and comment. > >> Here are the patch links: > >> IXGBE vPMD: > >> http://patches.dpdk.org/patch/57649/ > >> http://patches.dpdk.org/patch/57650/ > >> I40e vPMD: > >> http://patches.dpdk.org/cover/57651/ > >> http://patches.dpdk.org/patch/57652/ > > > > > > > >> > > > >> > > Best Regards, > >> > > Gavin > >> > > > >> > > > -----Original Message----- > >> > > > From: Zhang, Qi Z > >> > > > Sent: Thursday, July 25, 2019 8:11 PM > >> > > > To: Gavin Hu (Arm Technology China) ; > >> > > > Richardson, Bruce ; Ananyev, > >> > > > Konstantin > >> > > > Cc: users@dpdk.org; Honnappa Nagarahalli > >> > > > ; Phil Yang (Arm Technology > >> > > > China) > >> > > > Subject: RE: i40e vPMD fix out of order Rx read issue > >> > > > > >> > > > Hi Gavin: > >> > > > > >> > > > in vPMD, we read 4 or 8 packets as batch, we count DD bits for > >> > > > packet received, but not check the if they are continues or > >> > > > not, we assume it should always be 1000, 1100, 1110, 1111 > >> > > > ....(take batch size is 4 as > >> > > > example) while the out of order read instruction generated by > >> > > > compiler will cause driver to get un-continues DD bits, like > >> > > > 1011, the descriptor on the hole actually is invalid since when > >> > > > it is read , descriptor is not write back yet, but we still > >> > > > process this as 1110, it cause > >> > an > >> > > corrupted mbuf returned. 1) vPMD reads the descriptors in the reverse order. Let us number them D4, = D3, D2, D1 for this discussion (1 to 4 is the order of the incoming packets= , using D1 to D4 respectively). If DD bit for D4 is 1, does the NIC guarant= ee that the DD bit for D3, D2 and D1 will be 1? 2) There is a comment in the code that says 'Read desc statuses backwards t= o avoid race condition'. Can you elaborate on what is the race condition? I referred to [1] and I cannot find much information about this race condit= ion. Am I referring to the correct document? [1] Intel(r) Ethernet Controller X710/XXV710/XL710 Datasheet > >> > > > > >> > > > hope this is helpful. > >> > > > > >> > > > Regards > >> > > > Qi > >> > > > > >> > > > > -----Original Message----- > >> > > > > From: Gavin Hu (Arm Technology China) > >> > > > > [mailto:Gavin.Hu@arm.com] > >> > > > > Sent: Thursday, July 25, 2019 5:57 PM > >> > > > > To: Zhang, Qi Z ; Richardson, Bruce > >> > > > > ; Ananyev, Konstantin > >> > > > > > >> > > > > Cc: users@dpdk.org; Honnappa Nagarahalli > >> > > > > ; Phil Yang (Arm Technology > >> > > > > China) > >> > > > > Subject: i40e vPMD fix out of order Rx read issue > >> > > > > > >> > > > > Hi Qi, > >> > > > > > >> > > > > I am working on optimizing the i40e vPMD on aarch64 and I see > >> > > > > this patch relevant. > >> > > > > Could you illuminate what issue this patch was fixing? > >> > > > > I understand the PMD works at the driver layer, for delivery > >> > > > > of > >> > > > > L2 > >> > packets. > >> > > > > It does not own the responsibility to keep order(the > >> > > > > responsibility lies with > >> > > > the > >> > > > > protocol stack, like TCP)? > >> > > > > > >> > > > > http://patches.dpdk.org/patch/16665/ > >> > > > > > >> > > > > Best regards, > >> > > > > Gavin > >> > > > >