From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id C04E9A0613 for ; Mon, 26 Aug 2019 17:55:53 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1D0A85B3E; Mon, 26 Aug 2019 17:55:53 +0200 (CEST) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-eopbgr140044.outbound.protection.outlook.com [40.107.14.44]) by dpdk.org (Postfix) with ESMTP id 6CF502C19 for ; Mon, 26 Aug 2019 17:55:51 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kxUk6nwbCl10HoWwOzi+P1KYfYjjRsXVjsy+OsGuTlk=; b=XNON/xs0tOLMQeWny/5xuIYEypLP3FJfygXW0EUcoBldNfNvcXMh6K5sI1l7347l3VOinLUhQ5TWKqdoTgE5m+qL0XBxiO4IskIpiaVn8PPjP7+ACKHMN+0a6WRr0ttPABvD3yRu2SWmU4OZrgQplYflnDbVE8irQ0YWVUahe2o= Received: from VI1PR08CA0135.eurprd08.prod.outlook.com (2603:10a6:800:d5::13) by VI1PR0802MB2605.eurprd08.prod.outlook.com (2603:10a6:800:b0::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2199.21; Mon, 26 Aug 2019 15:55:48 +0000 Received: from VE1EUR03FT046.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e09::208) by VI1PR08CA0135.outlook.office365.com (2603:10a6:800:d5::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2199.14 via Frontend Transport; Mon, 26 Aug 2019 15:55:48 +0000 Authentication-Results: spf=temperror (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=temperror action=none header.from=arm.com; Received-SPF: TempError (protection.outlook.com: error in processing during lookup of arm.com: DNS Timeout) Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by VE1EUR03FT046.mail.protection.outlook.com (10.152.19.226) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2199.13 via Frontend Transport; Mon, 26 Aug 2019 15:55:47 +0000 Received: ("Tessian outbound 8b8b6ad907d7:v27"); Mon, 26 Aug 2019 15:55:45 +0000 X-CR-MTA-TID: 64aa7808 Received: from 3e958650caa6.1 (ip-172-16-0-2.eu-west-1.compute.internal [104.47.2.58]) by 64aa7808-outbound-1.mta.getcheckrecipient.com id D5605A97-4EAF-4A4C-9632-3D29C13FE2AD.1; Mon, 26 Aug 2019 15:55:39 +0000 Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01lp2058.outbound.protection.outlook.com [104.47.2.58]) by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 3e958650caa6.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384); Mon, 26 Aug 2019 15:55:39 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ysnt9Us283gEv9m1bENXMVgdg/S5tvM1z1eFC5AdghQ+eqjPCRgUGTq9nFDeLjwVpuF6oVq/VjrxSiuEMdEVjApO2uYNdGSKRAtcdko4rDbVTd3El1j1pDsYHGFTz81rrLsx1KL7HsOU2yWB5aladIYAaZawkDL/GuLQ3jYrJGcuNIk8UeN55oWBHiPI4y1R6RlyVdTv6AWkQhaaSIWiNkrj596zQ89Ano3U6YmC4TgWGYP79poWF4SEdtVE97RGZzeiGsA2KErUHAWXOApuS1+DHDbnlzruq+XZgNiSwWGYOHStzhW7byJOFLK3eHrXQjCfo23/1O87oqrAYq6wPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kxUk6nwbCl10HoWwOzi+P1KYfYjjRsXVjsy+OsGuTlk=; b=exlO/7RbxN8VQ2tGkfJEapOzsPbHMD9l7ftHPx8bYLADXb+pjfGdk9FKlEuQszIpzUTO96k2fYZALQxsZ5e8UXWH/eQo5yUgkbyrxGgtW4bfLE4l00GKt15KRtXtO+hBZ2zJoZxHXnLy5re1E2CK6H5P0kk4+rJqVK7rLExG/Med4EehswITXTUL3fAtXEPO5Lpm3ABCCSdi1g12E5CzXFZGCIA3WAQfi4tNdIPdD8mkNkix4jbYTX6trID92lCyeZZWeKCL6fQxqYw911EMf84qp8oCCjhGozvboHxhpsqdd8nupeZDGjMNpv32P0YR8jEbPq6Mkw8zMnJfYrervA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kxUk6nwbCl10HoWwOzi+P1KYfYjjRsXVjsy+OsGuTlk=; b=XNON/xs0tOLMQeWny/5xuIYEypLP3FJfygXW0EUcoBldNfNvcXMh6K5sI1l7347l3VOinLUhQ5TWKqdoTgE5m+qL0XBxiO4IskIpiaVn8PPjP7+ACKHMN+0a6WRr0ttPABvD3yRu2SWmU4OZrgQplYflnDbVE8irQ0YWVUahe2o= Received: from VE1PR08MB5149.eurprd08.prod.outlook.com (20.179.30.152) by VE1PR08MB4672.eurprd08.prod.outlook.com (10.255.115.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2178.19; Mon, 26 Aug 2019 15:55:37 +0000 Received: from VE1PR08MB5149.eurprd08.prod.outlook.com ([fe80::a8af:a9b8:4597:4128]) by VE1PR08MB5149.eurprd08.prod.outlook.com ([fe80::a8af:a9b8:4597:4128%3]) with mapi id 15.20.2178.020; Mon, 26 Aug 2019 15:55:37 +0000 From: Honnappa Nagarahalli To: Ye Xiaolong , "Zhang, Qi Z" CC: "Gavin Hu (Arm Technology China)" , "Richardson, Bruce" , "Ananyev, Konstantin" , "users@dpdk.org" , "Phil Yang (Arm Technology China)" , "Ruifeng Wang (Arm Technology China)" , nd Thread-Topic: i40e vPMD fix out of order Rx read issue Thread-Index: AdVCzJhDNbgBl9FATkWplfpJPawhKgAEGCLwAC59RaAAIKjSIAOMKbFQAAGo47AAYyTYgAH9EtTwABSBLUA= Date: Mon, 26 Aug 2019 15:55:37 +0000 Message-ID: References: <039ED4275CED7440929022BC67E7061153D71548@SHSMSX105.ccr.corp.intel.com> <039ED4275CED7440929022BC67E7061153D7485E@SHSMSX105.ccr.corp.intel.com> <039ED4275CED7440929022BC67E7061153D7D657@SHSMSX105.ccr.corp.intel.com> <20190816025058.GA53345@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: d18a290b-4ed3-40f0-bfa0-00bdcf08e220.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; x-originating-ip: [217.140.111.135] x-ms-publictraffictype: Email X-MS-Office365-Filtering-Correlation-Id: 9aea1c8c-282d-4130-f603-08d72a3ddc3b X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam-Untrusted: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600166)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020); SRVR:VE1PR08MB4672; X-MS-TrafficTypeDiagnostic: VE1PR08MB4672:|VI1PR0802MB2605: X-MS-Exchange-PUrlCount: 5 x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true x-ms-oob-tlc-oobclassifiers: OLM:10000;OLM:10000; x-forefront-prvs: 01415BB535 X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(366004)(376002)(346002)(396003)(136003)(189003)(199004)(13464003)(86362001)(7736002)(54906003)(966005)(6116002)(3846002)(8676002)(316002)(6306002)(186003)(81156014)(81166006)(76176011)(53936002)(55016002)(110136005)(74316002)(52536014)(305945005)(66066001)(9686003)(7696005)(5660300002)(99286004)(76116006)(64756008)(66556008)(25786009)(66476007)(66946007)(2906002)(6246003)(14444005)(4326008)(33656002)(256004)(8936002)(478600001)(2940100002)(102836004)(53546011)(26005)(446003)(229853002)(6506007)(71190400001)(14454004)(486006)(66446008)(6436002)(11346002)(476003)(71200400001); DIR:OUT; SFP:1101; SCL:1; SRVR:VE1PR08MB4672; H:VE1PR08MB5149.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Message-Info-Original: zzxcncMsCoGZwQBzkNsA4ZFml5cxj8Yc0soXVeDoZ2I2bCIpGHiaCDSvFXVORP0jnH9tmB7kRKyS6nqmLh2sRkmxuoOsTCoM8m0hOT2297Hy34d/vPlwct3Pslcpi148fqa9yjojD6vuBwcuNpBxqBvxVR0AGSLvAIY97CpQzuyGV47WnMkqbd0tycDT0j74VrZO3WIYv5C3D0kxdCOvRma2MLKMVbO2cM9RWnbeWfObI9/VybVx5/PuFoCFv0TPXizmaynNc5jzYBHwqqstdFT/Byct06rBNJcLGk0m1yGNWk67fk0xS5CyvTEtyfdC4Y9JXBFWb4sjEUzen+C1O72QXyXbQt3tPLh0D9ES+R8rhAQd8iZFXWyiwEdV7sE6j5LGisvdtEtWaqEX6GSFQJCssrt3qlCkJR7nlWYnQNs= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1PR08MB4672 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: VE1EUR03FT046.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; IPV:CAL; SCL:-1; CTRY:IE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(346002)(136003)(39860400002)(376002)(396003)(2980300002)(13464003)(199004)(189003)(126002)(26005)(476003)(25786009)(102836004)(966005)(336012)(446003)(110136005)(50466002)(76176011)(2906002)(486006)(6246003)(97756001)(36906005)(7736002)(23726003)(2940100002)(63350400001)(316002)(53546011)(63370400001)(478600001)(26826003)(229853002)(11346002)(14454004)(99286004)(4326008)(54906003)(74316002)(52536014)(305945005)(14444005)(6116002)(70206006)(356004)(22756006)(66066001)(55016002)(6506007)(8676002)(5660300002)(81166006)(81156014)(86362001)(9686003)(186003)(6306002)(7696005)(8936002)(3846002)(33656002)(8746002)(47776003)(70586007)(46406003)(76130400001); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR0802MB2605; H:64aa7808-outbound-1.mta.getcheckrecipient.com; FPR:; SPF:TempError; LANG:en; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; A:1; MX:1; X-MS-Office365-Filtering-Correlation-Id-Prvs: ba3869e8-c6ce-48df-36f4-08d72a3dd679 X-Microsoft-Antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600166)(710020)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020); SRVR:VI1PR0802MB2605; NoDisclaimer: True X-Forefront-PRVS: 01415BB535 X-Microsoft-Antispam-Message-Info: 1QJLhEWPUCFWKyvfki07b4pk53yYpxwuGTKcs+4XXrX7pdzyYT5pXBumStbZzDCgfcenfgTEAh6DCIVpUhhqshs8MJAbFdiRSFDokZwqWYY+JYkvHDNVSX+cXD+qnCH2xlyyUAD+kFk4OBknFLvCxXLstqK4g3hRGAIj8zazj2Zi37zg116FPyCtSA3InvldhWVsNX46YcngUj4PqiPMKncQnxOwzPbieDWG5EWLUq8mxpQ3F+vPYjtgFOcsT+EQEsJQxiWOWf8UT84SjcKwlm31TQX2alrb17TvTUV95F2hiz9vuSMBjHO27nQKDAXY2EsaDTx8MKtt9JDHT5hZAqtFUIRlhzjUxEDvbAPs1xYpw7dppf2D30LL2vI1cKoyJkC7dvW6j+ppQJqdpiGsfWFKoOrOVipyxsqKVE6xHps= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2019 15:55:47.1042 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9aea1c8c-282d-4130-f603-08d72a3ddc3b X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0802MB2605 Subject: Re: [dpdk-users] i40e vPMD fix out of order Rx read issue X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" > >=20 > Hi Qi, > Thank you so much for your explanation. I am trying to review this > patch and I have further questions on the NIC behavior. >=20 > > >> > > Subject: RE: i40e vPMD fix out of order Rx read issue > > >> > > > > >> > > Hi Qi, > > >> > > > > >> > > Thanks for your explanation! > > >> > > I did some testing and found the barriers caused a big drop in > > >> > > RFC2544 NDR performance on aarch64, how about it on X86? > > >> > For x86, I think we don't have performance drop, the memory > > >> > barrier just change the compiler's behavior to avoid generate out > > >> > of order read instructions, and x86 guarantee no out of read > > >> > execution, so it does not add new instructions that cost CPU cycle= s. > > >> > > > >> > > Is it possible to count DD bits in a way of surviving across > > >> > > the out-of-order descriptors reading? > > >> > > > >> > I think it is possible, but this will impact performance on x86, > > >> > but for aarch64, you can try out to see if that benefit and do > > >> > proper optimization on related vPMD implementation. > > >> The patches for aarch64 are in community review, and performances > > >> gain were measured both for ixgbe and i40e vPMD. Very appreciate if > > >> you can further review and comment. > > >> Here are the patch links: > > >> IXGBE vPMD: > > >> http://patches.dpdk.org/patch/57649/ > > >> http://patches.dpdk.org/patch/57650/ > > >> I40e vPMD: > > >> http://patches.dpdk.org/cover/57651/ > > >> http://patches.dpdk.org/patch/57652/ > > > > > > > > > > > >> > > > > >> > > Best Regards, > > >> > > Gavin > > >> > > > > >> > > > -----Original Message----- > > >> > > > From: Zhang, Qi Z > > >> > > > Sent: Thursday, July 25, 2019 8:11 PM > > >> > > > To: Gavin Hu (Arm Technology China) ; > > >> > > > Richardson, Bruce ; Ananyev, > > >> > > > Konstantin > > >> > > > Cc: users@dpdk.org; Honnappa Nagarahalli > > >> > > > ; Phil Yang (Arm Technology > > >> > > > China) > > >> > > > Subject: RE: i40e vPMD fix out of order Rx read issue > > >> > > > > > >> > > > Hi Gavin: > > >> > > > > > >> > > > in vPMD, we read 4 or 8 packets as batch, we count DD bits > > >> > > > for packet received, but not check the if they are continues > > >> > > > or not, we assume it should always be 1000, 1100, 1110, 1111 > > >> > > > ....(take batch size is 4 as > > >> > > > example) while the out of order read instruction generated by > > >> > > > compiler will cause driver to get un-continues DD bits, like > > >> > > > 1011, the descriptor on the hole actually is invalid since > > >> > > > when it is read , descriptor is not write back yet, but we > > >> > > > still process this as 1110, it cause > > >> > an > > >> > > corrupted mbuf returned. > 1) vPMD reads the descriptors in the reverse order. Let us number them D4= , > D3, D2, D1 for this discussion (1 to 4 is the order of the incoming packe= ts, > using D1 to D4 respectively). If DD bit for D4 is 1, does the NIC guarant= ee > that the DD bit for D3, D2 and D1 will be 1? The reason I ask this question is: looking at the non-vector PMD, function = i40e_rx_scan_hw_ring, the code counts the number of descriptors with DD bit= set to 1. However, it does not check if the descriptors with DD bit set to= 1 are contiguous. This makes me believe that NIC will ensure that DD bits = are set in order. i.e. in the example above (assuming we are reading the de= scriptors in order) if DD bit for D4 is set to 1, it is guaranteed that DD = bits for D3, D2, D1 are also set. > 2) There is a comment in the code that says 'Read desc statuses backwards= to > avoid race condition'. Can you elaborate on what is the race condition? > I referred to [1] and I cannot find much information about this race cond= ition. > Am I referring to the correct document? >=20 > [1] Intel(r) Ethernet Controller X710/XXV710/XL710 Datasheet >=20 > > >> > > > > > >> > > > hope this is helpful. > > >> > > > > > >> > > > Regards > > >> > > > Qi > > >> > > > > > >> > > > > -----Original Message----- > > >> > > > > From: Gavin Hu (Arm Technology China) > > >> > > > > [mailto:Gavin.Hu@arm.com] > > >> > > > > Sent: Thursday, July 25, 2019 5:57 PM > > >> > > > > To: Zhang, Qi Z ; Richardson, Bruce > > >> > > > > ; Ananyev, Konstantin > > >> > > > > > > >> > > > > Cc: users@dpdk.org; Honnappa Nagarahalli > > >> > > > > ; Phil Yang (Arm Technology > > >> > > > > China) > > >> > > > > Subject: i40e vPMD fix out of order Rx read issue > > >> > > > > > > >> > > > > Hi Qi, > > >> > > > > > > >> > > > > I am working on optimizing the i40e vPMD on aarch64 and I > > >> > > > > see this patch relevant. > > >> > > > > Could you illuminate what issue this patch was fixing? > > >> > > > > I understand the PMD works at the driver layer, for > > >> > > > > delivery of > > >> > > > > L2 > > >> > packets. > > >> > > > > It does not own the responsibility to keep order(the > > >> > > > > responsibility lies with > > >> > > > the > > >> > > > > protocol stack, like TCP)? > > >> > > > > > > >> > > > > http://patches.dpdk.org/patch/16665/ > > >> > > > > > > >> > > > > Best regards, > > >> > > > > Gavin > > >> > > > > >