From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 184CBA034F; Tue, 12 May 2020 23:22:59 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1040B1BFA4; Tue, 12 May 2020 23:22:58 +0200 (CEST) Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-eopbgr150057.outbound.protection.outlook.com [40.107.15.57]) by dpdk.org (Postfix) with ESMTP id 49F8A1BF0B for ; Tue, 12 May 2020 23:22:57 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=01h2Q+iS77W37ugUfWvMSi20jSHlMbz4fReidQYNq3M=; b=5Nwu6qUJMmYApbJ9pbbceK7REKRVdv2NwVFiE+Ms0cUaK4UF4vGtcRMZBGqQHNLUdWvuEeG1yrg7X5f/3SdmklrD54I5CK6EejXqXFsUwfSUe29VIBH1hOx6j9Ri5DBSPkgbr6mOhgzDlLaJA1IU76/Z4EfqKdfNWjNEZ/NjQlo= Received: from DB8P191CA0007.EURP191.PROD.OUTLOOK.COM (2603:10a6:10:130::17) by DBBPR08MB4664.eurprd08.prod.outlook.com (2603:10a6:10:f1::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2979.33; Tue, 12 May 2020 21:22:53 +0000 Received: from DB5EUR03FT037.eop-EUR03.prod.protection.outlook.com (2603:10a6:10:130:cafe::48) by DB8P191CA0007.outlook.office365.com (2603:10a6:10:130::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3000.20 via Frontend Transport; Tue, 12 May 2020 21:22:53 +0000 Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=bestguesspass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DB5EUR03FT037.mail.protection.outlook.com (10.152.20.215) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2979.27 via Frontend Transport; Tue, 12 May 2020 21:22:53 +0000 Received: ("Tessian outbound 5abcb386707e:v54"); Tue, 12 May 2020 21:22:53 +0000 X-CR-MTA-TID: 64aa7808 Received: from 7e1b776a21a6.3 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 24640A92-79F6-402E-9B8E-2BD467C697A9.1; Tue, 12 May 2020 21:22:48 +0000 Received: from EUR05-AM6-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 7e1b776a21a6.3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Tue, 12 May 2020 21:22:48 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ER+vLpRAZTQvoDCn+RNZyIZrTXG87OQsa+fht6XNlPz/BvwedAU0XtrWgHsO6IwPTNH58PztD4QoyIitBg46gzT7Jfbv9qjinPvAwCXoy+rJTTFrttEpVWa4yDMdX5bcD1sLWkgHpMsTHg+4dEJMzgIW74ASKyZXq7LbNwnUA6aRUzx2mG2ZzXlcqDdVPxIrPkKA43BvAEchZmChpuIxzwmLFja3ke6KeC9tfe7Q8m0BczuWwXVsOTd4DNHLlkV/FtQshfasaDK/l9JziM1pfy8xuRNg49MLVlw1eaVU6bEdHVKL1jBLW9YsUHYRBAfIQW4qSUKYBQKZZM+XDIc0DA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=01h2Q+iS77W37ugUfWvMSi20jSHlMbz4fReidQYNq3M=; b=VWrJ8ZtcdkS/MGeMh4Eam2sog6ydw9xX+b2So8VKFuisa7TJRJkbim04DtlrsZ37sXTqGODhn7WTNjWRyZAY5pebWsuzYStR7/WXrcHBEvvycl4m4PRw9dxDri+QjCDlcvQXfoBc7HGGbkCLRVprWtLG/SKJJp92hO3512fY77qLMBqj0SMLOi1Z11G+9FQanXQVWLFIWokVFXAApygl+L4F0xJfh20iGFuPRc3b3csPP2iPzVfqKJdmSWB3VptVMaNeUkjI74Vpcam/SezQgojcbLSim6Bayi99RZhwDj+LEkd/DopzeUvsDbjAG+lfOGWE0F/AsvsIOkQKNHI+2w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=01h2Q+iS77W37ugUfWvMSi20jSHlMbz4fReidQYNq3M=; b=5Nwu6qUJMmYApbJ9pbbceK7REKRVdv2NwVFiE+Ms0cUaK4UF4vGtcRMZBGqQHNLUdWvuEeG1yrg7X5f/3SdmklrD54I5CK6EejXqXFsUwfSUe29VIBH1hOx6j9Ri5DBSPkgbr6mOhgzDlLaJA1IU76/Z4EfqKdfNWjNEZ/NjQlo= Received: from DBBPR08MB4646.eurprd08.prod.outlook.com (2603:10a6:10:f5::16) by DBBPR08MB4903.eurprd08.prod.outlook.com (2603:10a6:10:df::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3000.20; Tue, 12 May 2020 21:22:45 +0000 Received: from DBBPR08MB4646.eurprd08.prod.outlook.com ([fe80::1870:afc4:b90f:609d]) by DBBPR08MB4646.eurprd08.prod.outlook.com ([fe80::1870:afc4:b90f:609d%5]) with mapi id 15.20.3000.016; Tue, 12 May 2020 21:22:45 +0000 From: Honnappa Nagarahalli To: Gavin Hu , "Chauskin, Igor" , "dev@dpdk.org" CC: nd , "david.marchand@redhat.com" , "thomas@monjalon.net" , "mk@semihalf.com" , "Tzalik, Guy" , "Schmeilin, Evgeny" , "mw@semihalf.com" , Ruifeng Wang , Phil Yang , Joyce Kong , "Bshara, Saeed" , "Matushevsky, Alexander" , Bruce Richardson , nd , Honnappa Nagarahalli , nd Thread-Topic: [PATCH RFC v1 0/7] relax barriers for ENA PMD and small fixes Thread-Index: AQHV/DIVvhCOm2Xh00q9xrfG3A7dKqh76lLggAd/BICAId2AwA== Date: Tue, 12 May 2020 21:22:45 +0000 Message-ID: References: <20200313091835.58039-1-gavin.hu@arm.com> <10e90cbc7bbe4d84979dabcb0fbe0c41@EX13D12EUA003.ant.amazon.com> <73990a21da324ef5ac759227017682fd@EX13D12EUA003.ant.amazon.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: ebaa2028-e3df-4e4d-adbd-13a456bfc17e.0 x-checkrecipientchecked: true Authentication-Results-Original: arm.com; dkim=none (message not signed) header.d=none;arm.com; dmarc=none action=none header.from=arm.com; x-originating-ip: [70.113.25.165] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: de8baeb7-1385-4ebb-5f71-08d7f6baa1f4 x-ms-traffictypediagnostic: DBBPR08MB4903:|DBBPR08MB4664: x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:9508;OLM:9508; x-forefront-prvs: 0401647B7F X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: 8zMxzY6x0ndaa283jvvauGykYqzeSoe6mlxCM77J8RctMJazaWSJLFF+mwJ327wQVJjoz+Vo8vNtqvjqH3UU+KMCzUD+MMIycJnkBU+NmRO3hr44H4XtPZcsnuiYj2PbBCHaiq3ShgRS6wKjRmF8az80xciKg8Nfkad60JQVn4/5i9HePcEz8Iy6awjooiT+xtGRW8E3VpR5yxWRNEScTZ/YRIpH2TS8E4Ql4CupERSMLO2J4rOMCWQyxUnpA8Bsvo3rzxPEf1ky/Io8AQKmhxRNsMre+f86Qi+fUy3qHLnjY14Dj/OETGa/RwCNjv7Z7pso/JIBbIOqJau3mv/QPaudx2yxPv+Drimoyw4quvW7K8N1eGpZvZ2HGpg+/1fwlPCnFunVXdDycAVEbJcypFhCOsLxbbKJHnsyplJPOt1OlAXWT+Ef5Qhq4UUiRlXCnVWvQamMchUNM9wM70IBXg3U9/FBa9LKp8piKwh7l83MoT6lhTaiYUJNoTBDUSNVqTGbsXbHkDP9qiePDfsvdpHUQXmO22dzQKPnmGgFfc+Be+W33o5SzKZtj0/h3U70hKAnmP8X7WbSMnTEIDevK7H7rXUIUaYWB4StLxeIQlcbHn+qdHO+uCJeTQCaxBUX X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBBPR08MB4646.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(376002)(39840400004)(396003)(366004)(136003)(346002)(33430700001)(6506007)(66446008)(26005)(64756008)(66556008)(186003)(66476007)(52536014)(110136005)(54906003)(316002)(53546011)(71200400001)(55016002)(7416002)(86362001)(33440700001)(7696005)(5660300002)(478600001)(33656002)(4326008)(76116006)(966005)(2906002)(9686003)(66946007)(8676002)(8936002)(21314003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: 5TBcTxSmwi1iSURObYTorn4mqAWAjtqn/MvKQ3eYl89U6GUwLNo7NmJRqY+s+wLJ3TYj3+WC63JYhpdRWGLfTHyF9KUCYxBqb2Gq8txSNFIVJqwDTdVmEFuaLLQxbGuEw5DMdyuOHVWQRsvFi8+S7lv2iog9nQBdTsVnkdljO4CbmML+0WhklfHEd1emGMa9REjXjwIBgHENbxI/gtqJ+3u6UkhXD7vA3kmu7JrLEJKPLpyzJBOieXd3emq6tmd/uTQFqV3/snFUAUN2vOoZJ8Qgkb+njRgS2WgCZmIYHLmvLbm+QtB0rEle8wA4BY3VaVy1p2/EUarWpcut+0pLfyOTepC2AMsaMTUzjylDI+uDBg+KfRhsX+Awwt5ki23C6mVKQeFZfS9nDX65NfKNwc3eqvHClyQjPGDfkE5Q7rbow5NoGlQYndgJcXXnCpNebm8VtAnWHrfVkEQIDblXvQhEZEN3lUkEkC0sNAOBwWJfB2IbktueESAWGj+Y65Q/ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB4903 Original-Authentication-Results: arm.com; dkim=none (message not signed) header.d=none;arm.com; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5EUR03FT037.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFTY:; SFS:(4636009)(376002)(39840400004)(136003)(346002)(396003)(46966005)(33430700001)(70586007)(356005)(316002)(966005)(110136005)(53546011)(26005)(5660300002)(47076004)(478600001)(6506007)(7696005)(9686003)(86362001)(52536014)(33440700001)(55016002)(336012)(70206006)(8936002)(81166007)(54906003)(4326008)(2906002)(33656002)(8676002)(82310400002)(186003)(21314003); DIR:OUT; SFP:1101; X-MS-Office365-Filtering-Correlation-Id-Prvs: 6f2f5a0d-e80f-4ac9-987b-08d7f6ba9cea X-Forefront-PRVS: 0401647B7F X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Bm6U+nVKIOxk1wkJyiQ6S2p5Zd4iYLatJ2mdWT9Cwhd3uZUqY8j/Sd3susW1WbfLjmjlYMLKYVWU6m8S0nh3Q6Ffva08NGs3NB1E6uwsfvfLuGnamc9q0z5Vlbd1/1ix/BASdl8iu4wqldnEALNuWgWkvOIHX0EvBQPb/8l6bXWzflg9QMtxZ+sZqA9EBnScgSEZuXNOOvnIMD6Oe3THbelNkdDXEmekvm9Hi4co8V86im5+UEXytUOTVtIekmss5TIcUkwqCn3pnoyECqsT5dfqAu0luhzMYZ8IMXMycVP5TRflJrr7g/22w8/+gE+4ZCzKAYK1oPdTIVhBC3isszFu5aPwjUIp0Cdb/7u3ieEPWCwY1Z/FE0OS4uKFSX8C0lsxeLXTnk8aA4QckBsZ75gldbUYgsI3v+KiSJ30ViNXIFdkO2fPu0d8nDNc0JLViXmO8LCN9DLWlukv9jW8Rsf01AsXz7V8nZV+6SwEAir12YhHmbTaWpfbTgdSBqJTzy+qmu08D1p+DXDv7VwE+QjMHVTS5EOweBQAnADReDXHgOr3eT421pldNOv+awk2VfnbKDqQ1qLPTeK6OBle5dj9rfqN5kG+bMLxROc8shdHOiz4YiThOs+F8qTRWzdK++u74mZYklkQXjamI8o4kV0oRh0UsWadiXAhO1TFMiLCpytFD8/qIztoxx4tbd83 X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 May 2020 21:22:53.7631 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: de8baeb7-1385-4ebb-5f71-08d7f6baa1f4 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB4664 Subject: Re: [dpdk-dev] [PATCH RFC v1 0/7] relax barriers for ENA PMD and small fixes X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Igor, Few comments inline. > > Subject: RE: [PATCH RFC v1 0/7] relax barriers for ENA PMD and small > > fixes > > > > Hi all, > > > > Please see the first batch of comments related to these patches: > > > > 1. Relaxing the register read/write isn't always good enough. > > Specifically, when barriers are required between different memory > > types, the reordering can occur even on x86. Yet in DPDK the > > io/cio/smp barrier flavors for x86 are defined as compiler-only > > barriers, which is not enough in cases involving different memory > > types. In ENA driver, when LLQ is on, there is a regular register > > memory access before the barrier and write-combined memory access after > the barrier. > That's makes sense, we realized that also, we don't mean to change x86 > behaviors. I think https://patches.dpdk.org/patch/70091/ should help in this. I have a= dded to that patch, please take a look. > > > > We're working on a more extensive change that will include the > > optimizations proposed for barriers relaxing while making them > > applicable to all platforms. > We are working also on an extensive change, helpful to arm64 while not > impacting x86. > More testing is ongoing. > > > > 2. Regarding the changes for statistics logging - the patch relies on > > c11 features. I'm not sure it's acceptable for all situations since > > we've already encountered a reports when even c99-compliant changes > > caused compilation issues. > C11 is already widely used in other components, even in other projects li= ke > vpp and ovs. > Maybe it comes to time to drop C99 as a stringent requirement. Do you have any particular compiler version in mind? There is a proposal [1= ] to stop using rte_atomicNN_xxx APIs and use wrappers built around c11 __a= tomic built-ins. C11 is supported in GCC from 4.7 and in clang from 3.1. [1] https://patches.dpdk.org/cover/70097/ > > > > 3. Removing redundant zeroing of sub-struct - we're currently working > > on some extensive changes to the Tx flow, which will include this > > change among other. > Ok, thanks. > > > > Thanks, > > Igor > > > > -----Original Message----- > > From: Gavin Hu > > Sent: Tuesday, March 17, 2020 9:59 AM > > To: Chauskin, Igor ; dev@dpdk.org > > Cc: nd ; david.marchand@redhat.com; > thomas@monjalon.net; > > mk@semihalf.com; Tzalik, Guy ; Schmeilin, Evgeny > > ; mw@semihalf.com; Honnappa Nagarahalli > > ; Ruifeng Wang > ; > > Phil Yang ; Joyce Kong ; > > Bshara, Saeed ; Matushevsky, Alexander > > ; Bruce Richardson ; > nd > > > > Subject: RE: [EXTERNAL] [PATCH RFC v1 0/7] relax barriers for ENA PMD > > and small fixes > > > > CAUTION: This email originated from outside of the organization. Do > > not click links or open attachments unless you can confirm the sender > > and know the content is safe. > > > > > > > > Hi Igor, > > > > > -----Original Message----- > > > From: Chauskin, Igor > > > Sent: Monday, March 16, 2020 5:35 PM > > > To: Gavin Hu ; dev@dpdk.org > > > Cc: nd ; david.marchand@redhat.com; > > thomas@monjalon.net; > > > mk@semihalf.com; Tzalik, Guy ; Schmeilin, Evgeny > > > ; mw@semihalf.com; Honnappa Nagarahalli > > > ; Ruifeng Wang > > ; > > > Phil Yang ; Joyce Kong ; > > > Bshara, Saeed ; Matushevsky, Alexander > > > > > > Subject: RE: [PATCH RFC v1 0/7] relax barriers for ENA PMD and small > > > fixes > > > > > > Hi Gavin, > > > > > > Thank you for the contribution. > > > Please do not merge these changes (patches 0..7) till we (the ENA > > > team) properly review and ack/nack. > > > These changes can potentially provide performance improvement, yet > > > we need to ensure they are applicable for all possible scenarios. > > > Specifically, the behavior on x86 platforms is likely to be different= . > > > What testing have you done for these patches? Was x86 tested? > > As noted in the cover letter, these patches were not tested as we > > don't have ENA NICs. > > We rely on you to do that, any concerns and comments welcome. > > Yes, the behavior on x86 platforms is also different, Intel people are > > welcome to comment. > > /Gavin > > > > > > Thanks, > > > Igor > > > > > > -----Original Message----- > > > From: Gavin Hu > > > Sent: Friday, March 13, 2020 11:18 AM > > > To: dev@dpdk.org > > > Cc: nd@arm.com; david.marchand@redhat.com; thomas@monjalon.net; > > > mk@semihalf.com; Tzalik, Guy ; Schmeilin, Evgeny > > > ; Chauskin, Igor ; > > > mw@semihalf.com; Honnappa.Nagarahalli@arm.com; > > ruifeng.wang@arm.com; > > > phil.yang@arm.com; joyce.kong@arm.com > > > Subject: [EXTERNAL][PATCH RFC v1 0/7] relax barriers for ENA PMD and > > > small fixes > > > > > > CAUTION: This email originated from outside of the organization. Do > > > not click links or open attachments unless you can confirm the > > > sender and know the content is safe. > > > > > > > > > > > > To ensure the stores to the host memory are observed by NIC HW > > > before a door bell ring to the NIC HW and the HW starts actions, for > > > example, doing DMA, a barrier is required on weak memory ordering > > > platforms, like > > aarch64. > > > > > > However, unnecessarily too strong barriers like 'dsb' on aarch64 > > > will dampen performance. > > > > > > In a typical doorbell use case, as NIC and CPU are in the outer > > > sharable domain, a lighter weight 'dmb osh' barrier is sufficient. > > > > > > The patch set relaxes the barriers in similar places and include one > > > more patch for statistics logging with relaxed ordering and the > > > other patch removing duplicate memset. > > > > > > Note this set is submitted for RFC as we don't have physical ENA > > > NICs in the lab and the patch set was not verified nor benchmarked. > > > > > > Gavin Hu (7): > > > net/ena: remove duplicate barrier > > > net/ena: relax the barrier for doorbell ring > > > net/ena: relax the rmb for DMA > > > net/ena: relax barrier for completion queue update > > > net/ena: relax the barrier for bounce buffer > > > net/ena: use c11 atomic for statistics > > > net/ena: remove duplicate memset > > > > > > drivers/net/ena/base/ena_eth_com.c | 2 +- > > > drivers/net/ena/base/ena_eth_com.h | 6 ++-- > > > drivers/net/ena/base/ena_plat_dpdk.h | 2 +- > > > drivers/net/ena/ena_ethdev.c | 46 +++++++++++++++++---------= -- > > > drivers/net/ena/ena_ethdev.h | 8 ++--- > > > 5 files changed, 38 insertions(+), 26 deletions(-) > > > > > > -- > > > 2.17.1 >=20