From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 66157A32A4 for ; Fri, 25 Oct 2019 18:34:42 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D41301DFE6; Fri, 25 Oct 2019 18:34:40 +0200 (CEST) Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-eopbgr130043.outbound.protection.outlook.com [40.107.13.43]) by dpdk.org (Postfix) with ESMTP id 310661D44E for ; Fri, 25 Oct 2019 18:34:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=i+4Ljb+jVo+Ant12hJa9ax1rSh77pJNqZufwomLHCm4=; b=4+MgjvTJh2Xj2cPaX3zFhLhPKmRjOCL4neERUYhMGV6Q3e6Wn84wiHkR+Pao3Kf55pOC03AAQVplLcMYhF0prqoE1nSptelbTwOCdE/TEGqsK4jsyKO4MAnt5sJsMftklBljxpWzue+gsHAlIixEFQ5G1mXRv+ulvQgJdUG7G8g= Received: from AM4PR08CA0071.eurprd08.prod.outlook.com (2603:10a6:205:2::42) by DBBPR08MB4840.eurprd08.prod.outlook.com (2603:10a6:10:dd::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2347.18; Fri, 25 Oct 2019 16:34:37 +0000 Received: from AM5EUR03FT052.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e08::200) by AM4PR08CA0071.outlook.office365.com (2603:10a6:205:2::42) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2387.22 via Frontend Transport; Fri, 25 Oct 2019 16:34:37 +0000 Authentication-Results: spf=fail (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=none action=none header.from=arm.com; Received-SPF: Fail (protection.outlook.com: domain of arm.com does not designate 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT052.mail.protection.outlook.com (10.152.17.161) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2387.20 via Frontend Transport; Fri, 25 Oct 2019 16:34:37 +0000 Received: ("Tessian outbound 0cf06bf5c60e:v33"); Fri, 25 Oct 2019 16:34:36 +0000 X-CR-MTA-TID: 64aa7808 Received: from d3799afd0375.1 (cr-mta-lb-1.cr-mta-net [104.47.5.52]) by 64aa7808-outbound-1.mta.getcheckrecipient.com id AAD4A10F-59ED-4E8A-A754-55202E61C2C0.1; Fri, 25 Oct 2019 16:34:31 +0000 Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-he1eur02lp2052.outbound.protection.outlook.com [104.47.5.52]) by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id d3799afd0375.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384); Fri, 25 Oct 2019 16:34:31 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QGSKcwQjcvaklJ+CFFtv2RuEHJFYo+vRCM0t4vC/PphjHdci6ZsNoXKxScpFjZWVjXf5iQIFGg99aTRtzgT+E0DlLI05l7CGgoMmrGKPgBCsYQ+biPy6+6KvHzPsRmXKCkb4CRRkjyyox1wIlEfTqQDedaRP35KjUT7olh9EgwVNuZR2Qmc98BAqdGzTJoH5hTF6ZX+9kQyulkHwNtu2BJklvZq7C0iR4yw2U4YNAJvsOC2JhzQNCim3R9eVsYNzkEyIyW7ahmFtWpJzkaGzqKwEmBa9EtxOXzYDlR1Umgh71U+74JqOu+PIkyG8Qtk6MndKnm/GcyZN05M+bNHtRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=i+4Ljb+jVo+Ant12hJa9ax1rSh77pJNqZufwomLHCm4=; b=RcN2llOAEbEIwakQCgmiwL6q+BW7lsB5JfXqYEJZ3HJJoQETdg64Slr+A62yWV1kXYStY7U2lWfNDjgn6M/p7jz1vD+OtZ6dxgeS7UJoJByHj2yOn87uYPrqyHI3KC3ZKxRa/PXo11/qhOuioOdlpNCqkTBZ3IDvlqZbcy1mElM7Ax4SXj0YOmva6g5RwFqhcxYBRgymGaV5Vaiqt1qDgeJxHSFReaTknVySW/NYZxOWE9F+sMBcH0CIXVLZcsHHyCn0p0zamKNMLCohVQ0Y+S2YST1d9m09eSp9RXWTgenislAPr4EsPsNtbKNSmpqcK+3GFEjLgvVgm/NRFkf98g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=i+4Ljb+jVo+Ant12hJa9ax1rSh77pJNqZufwomLHCm4=; b=4+MgjvTJh2Xj2cPaX3zFhLhPKmRjOCL4neERUYhMGV6Q3e6Wn84wiHkR+Pao3Kf55pOC03AAQVplLcMYhF0prqoE1nSptelbTwOCdE/TEGqsK4jsyKO4MAnt5sJsMftklBljxpWzue+gsHAlIixEFQ5G1mXRv+ulvQgJdUG7G8g= Received: from AM0PR08MB5363.eurprd08.prod.outlook.com (52.132.214.213) by AM0PR08MB3684.eurprd08.prod.outlook.com (20.178.23.210) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2347.21; Fri, 25 Oct 2019 16:34:29 +0000 Received: from AM0PR08MB5363.eurprd08.prod.outlook.com ([fe80::b483:301f:e382:a94d]) by AM0PR08MB5363.eurprd08.prod.outlook.com ([fe80::b483:301f:e382:a94d%6]) with mapi id 15.20.2387.021; Fri, 25 Oct 2019 16:34:29 +0000 From: "Gavin Hu (Arm Technology China)" To: Pavan Nikhilesh Bhagavatula , "jerinj@marvell.com" CC: "dev@dpdk.org" , nd , nd Thread-Topic: [dpdk-dev] [PATCH] event/octeontx2: use wfe while waiting for head Thread-Index: AQHViby5JmiY9ClsQ02w7Dx6nQWH86dp7+FQgADVVACAALwsIA== Date: Fri, 25 Oct 2019 16:34:29 +0000 Message-ID: References: <20191023161244.3284-1-pbhagavatula@marvell.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 2528da9d-76b5-45b2-b4a0-894e3c75afbd.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Gavin.Hu@arm.com; x-originating-ip: [113.29.88.7] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: d97ee7fb-aca4-49ec-6c2e-08d7596939b1 X-MS-TrafficTypeDiagnostic: AM0PR08MB3684:|DBBPR08MB4840: X-MS-Exchange-PUrlCount: 1 x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true x-ms-oob-tlc-oobclassifiers: OLM:227;OLM:227; x-forefront-prvs: 02015246A9 X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(376002)(136003)(366004)(396003)(346002)(13464003)(199004)(189003)(14454004)(6436002)(55236004)(11346002)(476003)(71200400001)(71190400001)(102836004)(53546011)(478600001)(74316002)(2906002)(7736002)(305945005)(256004)(446003)(14444005)(6506007)(5660300002)(86362001)(186003)(486006)(45080400002)(52536014)(25786009)(3846002)(6116002)(26005)(110136005)(54906003)(316002)(66446008)(76116006)(76176011)(64756008)(66556008)(7696005)(33656002)(99286004)(66946007)(8676002)(4326008)(81166006)(2501003)(8936002)(6246003)(81156014)(229853002)(66066001)(9686003)(55016002)(66476007)(6306002); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR08MB3684; H:AM0PR08MB5363.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: 16WhGXoSmyChO6Sz0SVrXWGSATjnoeqel+iTjtogS4jpAXuRJUQJh+asb8YM7rwCuAICHWeOgxbkh+D1/mOqvdsb9uRdvOS1qfPD8E3ycR6ocpXgxuhs/greKWQHjlMZMZJDy8ezG1iWyQxvJO2CiHTdHMjK5IJeF90OyvAtEcp2F9koxFNRt8ny89V3+5O9TNqYKvpAfYMpnxmyWtXn9yjFHtzK2v9PkfgJ9F/GnVi6H6RB3Be8y2/z+2OLFJdJXAcvefj1BTJgUe5e9PAlg+JpUlKFfGXmZzWrt+lZcMiF+t+kjrWjk4IAUYtNB3kCHypC5EWEx7lxil0wlxfl4QplWnRAVa10rqrzGZTHV92vaDtnT/IuKSgj99bBitRO9FlV0yKdIE2AB/whFgeWAPJNHnr8nT5Tib73u16xkMNlcrHB1RadoPnZuqS3asnqihjH/WMJa0Vq/fZSNAnOTz+0rDeFYr5D46LQ0PIzltA= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR08MB3684 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Gavin.Hu@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT052.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; IPV:CAL; SCL:-1; CTRY:IE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(346002)(136003)(396003)(39860400002)(376002)(1110001)(339900001)(189003)(199004)(13464003)(476003)(25786009)(6246003)(305945005)(52536014)(99286004)(97756001)(55016002)(6306002)(23726003)(50466002)(74316002)(7736002)(33656002)(14444005)(47776003)(46406003)(8676002)(86362001)(229853002)(8936002)(8746002)(356004)(5660300002)(2906002)(81156014)(81166006)(4326008)(14454004)(316002)(54906003)(110136005)(36906005)(76130400001)(102836004)(26826003)(76176011)(478600001)(7696005)(22756006)(446003)(11346002)(126002)(6116002)(3846002)(26005)(70586007)(66066001)(70206006)(2501003)(186003)(486006)(53546011)(6506007)(336012)(105606002)(45080400002)(9686003); DIR:OUT; SFP:1101; SCL:1; SRVR:DBBPR08MB4840; H:64aa7808-outbound-1.mta.getcheckrecipient.com; FPR:; SPF:Fail; LANG:en; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; MX:1; A:1; X-MS-Office365-Filtering-Correlation-Id-Prvs: ae2ccbb7-f767-4c7d-d6d0-08d759693556 NoDisclaimer: True X-Forefront-PRVS: 02015246A9 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qyYzv0ETOo1OxxsRM+iUmKLJGFs/kccNzkb/A8T1RvH85PHO1wCu1nRZSRF5s+7KCgxa7SJu+8h9Ekwj/swcP8EUXk5c2fqQpAFTvpn8qctBdOMWemwDGancnuFyC9zUcCwz35YC2F/wNgFt4vLEEALOK3f/ZZPnU3Fx6O5dkAff6Ee1zPL/sdzG9xQhwGT9vN0seS7I5hRrE0CN9cV6PvQ2NYAsLAisQKf/zBSyO5kQ9sxKTkaZkqXWvriX6ttGoIA6czFz/Tt6TUjgIB898R/mpZjeERpMylYZdgo4ZEnAbo8EQx6yf0TAR7D085pXT09O5JsztkyCVzFRfASRH/zq8glQkyG2UW8Uur6fYz8OFrawNEVjVEvQFXNnsHYsX3LFdGydmHr3xQSCpT5GY2jRxvHXz/7pPLKYp5cS8wYnWOyhPE56YrxG1ZSJULbWzSY3wWVqTKVi3FilM+JMgs4sYVHB2+A8eLwzw4+IJNo= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Oct 2019 16:34:37.0001 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d97ee7fb-aca4-49ec-6c2e-08d7596939b1 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB4840 Subject: Re: [dpdk-dev] [PATCH] event/octeontx2: use wfe while waiting for head X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Pavan, > -----Original Message----- > From: Pavan Nikhilesh Bhagavatula > Sent: Friday, October 25, 2019 12:26 PM > To: Gavin Hu (Arm Technology China) ; > jerinj@marvell.com > Cc: dev@dpdk.org; nd > Subject: RE: [dpdk-dev] [PATCH] event/octeontx2: use wfe while waiting fo= r > head >=20 > Hi Gavin, >=20 > >-----Original Message----- > >From: dev On Behalf Of Gavin Hu (Arm > >Technology China) > >Sent: Thursday, October 24, 2019 9:23 PM > >To: Pavan Nikhilesh Bhagavatula ; Jerin > >Jacob Kollanukkaran > >Cc: dev@dpdk.org; nd > >Subject: Re: [dpdk-dev] [PATCH] event/octeontx2: use wfe while > >waiting for head > > > >Hi Pavan, > > > >> -----Original Message----- > >> From: pbhagavatula@marvell.com > >> Sent: Thursday, October 24, 2019 12:13 AM > >> To: Gavin Hu (Arm Technology China) ; > >> jerinj@marvell.com; Pavan Nikhilesh > >> Cc: dev@dpdk.org > >> Subject: [dpdk-dev] [PATCH] event/octeontx2: use wfe while waiting > >for > >> head > >> > >> From: Pavan Nikhilesh > >> > >> Use wfe to save power while waiting for tag to become head. > >> > >> SSO signals EVENTI to allow cores to exit from wfe when they > >> are waiting for specific operations in which one of them is > >> setting HEAD bit in GWS_TAG. > >> > >> Signed-off-by: Pavan Nikhilesh > >> --- > >> drivers/event/octeontx2/otx2_worker.h | 30 > >++++++++++++++++++++++++-- > >> - > >> 1 file changed, 27 insertions(+), 3 deletions(-) > >> > >> diff --git a/drivers/event/octeontx2/otx2_worker.h > >> b/drivers/event/octeontx2/otx2_worker.h > >> index 4e971f27c..7a55caca5 100644 > >> --- a/drivers/event/octeontx2/otx2_worker.h > >> +++ b/drivers/event/octeontx2/otx2_worker.h > >> @@ -226,10 +226,34 @@ otx2_ssogws_swtag_wait(struct > >otx2_ssogws *ws) > >> } > >> > >> static __rte_always_inline void > >> -otx2_ssogws_head_wait(struct otx2_ssogws *ws, const uint8_t > >wait_flag) > >> +otx2_ssogws_head_wait(struct otx2_ssogws *ws) > >> { > >> - while (wait_flag && !(otx2_read64(ws->tag_op) & > >BIT_ULL(35))) > >> +#ifdef RTE_ARCH_ARM64 > >> + uint64_t tag; > >> + > >> + asm volatile ( > >> + " ldr %[tag], [%[tag_op]] \n" > >"ldxr" should be used, exclusive-load is required to "monitor" the > >location, then a write to the location will cause clear of the exclusive > >monitor, thus a wake up event is generated implicitly. >=20 > As I have mentioned in the commit log: > "SSO signals EVENTI to allow cores to exit from wfe when they > are waiting for specific operations in which one of them is > setting HEAD bit in GWS_TAG." If you have other expected wake up sources, that is ok. Just curious is thi= s signal explicitly sent to quit WFE?=20 Just wondering, implicit event(Clear of exclusive monitor) vs explicit sign= al, which has shorter latency? /Gavin >=20 > The address need not be tracked by the global monitor. >=20 > >You can find more explanation is here: > >https://urldefense.proofpoint.com/v2/url?u=3Dhttp- > >3A__inbox.dpdk.org_dev_AM0PR08MB5363F9D1BA158B66B803EA068F > >6B0- > >40AM0PR08MB5363.eurprd08.prod.outlook.com_&d=3DDwIFAg&c=3DnKjW > >ec2b6R0mOyPaz7xtfQ&r=3D1cjuAHrGh745jHNmj2fD85sUMIJ2IPIDsIJzo6F > >N6Z0&m=3DJMzT-4V2megNsFYxaO0V2wE0- > >GlK9UPUvE1K0pPA9aQ&s=3DJajU2VklhV_jFE0WKAZ076KjjWymIC- > >iTiJXU0Vwxr4&e=3D > >/Gavin > >> + " tbnz %[tag], 35, done%=3D > > \n" > >> + " sevl \n" > >> + "rty%=3D: wfe \n" > >> + " ldr %[tag], [%[tag_op]] \n" > >> + " tbz %[tag], 35, rty%=3D \n" > >> + "done%=3D: \n" > >> + : [tag] "=3D&r" (tag) > >> + : [tag_op] "r" (ws->tag_op) > >> + ); > >> +#else > >> + /* Wait for the HEAD to be set */ > >> + while (!(otx2_read64(ws->tag_op) & BIT_ULL(35))) > >> ; > >> +#endif > >> +} > >> + > >> +static __rte_always_inline void > >> +otx2_ssogws_order(struct otx2_ssogws *ws, const uint8_t > >wait_flag) > >> +{ > >> + if (wait_flag) > >> + otx2_ssogws_head_wait(ws); > >> > >> rte_cio_wmb(); > >What ordering does this barrier try to keep? If there is a write then w= ait > >for kind of response, should this barrier move before > >otx2_ssogws_head_wait? >=20 > The barrier is used to flush out write buffer to LLC (octeontx2 point of > coherence) so > that NIX Tx picks up all the modifications done to the packet. Looking at the otx2_ssogws_event_tx function, so far at the point of rte_ci= o_wmb, only the header is written? Should it be delayed after the whole packet written and before the submissi= on?=20 If NIX is not falling within the SMP configuration, should it be rte_io_wmb= instead? /Gavin > >> } > >> @@ -258,7 +282,7 @@ otx2_ssogws_event_tx(struct otx2_ssogws > >*ws, > >> struct rte_event ev[], > >> > >> /* Perform header writes before barrier for TSO */ > >> otx2_nix_xmit_prepare_tso(m, flags); > >> - otx2_ssogws_head_wait(ws, !ev->sched_type); > >> + otx2_ssogws_order(ws, !ev->sched_type); > >> otx2_ssogws_prepare_pkt(txq, m, cmd, flags); > >> > >> if (flags & NIX_TX_MULTI_SEG_F) { > >> -- > >> 2.17.1