From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 32375A32A4 for ; Fri, 25 Oct 2019 19:06:39 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 9709E1D418; Fri, 25 Oct 2019 19:06:38 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by dpdk.org (Postfix) with ESMTP id 5C19A1D412 for ; Fri, 25 Oct 2019 19:06:36 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x9PH1SoY010380; Fri, 25 Oct 2019 10:06:32 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=pfpt0818; bh=L/vmRnoxdfW/ATInlwstG7udkj3MgAnUTkbc3tYTGIc=; b=Bwjl6LDx/ffT0+wGhTv18szIi7xvx8uQ1sxrLZKaOA9zD2nGIDBGDET2HgD+uq1ymplQ udFu0Lo4RPlkUY4X/LiUJ7eJSU2GtIdGjDEVgZ8EF+hhVWNmU0DcidYbF+3NNZey+ZiT R0HBnOEYNicoOTRmKMQ5kqXq0YG9DS52QvgZd3C4bS47fYRGc33amDEZZPWndKBsXxFG xKa/fzjJbLrwvjYE3JLDttOvA61guyiaTv50/o70iJrZ9iH/lOfWcei0K3atQiqC7VBB 0dgaFY+CeIEGoTbzoPz5+lRUA9f9bcjW/GEZk4U+H14tATaW/lSOLq4apK/ePtyzrG7W yw== Received: from sc-exch04.marvell.com ([199.233.58.184]) by mx0a-0016f401.pphosted.com with ESMTP id 2vt9u5w8m6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Fri, 25 Oct 2019 10:06:32 -0700 Received: from SC-EXCH02.marvell.com (10.93.176.82) by SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Fri, 25 Oct 2019 10:06:31 -0700 Received: from NAM03-CO1-obe.outbound.protection.outlook.com (104.47.40.55) by SC-EXCH02.marvell.com (10.93.176.82) with Microsoft SMTP Server (TLS) id 15.0.1367.3 via Frontend Transport; Fri, 25 Oct 2019 10:06:30 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=f2jfGmKpxzj9kPgqfJZVFqr08nG4XyMj4mGeg0fSLkMFpN2fdazQ8+m8CQrOULdtlg7rb2JUJn2SyCxpm5J4Tx1qVI7RmveAn1+kA7+DGSTYmPRIdDskFAvn3o2V+7ZQlFZ6fdzUfa4hkSd9jm+sNFDjR3peC4q4mrN6SK+zrOspsGxwxU5h+EC0KWA+/xLzGbeStAp/qsbEHXfhV7zQpfIdN00LYX4jl0JkMhpCCFmcJ5NO1qJ7RzvoykHRRj4RnNhbZIpn+RlN3ryylWrv63WnqrK7DYq2u0G596ZFUVv9nNcw9BnZGp4nZfrz4H8t4QrUD835vITnD8FcKRk++w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=L/vmRnoxdfW/ATInlwstG7udkj3MgAnUTkbc3tYTGIc=; b=fOdx6ODeT5ysrldJ0lOaXDXgBreMPVhcy0poPYvn++3Kbs2tKVN1mVeNgy+sDdWcfsfzHtjRDkcqE9qzp8mHbMBys8i6DkO3zJaef9UK8M0IssltkJkvjXls1jCgd3DpUrLhRsvOH9/78Iv76bu5sVO8NDOnsHSRQCb7tyS1CyW/ixbmR5m9xrdvP/M0VcyqWzVWFUUZxjd5JmLPEAbwtyF7yQiOv0G8VUai9OhRABXg4HUMYM9HQ7lo4/04DdB1uHDoE5V2g358ANBKvNy2xvgnLkRyZJF9TcwWZbaPSg/FKNAbgUbcotFX8U0R8wd5dZVHohZUTUFoRD+SJ60CTg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com; dkim=pass header.d=marvell.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector2-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=L/vmRnoxdfW/ATInlwstG7udkj3MgAnUTkbc3tYTGIc=; b=aVeTzNL9nlHV8+iVLMJ8se6ReIB1CUj/KW1pehRx6dxA+NPg2dvMgsNa/088wWiDxIwDDwvz0Ma9V92wlNlMqiCfIlp6QAwF/EhUbHNzq740Oku2LEt6wja/oQuvFeFbpBkTFmnPq7YewIS1s2YUkiNlSEU86Yn9Rb0AfhsCbyo= Received: from CY4PR1801MB1863.namprd18.prod.outlook.com (10.171.255.160) by CY4PR1801MB2040.namprd18.prod.outlook.com (10.165.88.150) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2347.16; Fri, 25 Oct 2019 17:06:29 +0000 Received: from CY4PR1801MB1863.namprd18.prod.outlook.com ([fe80::d918:c54e:c3b1:3167]) by CY4PR1801MB1863.namprd18.prod.outlook.com ([fe80::d918:c54e:c3b1:3167%7]) with mapi id 15.20.2387.023; Fri, 25 Oct 2019 17:06:29 +0000 From: Pavan Nikhilesh Bhagavatula To: "Gavin Hu (Arm Technology China)" , "Jerin Jacob Kollanukkaran" CC: "dev@dpdk.org" , nd , nd Thread-Topic: [dpdk-dev] [PATCH] event/octeontx2: use wfe while waiting for head Thread-Index: AQHViby3nCGCIyXIT0CWL1WmagbrBqdp8tiAgADRIiCAAMyxgIAABXrQ Date: Fri, 25 Oct 2019 17:06:29 +0000 Message-ID: References: <20191023161244.3284-1-pbhagavatula@marvell.com> In-Reply-To: Accept-Language: en-IN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [122.171.232.163] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 2f894c78-8e53-4b3f-3c7c-08d7596dadd4 x-ms-traffictypediagnostic: CY4PR1801MB2040: x-ms-exchange-purlcount: 1 x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:439; x-forefront-prvs: 02015246A9 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(346002)(396003)(376002)(39860400002)(136003)(366004)(189003)(199004)(13464003)(71200400001)(66476007)(66946007)(66556008)(7736002)(305945005)(71190400001)(74316002)(76116006)(7696005)(81166006)(33656002)(81156014)(26005)(53546011)(6506007)(76176011)(316002)(14454004)(54906003)(102836004)(25786009)(8676002)(64756008)(110136005)(256004)(14444005)(45080400002)(66066001)(6636002)(446003)(186003)(8936002)(66446008)(99286004)(476003)(486006)(229853002)(11346002)(2906002)(478600001)(6116002)(3846002)(6306002)(9686003)(55016002)(86362001)(5660300002)(4326008)(52536014)(6246003)(6436002); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR1801MB2040; H:CY4PR1801MB1863.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: marvell.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 54PQqrBx0paJalIBkSrcFdVMPsXc9CAg1z5SEK4YUGl+o0pZAMUyOrjmGsdXvG5hMla1KR2/xyBYwwDXK8QrwKgF10WJmC0HzsZ6WrCrwzmMU9p5yxV5WEkf3AEknhj30vYpgSz0wYwAucHoqKxoBg8RADtzAZPlLaRrwvlgdekkVgf7vlYqpYPiIJLb7ry9NuE5939aPkc9olA1mTF6dT+gOSYL7h6enZICvYdefDyVnf/D3LGE7EO5FLmSgS/jm6Fab5jOmVv9ZO1vqUexDzcdZ5DJS9khBrMLmAdlqRl2TNKHtNSK2APpX8jD68RRVjR2lx98o0srf+Pbj+kRJqfV6BWiD4LWZCKGd3ZWIFBPJvj80A72j3m0BoGutxHsTZ14/nx31LpFggGqhzOV31XK6G+Lq+hYZBACse4EXuGGWP0pEpXZWQgaEBAIpwlzml3RA0XB6jovZ5+xqikD/hftCst/WLHnCRJQj5lfTJU= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 2f894c78-8e53-4b3f-3c7c-08d7596dadd4 X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Oct 2019 17:06:29.7610 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: FlsOFQ9zktcebuMWMrERVo0AQdniiVqRNKjSDLH+Bo/oQSyvbxsbMn7sch3B27IMDaGTH1v2RitcYb/tkanAet6u+EZ/Gv+6u6g2A4zkVAU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR1801MB2040 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.95,1.0.8 definitions=2019-10-25_09:2019-10-25,2019-10-25 signatures=0 Subject: Re: [dpdk-dev] [PATCH] event/octeontx2: use wfe while waiting for head X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Gavin, >Hi Pavan, > >> -----Original Message----- >> From: Pavan Nikhilesh Bhagavatula >> Sent: Friday, October 25, 2019 12:26 PM >> To: Gavin Hu (Arm Technology China) ; >> jerinj@marvell.com >> Cc: dev@dpdk.org; nd >> Subject: RE: [dpdk-dev] [PATCH] event/octeontx2: use wfe while >waiting for >> head >> >> Hi Gavin, >> >> >-----Original Message----- >> >From: dev On Behalf Of Gavin Hu (Arm >> >Technology China) >> >Sent: Thursday, October 24, 2019 9:23 PM >> >To: Pavan Nikhilesh Bhagavatula ; >Jerin >> >Jacob Kollanukkaran >> >Cc: dev@dpdk.org; nd >> >Subject: Re: [dpdk-dev] [PATCH] event/octeontx2: use wfe while >> >waiting for head >> > >> >Hi Pavan, >> > >> >> -----Original Message----- >> >> From: pbhagavatula@marvell.com >> >> Sent: Thursday, October 24, 2019 12:13 AM >> >> To: Gavin Hu (Arm Technology China) ; >> >> jerinj@marvell.com; Pavan Nikhilesh > >> >> Cc: dev@dpdk.org >> >> Subject: [dpdk-dev] [PATCH] event/octeontx2: use wfe while >waiting >> >for >> >> head >> >> >> >> From: Pavan Nikhilesh >> >> >> >> Use wfe to save power while waiting for tag to become head. >> >> >> >> SSO signals EVENTI to allow cores to exit from wfe when they >> >> are waiting for specific operations in which one of them is >> >> setting HEAD bit in GWS_TAG. >> >> >> >> Signed-off-by: Pavan Nikhilesh >> >> --- >> >> drivers/event/octeontx2/otx2_worker.h | 30 >> >++++++++++++++++++++++++-- >> >> - >> >> 1 file changed, 27 insertions(+), 3 deletions(-) >> >> >> >> diff --git a/drivers/event/octeontx2/otx2_worker.h >> >> b/drivers/event/octeontx2/otx2_worker.h >> >> index 4e971f27c..7a55caca5 100644 >> >> --- a/drivers/event/octeontx2/otx2_worker.h >> >> +++ b/drivers/event/octeontx2/otx2_worker.h >> >> @@ -226,10 +226,34 @@ otx2_ssogws_swtag_wait(struct >> >otx2_ssogws *ws) >> >> } >> >> >> >> static __rte_always_inline void >> >> -otx2_ssogws_head_wait(struct otx2_ssogws *ws, const uint8_t >> >wait_flag) >> >> +otx2_ssogws_head_wait(struct otx2_ssogws *ws) >> >> { >> >> - while (wait_flag && !(otx2_read64(ws->tag_op) & >> >BIT_ULL(35))) >> >> +#ifdef RTE_ARCH_ARM64 >> >> + uint64_t tag; >> >> + >> >> + asm volatile ( >> >> + " ldr %[tag], [%[tag_op]] \n" >> >"ldxr" should be used, exclusive-load is required to "monitor" the >> >location, then a write to the location will cause clear of the exclusiv= e >> >monitor, thus a wake up event is generated implicitly. >> >> As I have mentioned in the commit log: >> "SSO signals EVENTI to allow cores to exit from wfe when they >> are waiting for specific operations in which one of them is >> setting HEAD bit in GWS_TAG." >If you have other expected wake up sources, that is ok. Just curious is >this signal explicitly sent to quit WFE? AFAIK yes, explicitly sent to quit WFE. >Just wondering, implicit event(Clear of exclusive monitor) vs explicit >signal, which has shorter latency? Not really sure but SSO has dedicated bus inside each core. >/Gavin >> >> The address need not be tracked by the global monitor. >> >> >You can find more explanation is here: >> >https://urldefense.proofpoint.com/v2/url?u=3Dhttp- >> >>3A__inbox.dpdk.org_dev_AM0PR08MB5363F9D1BA158B66B803EA068 >F >> >6B0- >> >>40AM0PR08MB5363.eurprd08.prod.outlook.com_&d=3DDwIFAg&c=3DnKj >W >> >>ec2b6R0mOyPaz7xtfQ&r=3D1cjuAHrGh745jHNmj2fD85sUMIJ2IPIDsIJzo6 >F >> >N6Z0&m=3DJMzT-4V2megNsFYxaO0V2wE0- >> >GlK9UPUvE1K0pPA9aQ&s=3DJajU2VklhV_jFE0WKAZ076KjjWymIC- >> >iTiJXU0Vwxr4&e=3D >> >/Gavin >> >> + " tbnz %[tag], 35, done%=3D >> > \n" >> >> + " sevl \n" >> >> + "rty%=3D: wfe \n" >> >> + " ldr %[tag], [%[tag_op]] \n" >> >> + " tbz %[tag], 35, rty%=3D \n" >> >> + "done%=3D: \n" >> >> + : [tag] "=3D&r" (tag) >> >> + : [tag_op] "r" (ws->tag_op) >> >> + ); >> >> +#else >> >> + /* Wait for the HEAD to be set */ >> >> + while (!(otx2_read64(ws->tag_op) & BIT_ULL(35))) >> >> ; >> >> +#endif >> >> +} >> >> + >> >> +static __rte_always_inline void >> >> +otx2_ssogws_order(struct otx2_ssogws *ws, const uint8_t >> >wait_flag) >> >> +{ >> >> + if (wait_flag) >> >> + otx2_ssogws_head_wait(ws); >> >> >> >> rte_cio_wmb(); >> >What ordering does this barrier try to keep? If there is a write then >wait >> >for kind of response, should this barrier move before >> >otx2_ssogws_head_wait? >> >> The barrier is used to flush out write buffer to LLC (octeontx2 point of >> coherence) so >> that NIX Tx picks up all the modifications done to the packet. >Looking at the otx2_ssogws_event_tx function, so far at the point of >rte_cio_wmb, only the header is written? >Should it be delayed after the whole packet written and before the >submission? We only care that the writes to the actual packet buffer ex. Start of ether= net header are committed. The rest of mbuf fields are translated into a HW command after the barrier = and written=20 to a LMTLINE using ldoer. >If NIX is not falling within the SMP configuration, should it be >rte_io_wmb instead? Octeontx2 has only single shareability domain i.e. it makes no distinction = between=20 Outer and inner sharable domains.=20 Since all IO devices are interpreted to be on outer sharable domain, we lik= e to use=20 rte_cio_(r/w)mb for IO devices.=20 >/Gavin Regards, Pavan. >> >> } >> >> @@ -258,7 +282,7 @@ otx2_ssogws_event_tx(struct otx2_ssogws >> >*ws, >> >> struct rte_event ev[], >> >> >> >> /* Perform header writes before barrier for TSO */ >> >> otx2_nix_xmit_prepare_tso(m, flags); >> >> - otx2_ssogws_head_wait(ws, !ev->sched_type); >> >> + otx2_ssogws_order(ws, !ev->sched_type); >> >> otx2_ssogws_prepare_pkt(txq, m, cmd, flags); >> >> >> >> if (flags & NIX_TX_MULTI_SEG_F) { >> >> -- >> >> 2.17.1