From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-bl2nam02on0083.outbound.protection.outlook.com [104.47.38.83]) by dpdk.org (Postfix) with ESMTP id 66A8EFE5 for ; Fri, 31 Mar 2017 21:36:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=Y6aGOul+TufslnXQ4ZFf0XjlOzWm2mgSPnG/enRi0+M=; b=NhR1X51KExfXskozFbectpfeSBDJlYHQWmhIcvz3YFTrkiUKlUH0i6jwzBxKIKWh9ZtYq1/ygYT9KYNlN8MUEi6Hv1e5T5YgDZffXKVRJ55wpDXmaRaIoaCPKfLpPrdUDYwRhx5+olTOAAInwhSlO+RJnaIdpI+EtPW8tv8KBvg= Authentication-Results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=caviumnetworks.com; Received: from localhost.localdomain (14.140.2.178) by BY1PR0701MB1723.namprd07.prod.outlook.com (10.162.111.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.991.14; Fri, 31 Mar 2017 19:36:30 +0000 From: Jerin Jacob To: dev@dpdk.org Cc: thomas.monjalon@6wind.com, bruce.richardson@intel.com, harry.van.haaren@intel.com, hemant.agrawal@nxp.com, gage.eads@intel.com, nipun.gupta@nxp.com, santosh.shukla@caviumnetworks.com, Jerin Jacob Date: Sat, 1 Apr 2017 01:04:41 +0530 Message-Id: <1490988905-12584-15-git-send-email-jerin.jacob@caviumnetworks.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1490988905-12584-1-git-send-email-jerin.jacob@caviumnetworks.com> References: <1488562101-6658-1-git-send-email-jerin.jacob@caviumnetworks.com> <1490988905-12584-1-git-send-email-jerin.jacob@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [14.140.2.178] X-ClientProxiedBy: MAXPR01CA0058.INDPRD01.PROD.OUTLOOK.COM (10.164.146.158) To BY1PR0701MB1723.namprd07.prod.outlook.com (10.162.111.142) X-MS-Office365-Filtering-Correlation-Id: 4712cc21-79ba-4577-b251-08d4786d3d06 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(201703131423075)(201703031133081); SRVR:BY1PR0701MB1723; X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1723; 3:CGlPsgXFPgBbUzv7ue6gtvn4KOux2G1odLeUhZC96jvjgLLlQBfzrZ7sVb+39CmVeMr36o7sk+/YdaCQgbHjHecmGEebHu+T4iNMxcr2SJxYsbeBnASVTNXTHuz9ezihA9V8g3Abi6G8pcVtLTOBoMBF/Wo2VXiUWYNXcnXtTVxUn2Q49kRX1aJAMMQ0K0Y2UMqdL1XO93w3aBUbtnx2pxsJFPvKVwpUggPhHUaks4n8dTHVvCcpTfkq8s5yYPuMYXL4qCb15Q0bM+fnoE70ew+MO/3dZOZ/ys+5ejSXAH8OlzripABVaTGeVrkrj5RyT+CdJqMXY/UXKI2/3qPxpg==; 25:wqkl0jArjiard9SIeFn9pH3yBik25h8J18xOMYK+KibRW1i9y6chDPss1SCvBrrOufXgdsg93O9Jy1arxfhyA08jl3yzCOCYwmP67+owinWxfF0HASmIeGjqnGfJKCxYjYIR0JJzYXDULlkCn/OVEmDKfbJtiDyvnd0Drb7rGnKlPA0dz8HyER8LIXFlTbQKAdC/p0dA6xfWK4i9c0blCABZqOSOgN+PH3Er+M4Ct7MbE4/vw3DeMwdFmw9oSMIuNBBeQDmpBgASieqDZprkosZGeThXcq7w7l7kZ2TuTHwWeei1Oi/iDdy0pQ69ZaDcvQlT03H6SubHFRA4YHZF/8jGpu88V6MFGku8/110fNX/bC3UgCALateXP8G8eILac+O3cUtJKrl4G9hwunZTtPVeAXpynuNUVh5keAiGaJyCmaoyKOCAO8/z8GhXfgkKce31Z5jAuoo9AwhdOeP+Hw== X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1723; 31:+RPT70Ic6mwIqPYxrFT28BsT0OBQZ9RyVye9WfObf/80yjHQ22a58PHEN94Sgvu6KuQNz5jycvhhI/Bw7HNj3e3nxxjEWtH80oWw6oEjkiGda26qXMor2OvFaERWnz+kgnyJcCBxXv/VQLW4WdX+bb9EuRUM7mH6QeXGljeZJkxHyFmaR6z3ZRhQmeWPuehknCi1bdriPnz0iAUT5IK2U8cyRkLmHI+jcsYtBiOUTzaIJl02doNEweyKPiCJcXMz; 20:6ZlO+w6NKfybJzu+8Z0SsCnKX8JMtDe6ZvbRv2J2MRJVaYSNSBo4NQZpWv04VE681RXnI61PbNYW3ch5Q7jmxXi9y05vEh+qeKJO3rU0fMb6mNehc8fmhXtx1z82QtazQ0DEhbLPCiLNNTRcVvMluloBcLf65e7gk3W8fYnyt0FAqGMO3Ld+ypdhJP/CszX7KYpPkQ2rFS89IBBsTyuZ6eruHD7oEtAd7Ddt2gx6p8C2zADiMWrW0DsekDSCVDa685/oLeTviShlFZ5I0BGjzpJrNkv+Cr7nHp1btMbgrNwneqivikyVGPcyRnOzJ6W8qIdnng8ht2Bm4pdGSch8Ka4M9cWeEMajYY70mHaqN1DAIPD/9MPgoKqS3tzC90WEUYMiHMS1yoLkkex25WvcvhcJODY3JakcaGYTzRUK1BGkh8fEHvp+1r2PSZuW77DhAu6gxf2EckpmDEcOFeLPFZt/SCGJXUpnON5H/By4NvOHBgF+N9gHEBzmx3pUr578f3+uJLFrs06MsH3CmS8G09O9tFgut6knHDp7mK9NZlONPTj9TQta1mNOj7Ejw5D6U/v/yltev+4+bX+NgYG1fjnVxwXJX3ll1gQa+UJPw+U= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(228905959029699); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040450)(601004)(2401047)(8121501046)(5005006)(10201501046)(93006095)(3002001)(6041248)(20161123555025)(20161123562025)(201703131423075)(201702281528075)(201703061421075)(20161123560025)(20161123564025)(6072148); SRVR:BY1PR0701MB1723; BCL:0; PCL:0; RULEID:; SRVR:BY1PR0701MB1723; X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1723; 4:vccKjDwBAk9SOvPfGqepokPezI5bMfVfDDQSHl5oUKS1f7idJqRImQewxL7qa+6uC4Tc8WdEffRa+DYpbrhPYDJ1O13142Gd27uai6y6gKtff9mDk3FDIdo0L0yAsJlAnEVAC5TRXUqRJGwey5ccIBazKiovkzups3nXNAtHcwsjLmy5aTg3YwwnBsm45ZkrS3bdmW21Mcz959wVvkjF1U2XlqiTxPLA0wpxyh5AnVDkuTHTcjxBoRoz/KvvuetUS3HfyG0fX8CBuKYGTysfVP44oIcV+DgR/GxHFXqH3FXB9TffDVOYO4297I57eDEFuAZV670ooQzuUJQkDa7N/u/A/W49TDJRJib4zljD5oWLW+H4rxJo5sV+nwrbZj5c8ORRmPGYmeZrp/t9PCfjFhzSSuKigtt3i/gvGatvjXcsjnFSQZ9WU1J+0jR1eCXx95KO/zygAiCDFHEGRWll3xoeENkovduyClKqKLCm5ubIIGHvly1dWI6MOPKlBLTnJKxUgnM1dJ2dDHT4D0tM3LN3SszZIJAVpgH+ts87+ytMPlJ6fsSsP+FKfzGKhTt1PPqOxO5wTZSXoqSmOHkUKwyIL3D8nqXLbTE+id93n+kE4srTSPD/SpwF7/wwa25jt2WPlzr3uQdUSMe4kHA65nURmQQANARyCu06qjC2ZRlWOfduv4UVmCjiiQHFXenfS6fZgDxeJo1md93BCPUH6mffKB96QxCOifJEpvXRuYyvWv9miyIUay7k235v/Tx3 X-Forefront-PRVS: 02638D901B X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6069001)(6009001)(39450400003)(39410400002)(39400400002)(39850400002)(39840400002)(38730400002)(8676002)(107886003)(110136004)(50466002)(189998001)(305945005)(6506006)(36756003)(50226002)(25786009)(48376002)(6666003)(8656002)(6486002)(7736002)(2950100002)(5660300001)(53936002)(81156014)(81166006)(3846002)(42882006)(6916009)(6116002)(6512007)(42186005)(33646002)(2361001)(2351001)(76176999)(50986999)(5003940100001)(47776003)(2906002)(5009440100003)(4326008)(66066001)(110426004)(309714004); DIR:OUT; SFP:1101; SCL:1; SRVR:BY1PR0701MB1723; H:localhost.localdomain; FPR:; SPF:None; MLV:nov; PTR:InfoNoRecords; LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BY1PR0701MB1723; 23:SDz1WfCH8+5GyYety7cCiz2p3xAGGFV7djSjfSQ?= =?us-ascii?Q?NzfZCAjXFzNWtDHYifmGOFaDpTEEq42peHO/3q0GnDrDjVZuyepX9MjerZT2?= =?us-ascii?Q?rWLgVDyJngyOMP84NfLRk6jVdhlCnAUIn9KcMlBAeAJmLP1809321qx+8FRm?= =?us-ascii?Q?VkAHraDf7AUKiIGi0bUN61JNHNg5IJG5tr7/YEyX/EdgI+YhmgC3lZhNIO07?= =?us-ascii?Q?Npx/qxHIKiR2XPYgME/ZKTgkZViF+CiQe65ieOGZ7Ip+FHx+jCIGlSsKOdqb?= =?us-ascii?Q?hlIswV8Djy3f73ykfAlzQn0zrGOMwPnS5Wwd//wn/DjPz3p4SPY66xRVUgTU?= =?us-ascii?Q?F3Bfy68fbXbOsuE9UY/WHbvSXPlGHgFf29zldW1PCpv3pm+6J28RPw9Q/5cf?= =?us-ascii?Q?ZvUsSjK3jSucXiKQAp/CfAwRSa9NPPZOqf7uORN+W0zlNPQhqm7weBRF++WW?= =?us-ascii?Q?0SZ6HGzDFgZMCgnUA5GFzunC7CRVGjupLlUxBkY+LbDsY/rWNs6EKZiKF1hu?= =?us-ascii?Q?CAJawkmigYU2MMmh3CiKebpUfbpGerqPxxSKDUhO7OnqNb+mSH/9nwy4W51/?= =?us-ascii?Q?9r2EUy00MBpw5DRDysqvEOI9DUnd+HA8BDkzsL8mRVpAJyh9ZUji3a3X/TRP?= =?us-ascii?Q?52BsZPTojxJoWifWpeIX08qFAvtgqhvngRie+kC8Dq6nt2wdUKsXa3euiF5b?= =?us-ascii?Q?u15sxGAabubtJWhDZ+unHvK1BHRiVbI48EGuUqPli0XsufPLdTNMmshEm5Nn?= =?us-ascii?Q?Jym6SCKjsE6ypIyoSJrzDUTz3PT1o/3xXXhNRpNKOCUv4jV9ywlMisfCqaq9?= =?us-ascii?Q?DR75gaM/d7ZS3b+6ZVctZlVcben9xP5Re0/wpQ1+4gEYR32fiReOWdeDSbEg?= =?us-ascii?Q?Px/32PGW7cme0nc5hhP51bL4OuaMUhUCFZ7UpymBC+b+1cSLW++cFUBbQMDH?= =?us-ascii?Q?Okvr3erj47c5YpVkjUKtiUZyxSigiGJvikRIb3fbPLw5mJ4Z4/Rk3Hp6qLkR?= =?us-ascii?Q?pC65yxB6/nykldQozLHIDPdISyVc/wqzIQkhTQt+QeNMnow9XIGrmC457oXq?= =?us-ascii?Q?7oxLBVjk9o7usm54GJb3Jlw9h6vVoItiDZvCYfojk/oP+bCbwwUqsyZIgrOu?= =?us-ascii?Q?hnoMdye22gw9AiU285bxXvf+9Ivvi5xoy3ebZUs/vZ5pmr1aSzrYIsbJ/RYu?= =?us-ascii?Q?KhMgxf+yJiC9mQHnhc7WQrY4SBVNdjJX2VOgm?= X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1723; 6:G+itpCohwZKFnQDC3NxdvJ09TV2Fm9W2O0cXZ81vgV6XzEVKMpiMQfT0xwY8VJ25V9Hy5IsRI/jUWpHRQCh9dC5Jnw3i2uGfRRU2e2mJO+CvBc91WbQZj29edkfabeW4qyUw9VVausjCp4sF1gF7l5m63J7b+0XX25Ns562NYVvIxeY5dXUwedC7IHgbW5OVwjnFXClqlbfyJmErqrT3b6WoEIsqMPLHdlgx//9yYKBw3tMmGW1fPbniis8Mdh3RrAYhnMMDtyXKXandz4lYimy6rP2MbYIXkXPq/SI1xKPi8fz8KEo0dr2asnZ1aZpurIN1EGr2C3wOVrOEiQPeNQ5Be5jsnH4YjCb74/5rjBvruy5T/EKFiWSs0eErGizXtKtdNtM0GUcuzR3eySwHyQ==; 5:4F3x8T4boKrgy7GASgtlfOnGxN1JvT5AQyfVLSa0W+8Ck53c8sEK80HrBTdcknScn7aUDn471ZKDezzyRvXliUsCoi1/s2aAnr1j9e3QQy8DtrNheef6XL2IBMe3S1FVVe2nouURpab+MHkouG0SzpDDie2Q8+KPJHru3E1oKcI=; 24:myi9gLDG5OoblzsmvD5MSoKWAS6Xc4Qxwclj69VTAV59idQPx7DjN+4sgffoCs1Zr1XoVq7z42rla5kFB6GzHf1Mt9XtQvUwaidPSHZ92cA= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1723; 7:cDzdl/2d7TOtjaUtM3Xkq3rddCRJghEucUAMPJzCNSohS+E2zwber9sddM+wo0L03it0sE4nnzwemdxO9NTzn1JkP5LR51DLSHm6+YACBfAnIrTAWNW9yFAovRq1sstrWdC+IxSA26surPDF2CCj+DzMFK5HyZ/8joOqGVlwPD8XR6J9GhAJdDCsWmZBITrX2x4HTnf5liYoh33Kvv6UML2EewZCQqENE9Utu+Ttclw662vfW2Rj1/TjnhKzsaK1f9yAPOo/z+HzBnCdJv4Nz2hzv1wJ2IAyy16d44cG7mE/X/facu8txoVmOw0dXJIqzR+t4rw5xKD/XYs5Vz7zow== X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Mar 2017 19:36:30.2138 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY1PR0701MB1723 Subject: [dpdk-dev] [PATCH v2 14/38] event/octeontx: add SSO HW device operations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Mar 2017 19:36:36 -0000 Signed-off-by: Jerin Jacob Signed-off-by: Santosh Shukla Acked-by: Gage Eads --- drivers/event/octeontx/Makefile | 5 ++ drivers/event/octeontx/ssovf_evdev.h | 30 ++++++++ drivers/event/octeontx/ssovf_worker.c | 124 ++++++++++++++++++++++++++++++++++ 3 files changed, 159 insertions(+) create mode 100644 drivers/event/octeontx/ssovf_worker.c diff --git a/drivers/event/octeontx/Makefile b/drivers/event/octeontx/Makefile index 6e8227b..c921303 100644 --- a/drivers/event/octeontx/Makefile +++ b/drivers/event/octeontx/Makefile @@ -46,9 +46,14 @@ LIBABIVER := 1 # # all source are stored in SRCS-y # +SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_worker.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_evdev.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_probe.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_mbox.c +ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y) +CFLAGS_ssovf_worker.o += -fno-prefetch-loop-arrays +endif +CFLAGS_ssovf_worker.o += -Ofast # install this header file SYMLINK-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF)-include := rte_pmd_octeontx_ssovf.h diff --git a/drivers/event/octeontx/ssovf_evdev.h b/drivers/event/octeontx/ssovf_evdev.h index 16bf3d5..16846b8 100644 --- a/drivers/event/octeontx/ssovf_evdev.h +++ b/drivers/event/octeontx/ssovf_evdev.h @@ -129,6 +129,36 @@ #define ssovf_read64 rte_read64_relaxed #define ssovf_write64 rte_write64_relaxed +/* ARM64 specific functions */ +#if defined(RTE_ARCH_ARM64) +#define ssovf_load_pair(val0, val1, addr) ({ \ + asm volatile( \ + "ldp %x[x0], %x[x1], [%x[p1]]" \ + :[x0]"=r"(val0), [x1]"=r"(val1) \ + :[p1]"r"(addr) \ + ); }) + +#define ssovf_store_pair(val0, val1, addr) ({ \ + asm volatile( \ + "stp %x[x0], %x[x1], [%x[p1]]" \ + ::[x0]"r"(val0), [x1]"r"(val1), [p1]"r"(addr) \ + ); }) +#else /* Un optimized functions for building on non arm64 arch */ + +#define ssovf_load_pair(val0, val1, addr) \ +do { \ + val0 = rte_read64(addr); \ + val1 = rte_read64(((uint8_t *)addr) + 8); \ +} while (0) + +#define ssovf_store_pair(val0, val1, addr) \ +do { \ + rte_write64(val0, addr); \ + rte_write64(val1, (((uint8_t *)addr) + 8)); \ +} while (0) +#endif + + struct ssovf_evdev { uint8_t max_event_queues; uint8_t max_event_ports; diff --git a/drivers/event/octeontx/ssovf_worker.c b/drivers/event/octeontx/ssovf_worker.c new file mode 100644 index 0000000..052d158 --- /dev/null +++ b/drivers/event/octeontx/ssovf_worker.c @@ -0,0 +1,124 @@ +/* + * BSD LICENSE + * + * Copyright (C) Cavium networks Ltd. 2017. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Cavium networks nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + + +#include + +#include "ssovf_evdev.h" + +enum { + SSO_SYNC_ORDERED, + SSO_SYNC_ATOMIC, + SSO_SYNC_UNTAGGED, + SSO_SYNC_EMPTY +}; + +#ifndef force_inline +#define force_inline inline __attribute__((always_inline)) +#endif + +#ifndef __hot +#define __hot __attribute__((hot)) +#endif + +/* SSO Operations */ + +static force_inline uint16_t +ssows_get_work(struct ssows *ws, struct rte_event *ev) +{ + uint64_t get_work0, get_work1; + uint64_t sched_type_queue; + + ssovf_load_pair(get_work0, get_work1, ws->getwork); + + sched_type_queue = (get_work0 >> 32) & 0xfff; + ws->cur_tt = sched_type_queue & 0x3; + ws->cur_grp = sched_type_queue >> 2; + sched_type_queue = sched_type_queue << 38; + + ev->event = sched_type_queue | (get_work0 & 0xffffffff); + ev->u64 = get_work1; + return !!get_work1; +} + +static force_inline void +ssows_add_work(struct ssows *ws, const uint64_t event_ptr, const uint32_t tag, + const uint8_t new_tt, const uint8_t grp) +{ + uint64_t add_work0; + + add_work0 = tag | ((uint64_t)(new_tt) << 32); + ssovf_store_pair(add_work0, event_ptr, ws->grps[grp]); +} + +static force_inline void +ssows_swtag_full(struct ssows *ws, const uint64_t event_ptr, const uint32_t tag, + const uint8_t new_tt, const uint8_t grp) +{ + uint64_t swtag_full0; + + swtag_full0 = tag | ((uint64_t)(new_tt & 0x3) << 32) | + ((uint64_t)grp << 34); + ssovf_store_pair(swtag_full0, event_ptr, (ws->base + + SSOW_VHWS_OP_SWTAG_FULL0)); +} + +static force_inline void +ssows_swtag_norm(struct ssows *ws, uint32_t tag, uint8_t new_tt) +{ + uint64_t val; + + val = tag | ((uint64_t)(new_tt & 0x3) << 32); + ssovf_write64(val, ws->base + SSOW_VHWS_OP_SWTAG_NORM); +} + +static force_inline void +ssows_swtag_untag(struct ssows *ws) +{ + ssovf_write64(0, ws->base + SSOW_VHWS_OP_SWTAG_UNTAG); + ws->cur_tt = SSO_SYNC_UNTAGGED; +} + +static force_inline void +ssows_desched(struct ssows *ws) +{ + ssovf_write64(0, ws->base + SSOW_VHWS_OP_DESCHED); +} + +static force_inline void +ssows_swtag_wait(struct ssows *ws) +{ + /* Wait for the SWTAG/SWTAG_FULL operation */ + while (ssovf_read64(ws->base + SSOW_VHWS_SWTP)) + ; +} + -- 2.5.5