From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-BN3-obe.outbound.protection.outlook.com (mail-bn3nam01on0081.outbound.protection.outlook.com [104.47.33.81]) by dpdk.org (Postfix) with ESMTP id 64C0DF97C for ; Fri, 3 Mar 2017 18:30:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=sFYWo5T3XuAGvTx2G121N/9FEhOgYIyM031WEf7tFsU=; b=lGqzrJDjyql/ySiiV10BRGMC5L2Q5pKpRZsez0NkdBiz/qSV6OSKsA/F1wd1zthLRMqNjnvYh5KdPqTy3ufUHhPvd5cU78TMUvj9XzS24/ZBk1oe/EToQ5LAWc2OkhpzVWbuT/HO/UeeDdx2G7/oLqBljomIW96miiqHHdCCor0= Authentication-Results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=caviumnetworks.com; Received: from localhost.localdomain.localdomain (14.140.2.178) by BN3PR0701MB1719.namprd07.prod.outlook.com (10.163.39.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.933.12; Fri, 3 Mar 2017 17:30:01 +0000 From: Jerin Jacob To: dev@dpdk.org Cc: thomas.monjalon@6wind.com, bruce.richardson@intel.com, harry.van.haaren@intel.com, hemant.agrawal@nxp.com, gage.eads@intel.com, nipun.gupta@nxp.com, santosh.shukla@caviumnetworks.com, Jerin Jacob Date: Fri, 3 Mar 2017 22:57:59 +0530 Message-Id: <1488562101-6658-18-git-send-email-jerin.jacob@caviumnetworks.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1488562101-6658-1-git-send-email-jerin.jacob@caviumnetworks.com> References: <1488562101-6658-1-git-send-email-jerin.jacob@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [14.140.2.178] X-ClientProxiedBy: BMXPR01CA0021.INDPRD01.PROD.OUTLOOK.COM (10.174.214.159) To BN3PR0701MB1719.namprd07.prod.outlook.com (10.163.39.18) X-MS-Office365-Filtering-Correlation-Id: 2b378ab7-e626-4b5f-8dde-08d4625aedfd X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001); SRVR:BN3PR0701MB1719; X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1719; 3:sEXIi+nat2S9gX/WNDniV9BesbxzvJz6KlA9TEbRVmHhD7dc6lwlecCxBP7Fk54ZUG12dFlj4jAErbFRU09Yxqe2R4uXXwNHKYSTznXvMDpukaYhJNf+6M9fFtvMgpTbS/hQrS9MR+cEiL0xiwydD3xdR8pFU/3EmUXRYy0NZXrnUjgALIVOthcrLH7lpwgRwSgxDd0VUItIi24SqJ6pw5m4jextEUKi+boFJrNMX8+47eIAr/EDxZK1yDZrBNtTqUTonysE9XYhL7HoQ4f+cA==; 25:FKKKxU5bPTpJ3wcH0SDIviFlLVzl+2xGKmXa39bZE2eQ4CVzDVP4EW6EIR0oJV++e0Lkywt3ar0bF3R+P+E+gc2B/Xrxvr9egmEs0JY6nVM6FqFD5Tp9eX7ArrnYgFlTY7ZSmkZuxtvbBBIGTi7HDXuX5qvSqLIlRAFd0PaVEzRxpCpGuQxanKcj4wguc+HgFt0apL85nhPeb+bW/y45Unf4RC6qjjdvhZiSyZ5vGecmgEcoZfaUGkqOuHvm7dgeKHen2oSxIVxXbHCapciD6kCN0vB8hTy0WKn09QU8Iak4iTjrEFygmAR+BDV+XA/4l5BdrxcOe+3pMPlGzf4+LyCWDW9+LbmZ7mWy2fYi1Ct1+bxzPQFvF3uFU2tfnx5IIbWJLemiDPM9dxGyqcRQlkocNbE6v/u0XNw+gwnPPsHaFEYFKgigs1qitNnql97aliPWXqS+pqxE628R79lPxA== X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1719; 31:lsD8IBgTt5EgiSMTSJln2m0nD/ZBAVBAro//ukN9g5rTBFGY5zl0BVOOQRiuogKv9IM6SV+7FxJT0j2I+Wijca6m9EUL69rUa54DTIBZgoGg/K2yoOl3UudiYvuVvFUfP457iyJt/Ja6/He6aHOLToUdkch0VEeG0/roe9oKlPdjrqLVuZrKQcjNYeCq92vxWwaXsxXXQ6x9unBr5LTtyMMXiuMRZOCLxpPQNqmlQFp6yESx++5IUd46BE3B2Nr9; 20:Nb9QatLKtoPffy8PHNtn+A7+uGyG0HU018KZkkcxMYUA6d7tEtRlCEUbTePajyTsv5bRftjccs72NppcX+6olOdZYqVBCJBFlIkbbax1HQjW5utRlgT2Lh856EDkFGrH9dV64juxoV2M090iVVqnbvEO2HfuklYkojn/C0vu3+5IDIzZI2Pj/oeNw9hQIr2dwlybHX9KLrEwhC5v+0mZTmrJMpP+odC5sNuF4jpTxoZoWPa+5TbzQK9j0uKM+5+ptxw7twezlUGGIIwfjcdjCd+TbURtbRWVpvzlIk9uvpt8635Su394iQIYq7cQH3+WL4T26FxNJvR/dubQODQ7Yz7HejqqXj1sqHAeMXlmHi/3evhoS/mtVl9whLNAWXsLjCCkWaiz86JJL8xqw0HnebeTepwKhr9IQWn3l8l27zh5cLiE3gQdGbxLc2jOoK2nzvqEXRJrVHTWz4+X/Vg9NsPe6tnrGDfx9JB1NTXGlXuBTJYK4vQiweOGlEFRjBcuR06w6JotvW/0qHFH00ko7LNEiNUrkc++eobkVDGDGcvNv97yeZlbTboSUxGt90FfP6DOymd4wfrZyeTOmO2ZTsCuQEoLlkH5zLB7fsbQeHM= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6041248)(20161123555025)(20161123560025)(20161123562025)(20161123558025)(20161123564025)(6072148); SRVR:BN3PR0701MB1719; BCL:0; PCL:0; RULEID:; SRVR:BN3PR0701MB1719; X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1719; 4:NuVFrHHty23rBYoNK9Qzd9PvD/w6rZDFGrW4UsPitVqpchJZyJQExRrHxolVku656ZTV7QiGVpyQGSF9E+vBxX8Q7Sa2vpdugxJrFtXhSgohqOjuD6ApQvpzbUjRcm0UJZNTLoJXhgbVgSVVBJLWJ9sko59xG8DiPwa/fxGlNKPtSfZdRIyj2jTUm9qg4gdDNTTTM54SSfupj682sEIG5tKmLLO9nscqx4kyICVrkZST7pzCwj3nKYLdLuB/yLULKRTTmkxcmpM4yHNWJCuRIgIw6BYn1OdKPVRki2iOlTbukWxnXF2jRywXyxzF4NBAdoNj5MQEyBCn6gOnkbjy1cV8bBABwl8bJi4nj3Xu7SSvqbroPePg7ruBlC/aLAt9MlzzrgtnKoLQQuai2Czu+RfGSv9f+vEfWiGgJc/jMaw2vgRgX3d1jq2Pl23uX7JHl6Oj8Ka+x1xol7Q4j5A5F1uF8jUpwwlYNdHiPB3y+zZopLidCZG26hxTfi9JiVZ6RvPNRiWCD3h5ObiHYdwZ2xCquGNGZypgk2VW6fYRakmaxctJkj2z76qQ2aLVaqn6/hHraOcmuCVGzh1vryfTq+ksjmCXHGAmtImebNdZ3Jw= X-Forefront-PRVS: 0235CBE7D0 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6069001)(6009001)(7916002)(39450400003)(3846002)(50986999)(6116002)(36756003)(81166006)(6512007)(97736004)(2361001)(2351001)(6506006)(5660300001)(42186005)(50226002)(8676002)(106356001)(8656002)(2950100002)(2906002)(25786008)(47776003)(6486002)(76176999)(33646002)(7736002)(107886003)(42882006)(66066001)(110136004)(4326008)(38730400002)(6916009)(305945005)(48376002)(92566002)(5009440100003)(5003940100001)(50466002)(189998001)(53936002)(7099028)(110426004)(309714004); DIR:OUT; SFP:1101; SCL:1; SRVR:BN3PR0701MB1719; H:localhost.localdomain.localdomain; FPR:; SPF:None; MLV:nov; PTR:InfoNoRecords; LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN3PR0701MB1719; 23:SH2gnrNnYMwpsiGF3aVLffDdTKEFa/CEkUi6b3R?= =?us-ascii?Q?Uasq/McKSgLLWpHY/uj64kiBU3f6zue70AaxYZzTQ8+sK1zvhX+97YHDZcBU?= =?us-ascii?Q?nwdkaZarOIca5XZy5GFkR+Ay42CoyT41km7tKNMfs+5l4AXt9aTOxWiEDtr6?= =?us-ascii?Q?/SR3ce61ygXhPsVWP/bFOAV+0RAUJjum3FgdzLDhtrrzSJIzisQnhTxERH7F?= =?us-ascii?Q?EZHDt9hxqh523j6Is0VEofqGoSbGJayrZAz3c9yhVmZUwpV4XrAHGkpdBq3G?= =?us-ascii?Q?CurL3FWEhJGgluhTrMQnz2Ksmw5W5Kl9pPDsD2XUHljh/tA9kuGT0IM4tB0c?= =?us-ascii?Q?cWZt5H0m+AcoGb1+PZB7DiwgOV/XmOrWB5UFOLDMh64wt9ummEIFEC6ckVLo?= =?us-ascii?Q?afu9Rj3e2EZ8s2LB/v3CK18LWmKBlEZ7YPIxrgOYbCHTZ8isURJuP7v+83s3?= =?us-ascii?Q?q4MGLA+onjX+X46QuZyNzcoIuBDDEbL0hMCupQTG+MUoq8e6WpSZL3cXVa/x?= =?us-ascii?Q?45MhYrKtMBpE1j/uyh+DUZqctLILTHDVwM/JN7xGVFnquIcFju/f0DGgUp5K?= =?us-ascii?Q?qZFUH3E57moxlJVgexXNfeX4h8wLUBs9QSnh45eb+YOaQ4FhqBKi3ufLDyPz?= =?us-ascii?Q?C7tK9MF2owkBKURaRCBeLB8Qgdv89xNVLLr+k0DfNf4yJV/jLEVAomsZy+e9?= =?us-ascii?Q?tiuoa8q2SIUgtbHqAbPSXg+WEc7XZPD5BGtKD4XEninqvgh1CwLMrQ+xuJpJ?= =?us-ascii?Q?2oLTBeC8OdVgeXPZPKQq7y/JtcfMlVgItzkxyOP+beU4yVf2T+UbB87ZaXkR?= =?us-ascii?Q?pWf4jqHOHmo23O2fBcGMTWPYEmzqN06/DIrWyoopdimEojLfPY+pbm1f5bAR?= =?us-ascii?Q?Be6giRtqWGpAp6adFCQiSIoyB2dQBNC7034Iceok3m7M7mkB2BZsRm5xJ2zs?= =?us-ascii?Q?j12kQBulBtVg1AVx+RtsdXLYKBwYYxP1j2x0CmMpGIel7zQW6DBlb6HU6JKN?= =?us-ascii?Q?QeBPhPhoVU64anV3vE1jReedu71bKL1IJ4IeSKRMIu65RjoTHDH8wThO235g?= =?us-ascii?Q?tKeR3/H2o0X2C+3pipklVR0rNfy0JInfmXUhU8JOZ+yqqiV5yRawuyiD+OoY?= =?us-ascii?Q?fhHvs/zlpWihDygFlDCHubXedxM6G7yBFfTgkVVAILy1lgvLJBY2e/g=3D?= =?us-ascii?Q?=3D?= X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1719; 6:V1qyTMm62Szm+1NNB+HCHxj4/bWXUlskW+Yd0q6kR/PZ95QFjIpEKCYmDLJ+gpiaEoZfKfDYBiy28MfAvkxmp3vC8ZPpR6X4IvSfJFm3wfyriEsPr/DpOXS5eDYw4jUgTotzgpiD65lsKbhqm1zV+nEIwBudluo7Y1pSuuxtZKDPjksoMtY45bsSRsWmzGPSj80UE4XX57pHYRpsdQeYwJYzAB3P5aeSqZWUWhr05Cz1zQx/oCuaqmgjm2LieHg4oIX/uq3vUzlyHW8sPKsfxL447d1YqOPhWQF9ioAIjWB+iczHuq58Py1UjCSGounR286CoqQH0vEdpUsiEygNT49iIUnFEfiv0jdDlfBSRfzGVr0HGZDkBlVS7UA2HXRns3rfnKuDXn8Ngum70V9eZQ==; 5:UzxbaVD86onqQb5ZibPkUtpPFBWxSVRIWVlIePhnGu55GOqPRbVZyprsNwecl7Aa159Q9/JkIWaUthxFtXHmlp2eJmR9Cm1+Ot7oqPWxI7Pw8YHx+dtr0j4jfcOZRgOLDrJjl/lDBBWEgeEGSdPXwQ==; 24:tdZ6/EfDky9stejPOTHFj4MccJNoXTTkUnxf3+GMFsP+Ol6dqIL/uXMvwsjt6ZOuYi+LDskil4N/3DOPbiHnvS/4w1Kqbox1d6zmDN6hQkQ= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1719; 7:x4LMwEN0/6O1UaHgVKxowS5pp76uvFYebdDavX/r9GpTehQvso+EU8JvWi8RnUX4TUh2cOxrTvEbmhbPAI2s378tG4R4BtcTWweMr+pSwJf88311eJbsHPbX9v8wWMxfQwjRR7NnVb4VEgwJ8k+npvAfgbhxzkGVpfFcsiYxFHWVBPzqO7Fqe1zsC9bHeaeMH/quJ6bpRzEHs36EuxP0uOJriEUa8ESY78bwyQyBFsOMOZP7vVeQdKQupZhAFA1ZkFla1UICwvzi6m25RzZHcmLrkvIU/0Ep5dA5H+W5GZuu0d4MuQvFsbQYH+YZdCMmYznla1lBYIBA8eM/9Y1LlA== X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Mar 2017 17:30:01.1727 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR0701MB1719 Subject: [dpdk-dev] [PATCH 17/39] event/octeontx: add SSO HW device operations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Mar 2017 17:30:06 -0000 Signed-off-by: Jerin Jacob Signed-off-by: Santosh Shukla --- drivers/event/octeontx/Makefile | 5 ++ drivers/event/octeontx/ssovf_evdev.h | 30 ++++++++ drivers/event/octeontx/ssovf_worker.c | 140 ++++++++++++++++++++++++++++++++++ 3 files changed, 175 insertions(+) create mode 100644 drivers/event/octeontx/ssovf_worker.c diff --git a/drivers/event/octeontx/Makefile b/drivers/event/octeontx/Makefile index 6e8227b..c921303 100644 --- a/drivers/event/octeontx/Makefile +++ b/drivers/event/octeontx/Makefile @@ -46,9 +46,14 @@ LIBABIVER := 1 # # all source are stored in SRCS-y # +SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_worker.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_evdev.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_probe.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_mbox.c +ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y) +CFLAGS_ssovf_worker.o += -fno-prefetch-loop-arrays +endif +CFLAGS_ssovf_worker.o += -Ofast # install this header file SYMLINK-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF)-include := rte_pmd_octeontx_ssovf.h diff --git a/drivers/event/octeontx/ssovf_evdev.h b/drivers/event/octeontx/ssovf_evdev.h index 16bf3d5..16846b8 100644 --- a/drivers/event/octeontx/ssovf_evdev.h +++ b/drivers/event/octeontx/ssovf_evdev.h @@ -129,6 +129,36 @@ #define ssovf_read64 rte_read64_relaxed #define ssovf_write64 rte_write64_relaxed +/* ARM64 specific functions */ +#if defined(RTE_ARCH_ARM64) +#define ssovf_load_pair(val0, val1, addr) ({ \ + asm volatile( \ + "ldp %x[x0], %x[x1], [%x[p1]]" \ + :[x0]"=r"(val0), [x1]"=r"(val1) \ + :[p1]"r"(addr) \ + ); }) + +#define ssovf_store_pair(val0, val1, addr) ({ \ + asm volatile( \ + "stp %x[x0], %x[x1], [%x[p1]]" \ + ::[x0]"r"(val0), [x1]"r"(val1), [p1]"r"(addr) \ + ); }) +#else /* Un optimized functions for building on non arm64 arch */ + +#define ssovf_load_pair(val0, val1, addr) \ +do { \ + val0 = rte_read64(addr); \ + val1 = rte_read64(((uint8_t *)addr) + 8); \ +} while (0) + +#define ssovf_store_pair(val0, val1, addr) \ +do { \ + rte_write64(val0, addr); \ + rte_write64(val1, (((uint8_t *)addr) + 8)); \ +} while (0) +#endif + + struct ssovf_evdev { uint8_t max_event_queues; uint8_t max_event_ports; diff --git a/drivers/event/octeontx/ssovf_worker.c b/drivers/event/octeontx/ssovf_worker.c new file mode 100644 index 0000000..8a96076 --- /dev/null +++ b/drivers/event/octeontx/ssovf_worker.c @@ -0,0 +1,140 @@ +/* + * BSD LICENSE + * + * Copyright (C) Cavium networks Ltd. 2017. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Cavium networks nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + + +#include + +#include "ssovf_evdev.h" + +enum { + SSO_SYNC_ORDERED, + SSO_SYNC_ATOMIC, + SSO_SYNC_UNTAGGED, + SSO_SYNC_EMPTY +}; + +#ifndef force_inline +#define force_inline inline __attribute__((always_inline)) +#endif + +#ifndef __hot +#define __hot __attribute__((hot)) +#endif + +/* SSO Operations */ + +static force_inline uint16_t +ssows_get_work(struct ssows *ws, struct rte_event *ev) +{ + uint64_t get_work0, get_work1; + uint64_t sched_type_queue; + + ssovf_load_pair(get_work0, get_work1, ws->getwork); + + sched_type_queue = (get_work0 >> 32) & 0xfff; + ws->cur_tt = sched_type_queue & 0x3; + ws->cur_grp = sched_type_queue >> 2; + sched_type_queue = sched_type_queue << 38; + + ev->event = sched_type_queue | (get_work0 & 0xffffffff); + ev->u64 = get_work1; + return !!get_work1; +} + +static force_inline void +ssows_add_work(struct ssows *ws, const uint64_t event_ptr, const uint32_t tag, + const uint8_t new_tt, const uint8_t grp) +{ + uint64_t add_work0; + + add_work0 = tag | ((uint64_t)(new_tt) << 32); + ssovf_store_pair(add_work0, event_ptr, ws->grps[grp]); +} + +static force_inline void +ssows_swtag_full(struct ssows *ws, const uint64_t event_ptr, const uint32_t tag, + const uint8_t new_tt, const uint8_t grp) +{ + uint64_t swtag_full0; + + swtag_full0 = tag | ((uint64_t)(new_tt & 0x3) << 32) | + ((uint64_t)grp << 34); + ssovf_store_pair(swtag_full0, event_ptr, (ws->base + + SSOW_VHWS_OP_SWTAG_FULL0)); +} + +static force_inline void +ssows_swtag_desched(struct ssows *ws, uint32_t tag, uint8_t new_tt, uint8_t grp) +{ + uint64_t val; + + val = tag | ((uint64_t)(new_tt & 0x3) << 32) | ((uint64_t)grp << 34); + ssovf_write64(val, ws->base + SSOW_VHWS_OP_SWTAG_DESCHED); +} + +static force_inline void +ssows_swtag_norm(struct ssows *ws, uint32_t tag, uint8_t new_tt) +{ + uint64_t val; + + val = tag | ((uint64_t)(new_tt & 0x3) << 32); + ssovf_write64(val, ws->base + SSOW_VHWS_OP_SWTAG_NORM); +} + +static force_inline void +ssows_swtag_untag(struct ssows *ws) +{ + ssovf_write64(0, ws->base + SSOW_VHWS_OP_SWTAG_UNTAG); + ws->cur_tt = SSO_SYNC_UNTAGGED; +} + +static force_inline void +ssows_upd_wqp(struct ssows *ws, uint8_t grp, uint64_t event_ptr) +{ + ssovf_store_pair((uint64_t)grp << 34, event_ptr, (ws->base + + SSOW_VHWS_OP_UPD_WQP_GRP0)); +} + +static force_inline void +ssows_desched(struct ssows *ws) +{ + ssovf_write64(0, ws->base + SSOW_VHWS_OP_DESCHED); +} + +static force_inline void +ssows_swtag_wait(struct ssows *ws) +{ + /* Wait for the SWTAG/SWTAG_FULL operation */ + while (ssovf_read64(ws->base + SSOW_VHWS_SWTP)) + ; +} + -- 2.5.5