From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-by2nam01on0083.outbound.protection.outlook.com [104.47.34.83]) by dpdk.org (Postfix) with ESMTP id 200061B2CB for ; Fri, 16 Feb 2018 22:37:48 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=xxhFb1GG+2bZ2xWpRPwH7G69nspdTe4yKMiUpWGOfsk=; b=CtWSNpPFxr6sTLXVyEwtmbg1lAwyge68cJdsLKyVJF5mp5rC6BbfdEWAngJBVdFLJ4v75djnRQwgqJ9SweonNjY6U8/eYJ7y+Nnnj/+KxU9mmN6H73fuf/mjjWn/vbDHlxOISmJ7E2OUUHka4dj9K7aDoITUW4xBJnPOuzM4Ck0= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Pavan.Bhagavatula@cavium.com; Received: from localhost.localdomain (111.93.218.67) by DM5PR07MB3467.namprd07.prod.outlook.com (2603:10b6:4:67::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.506.18; Fri, 16 Feb 2018 21:37:44 +0000 From: Pavan Nikhilesh To: jerin.jacob@caviumnetworks.com, santosh.shukla@caviumnetworks.com, erik.g.carrillo@intel.com Cc: dev@dpdk.org, Pavan Nikhilesh Date: Sat, 17 Feb 2018 03:06:55 +0530 Message-Id: <20180216213700.3415-6-pbhagavatula@caviumnetworks.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180216213700.3415-1-pbhagavatula@caviumnetworks.com> References: <20180216213700.3415-1-pbhagavatula@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: SG2PR01CA0090.apcprd01.prod.exchangelabs.com (2603:1096:3:15::16) To DM5PR07MB3467.namprd07.prod.outlook.com (2603:10b6:4:67::22) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: dce005f7-3300-4aac-57ad-08d5758584b5 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(2017052603307)(7153060)(7193020); SRVR:DM5PR07MB3467; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3467; 3:zWMXZa5MVZ+ddo69u4fAP87B+ZprCpd94F7Blnjhih+12jPdbhPEbO0pQhFv4v1H28k6MtXX6vZr3b1fu6DnLDFOQXu7uA080oib04s8o1W4wcr5Fkf5ii1nC6vRfrJvbtzYm8G8yKTkmdMwqMqLGr76I6NpSFWROf8bXKfXVX7ki3QBX7CE69530V5gPN/P4Aq9C/5yoL357pBPT0zFWwaPShGynVxHnbkaerKYmbriODmqKcwOxRK1Is8ZHgAP; 25:cOSng4lTpqesfOpc9yjYSoDaafDBdsEeE/Svc4wC1uuuPOvn3oiPX8XPe3SKfwstErFVz02/SazTlS/f8Qd7knm5qOUhuOSqAi0GsK32AJlPzq8r4StYuGveosiar6TzdVCwUOvfVtjHnTIID63INhkJds/3UwUzb+1RNxLx9A9nbK22jbG3rTc/a6Wz6HFjyDhgHaxkJPz1JBiwB+C6RgevpnC8ZQu4E9C5VqSqOzNmJtlbx50WGPjn8g+NMw9zfXC1Y4uexS8p5tMnYRm2rSc9ZiO4RZWOosVrGyJ2H/41mhjEBjvSf9SBEq/kxLN99P0RIcLBs7BoPUfISBhGvw==; 31:XUSGC7qy61D/FSCGi5OliRiJR8gnTiKDOW1tZOt/0SpqVFcxrCfb+aCJwTWXKbNUiHD/mT/CSESNJC2WYSinpTwKx7aTYH3+fNuo8QvH74IGV5niwRkwNFRn0HJMCLai4ZEO7b9D9A8XqOv0Cxl1pNiJSed3WRuH2epQZKbxMkR2mxHbglIjgqDRlPtGlfK06x3gcjlSbHdald8LGeA8Dh2tz35jssYQmkSY6bBeWmQ= X-MS-TrafficTypeDiagnostic: DM5PR07MB3467: X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3467; 20:SZIr76HqrpJGksE2CgyCp1xYvTsK8+iY95FwjdYQ/e7b92r68ypd35faDNKglRnwi6jvzcQ5NkwL7CNsC2ck7y4aQvPdiyWtQxzYawXTqkmt5WFduOWW/S7+l9GAEEVKXXGdASirGORmXb21yVwRvtVFXdAfOqL2ThJmGaEyj8NH1+xV1mJCahLfvHCR0UBWYaQPLMGJOWYkEtsoDoglyyfHXDCrfCH7pQg5nnxXT2v9mDsD2oncsjNpy1PsGd1agIFwoYt2oh0TJgk305QG+mygBA40hw0Ga7Uf40+eXZDig9JU72BOt3Ncyhep5QjudbuWZRD4HjZ5hpClZaetQhbFVY1vZuCQ5Kym8MIjBVYsvHe6gKBmNS8ag4wqh3PSRVXlLyRuHpRSg6G02BXzZxILzIFzPOlDz4MBjLpOe4OQ29SK22xBrJw25UTXJYt+N/XrQdZ7THYXOw70bDdpnwyTIQNe+Z0Jd119uab47TWLF60xzgowZQwXvNdNDRjErSxZofCs9IBBTN9va/V3zHKuoqEngbImvsErm3ebrWRpo36Tb8P2xlBkTurIHskClDXGXnQTMgJcGri23IWUcnjd8MprtCqWDPjwyHwpH2w=; 4:hM76wr8J5/JUm2fthxFM8G1vs3L4pEd5XRtKSHXuhv+DLUQeWNX4Qi49hlE3OyhoOD6rq+0UxprtCGb78NK28q55JDSB5vwpSjG8cxi3US4gvtvLssru1kzDoHX2rzAQT7tHlwQMJ7B1RXLYy6BTJDLWVvt5FpQYF5euqIKix+69X0HXVsViva/bFeGhCWAZ4kmh/oI51/qbxPY9S+EMZ3rBnYW31iKC4DB+XQ3aGG6FeosqcrhbzQJ1USfsNoqIRSRmSow11+ws3EotaP5q1w== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040501)(2401047)(8121501046)(5005006)(10201501046)(3231101)(944501161)(93006095)(3002001)(6041288)(20161123560045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(20161123558120)(6072148)(201708071742011); SRVR:DM5PR07MB3467; BCL:0; PCL:0; RULEID:; SRVR:DM5PR07MB3467; X-Forefront-PRVS: 0585417D7B X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6069001)(346002)(366004)(396003)(39860400002)(39380400002)(376002)(189003)(199004)(6486002)(25786009)(2950100002)(386003)(6506007)(53936002)(42882006)(5009440100003)(4326008)(107886003)(6666003)(59450400001)(6512007)(16586007)(106356001)(316002)(105586002)(36756003)(478600001)(2906002)(47776003)(6116002)(76176011)(3846002)(1076002)(97736004)(66066001)(8676002)(51416003)(50466002)(48376002)(81156014)(5660300001)(68736007)(8936002)(575784001)(52116002)(16526019)(26005)(305945005)(1857600001)(72206003)(81166006)(50226002)(7736002)(42262002); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR07MB3467; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM5PR07MB3467; 23:nKJ1yLXlKhj57HJCcnZ1uzw5teczNAp6MjVsjWFBX?= =?us-ascii?Q?t9qERUF8PRxU906cm8RuRKJf/kuM0xXEy9Sd9kWqFI3A5Qp5GZ3SDSij/dhK?= =?us-ascii?Q?z6v2hZRIps+XKmxbm4ce04PJQ5GqnkIuC09prt2dPNY8MNh+rIggw8upWL57?= =?us-ascii?Q?5jHOXO5qLEg8y2mGeddEZq8EC8ClYW1mNY6KOfM1vHoct8aDgo1MFg6OT1kN?= =?us-ascii?Q?MFYhM2ehBycWUkByMbQrZoKeTrpwS/Ueg9VSMmklhwQ0SwpJRDjne2yCWBZJ?= =?us-ascii?Q?VjDncyZQ5qrZhT+SyVLkxv8ohjP7DR9BWoHHZXGcsX5+sKYsF1QBOPhjXCAj?= =?us-ascii?Q?CuIOPYcOjzmI/rSvO5Q2q+IZ7LdyfiFAy9t3Hp8EnhOfcl+LK01dK4R87t7w?= =?us-ascii?Q?rwAoaHgH8/SEJyhvmPeCf2Zh9ZcjLUuE1y8CJVU1Rsm/4hs3ijkiQvozgH6P?= =?us-ascii?Q?r0qLHrJHx6nwySDFyXwTWuU071Yza7ckSfozWSQzwdUFAu6z6noAhBzki/BM?= =?us-ascii?Q?RWcfhgJ1cxGaWqY5scOP72AKok4uJHCCzJ3V74W98h30Xc4mC0v66PORq7kM?= =?us-ascii?Q?hIhBjFF52mUTmNHlHtMOFQ/cW3BTwQKTu2AtGyDhrWAHd213PqRRGOhakr5R?= =?us-ascii?Q?S0TxnD+AwhyfpFlq1R8hWDnNAsysjuJss+vHkADHBlxzD4EOaBTLzK7695yW?= =?us-ascii?Q?3IifxOEyCQbReynZrdxWwZYL7ZmKQYeZ/YHJ1z6W9HJBnxZdNy8i5iq2EV4M?= =?us-ascii?Q?jrADkM6HGPOszE8P53KIBVTGEAJVlqq80qAm4lWk8B7pR0uVN/0jcN9P58iB?= =?us-ascii?Q?V9QS/39PjKvsnDhWf+ZTA2oqCmMs9mR/AFRObZNG/5tN3tBIDGL/k0iW/J8g?= =?us-ascii?Q?K4rIaTfcKvI/hm3pmzAhT+jYZbaXiSPwPJp8T6Slyof1pH8xrJBjLvZ17JmV?= =?us-ascii?Q?ZbQYoV9peVQMK72a3TpZo8iCRBcceyh558M9Id+Wwta7IXMbY4RY5E/nXsD3?= =?us-ascii?Q?i9zMQQZtnKyg8a8yLSk0vnrnHouxaDk3OO9EXVvHGR93NkMppcPEw4jFpBMF?= =?us-ascii?Q?JJ3Y07Cft9F8w/By68MNqwN+JFDFE4jK72T40h1JsuRTbqu5zxMvj3OBL+3m?= =?us-ascii?Q?8q2MMAzQdc4m4J298NjlA5T5gS4B5t2XxJyM0zPOOQ3kcYfGMIfHUNV8iWTi?= =?us-ascii?Q?CUiGdf6Uty+WcDVMbPwX1MGq3QlJ/Aq8PobOB3kufNa6UK8TzzyW6gEIltCA?= =?us-ascii?Q?70jPhUaiTWkaVEwukoiUgj1q4cffWjtyQXz4JdY?= X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3467; 6:FBH6bwmpZXuVDOu0Kyj7oyCKVUod9uPaZ5OaPUtttcoxRke6PlVbTi4yn/3127eOOOksL4pjhnw6xubmqkK0hbMV6TYdqebQkza9HAougf+rX2h9pcNqSmpAxSjnjXMRrKENZKthZ0XdCSoPkmj5TFXsREVpjCvAB0AxyxFmTzGeMELF6CkfPKh/yOYyflXmhp/BpR9thJh/ovRwLPiarqmUJDcHOvSGYO/Wv7h3LGFLF2fFzeZ+QotJZBTOVhhXbSSUdKF2ro1w8gwWbqlDEHtZ9ekzaZdwE3eerQ47MgyxAONCtpnNGiON67IZyvs/53oN2/aYuvWYIRdISU/6DWWjhCeYly1h1du7LHilg/E=; 5:2ZLPyOzVc2oqV7BkUOP83MPNdCpTuOOipaew+QcxmcWu/ICUBS0RhL4eZ4dHAinqa4jbtvnC3jYXp3RqRADvHQL6Rz1LdUtadiVrVed9xQxc/Top5WAdnsfa9O+GD3XmE02NYRDQxehg1BMWXlYDLvNDcrWFaE8ZghpStuZrYYE=; 24:65bjivGfjQq3gsORnjDBqty3K4cguzNE+baEIrlby6rzev7SePbLPOt7Vp9BMQ0BTCJFkUNLOOe9KuVDZyYL8w9RY3IDv5sDaX3FJojdIO8=; 7:Y8UeSzP63T4sZcCAJBYuzklNtbbbVY4f+xNYbiVX+DaRyGgKqsEEWQ4P4Fdwx/zBU2WdGttFgUWwvKAZBAnjMHU9W8fnaUrytKE+uDXFhRSdOiKAXcLQihPxlHF05PMT78Sn7kUmIoEaWZLEnFF9+QncZ3mHngxtzgDLHmQJOwEcd1Ha5WHWRlGWMlh486PHesdZ9rfz5L6O9fufklv3oo5E8MUZeJi79HgSZL1luRR8RzW1kPvehGgG0O0LFIsP SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2018 21:37:44.1464 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: dce005f7-3300-4aac-57ad-08d5758584b5 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR07MB3467 Subject: [dpdk-dev] [PATCH 05/10] event/octeontx: add support for arm and cancel X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Feb 2018 21:37:48 -0000 Signed-off-by: Pavan Nikhilesh --- drivers/event/octeontx/Makefile | 5 + drivers/event/octeontx/meson.build | 1 + drivers/event/octeontx/timvf_evdev.c | 11 +- drivers/event/octeontx/timvf_evdev.h | 31 ++++ drivers/event/octeontx/timvf_worker.c | 53 +++++++ drivers/event/octeontx/timvf_worker.h | 278 ++++++++++++++++++++++++++++++++++ 6 files changed, 378 insertions(+), 1 deletion(-) create mode 100644 drivers/event/octeontx/timvf_worker.c create mode 100644 drivers/event/octeontx/timvf_worker.h diff --git a/drivers/event/octeontx/Makefile b/drivers/event/octeontx/Makefile index f1d10a99e..ba1ccfb04 100644 --- a/drivers/event/octeontx/Makefile +++ b/drivers/event/octeontx/Makefile @@ -27,19 +27,24 @@ LIBABIVER := 1 SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_worker.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_evdev.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_evdev_selftest.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += timvf_worker.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += timvf_evdev.c ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y) CFLAGS_ssovf_worker.o += -fno-prefetch-loop-arrays +CFLAGS_timvf_worker.o += -fno-prefetch-loop-arrays ifeq ($(shell test $(GCC_VERSION) -ge 46 && echo 1), 1) CFLAGS_ssovf_worker.o += -Ofast +CFLAGS_timvf_worker.o += -Ofast else CFLAGS_ssovf_worker.o += -O3 -ffast-math +CFLAGS_timvf_worker.o += -O3 -ffast-math endif else CFLAGS_ssovf_worker.o += -Ofast +CFLAGS_timvf_worker.o += -Ofast endif include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/event/octeontx/meson.build b/drivers/event/octeontx/meson.build index 8941f8a56..efac94cba 100644 --- a/drivers/event/octeontx/meson.build +++ b/drivers/event/octeontx/meson.build @@ -4,6 +4,7 @@ sources = files('ssovf_worker.c', 'ssovf_evdev.c', 'ssovf_evdev_selftest.c', + 'timvf_worker.c', 'timvf_evdev.c', ) diff --git a/drivers/event/octeontx/timvf_evdev.c b/drivers/event/octeontx/timvf_evdev.c index dccdf1ddb..d0ba42263 100644 --- a/drivers/event/octeontx/timvf_evdev.c +++ b/drivers/event/octeontx/timvf_evdev.c @@ -261,6 +261,10 @@ timvf_ring_create(struct rte_event_timer_adapter *adptr) timvf_write64(0x7, (uint8_t *)timr->vbar0 + TIM_VF_NRSPERR_ENA_W1C); timvf_write64(0x7, (uint8_t *)timr->vbar0 + TIM_VF_NRSPERR_ENA_W1S); + adptr->arm_burst = timvf_timer_reg_burst_mp; + adptr->arm_tmo_tick_burst = NULL; + adptr->cancel_burst = timvf_timer_unreg_burst; + return 0; mem_err: rte_free(timr); @@ -294,7 +298,12 @@ timvf_timer_adapter_caps_get(const struct rte_eventdev *dev, uint64_t flags, { RTE_SET_USED(dev); RTE_SET_USED(flags); + + timvf_ops.arm_burst = timvf_timer_reg_burst_mp; + timvf_ops.arm_tmo_tick_burst = NULL; + + timvf_ops.cancel_burst = timvf_timer_unreg_burst; *caps = RTE_EVENT_TIMER_ADAPTER_CAP_INTERNAL_PORT; *ops = &timvf_ops; - return -EINVAL; + return 0; } diff --git a/drivers/event/octeontx/timvf_evdev.h b/drivers/event/octeontx/timvf_evdev.h index a6a64587f..c80e147e8 100644 --- a/drivers/event/octeontx/timvf_evdev.h +++ b/drivers/event/octeontx/timvf_evdev.h @@ -74,6 +74,33 @@ #define TIM_VRING_AURA (0x108) #define TIM_VRING_REL (0x110) +#define TIM_CTL1_W0_S_BUCKET 20 +#define TIM_CTL1_W0_M_BUCKET ((1ull << (40 - 20)) - 1) + +#define TIM_BUCKET_W1_S_NUM_ENTRIES (0) /*Shift*/ +#define TIM_BUCKET_W1_M_NUM_ENTRIES ((1ull << (32 - 0)) - 1) +#define TIM_BUCKET_W1_S_SBT (32) +#define TIM_BUCKET_W1_M_SBT ((1ull << (33 - 32)) - 1) +#define TIM_BUCKET_W1_S_HBT (33) +#define TIM_BUCKET_W1_M_HBT ((1ull << (34 - 33)) - 1) +#define TIM_BUCKET_W1_S_BSK (34) +#define TIM_BUCKET_W1_M_BSK ((1ull << (35 - 34)) - 1) +#define TIM_BUCKET_W1_S_LOCK (40) +#define TIM_BUCKET_W1_M_LOCK ((1ull << (48 - 40)) - 1) +#define TIM_BUCKET_W1_S_CHUNK_REMAINDER (48) +#define TIM_BUCKET_W1_M_CHUNK_REMAINDER ((1ull << (64 - 48)) - 1) + +#define TIM_BUCKET_SEMA \ + (TIM_BUCKET_CHUNK_REMAIN) + +#define TIM_BUCKET_CHUNK_REMAIN \ + (TIM_BUCKET_W1_M_CHUNK_REMAINDER << TIM_BUCKET_W1_S_CHUNK_REMAINDER) + +#define TIM_BUCKET_LOCK \ + (TIM_BUCKET_W1_M_LOCK << TIM_BUCKET_W1_S_LOCK) + +#define TIM_BUCKET_SEMA_WLOCK \ + (TIM_BUCKET_CHUNK_REMAIN | (1ull << TIM_BUCKET_W1_S_LOCK)) #define NSEC_PER_SEC 1E9 #define NSEC2CLK(__ns, __freq) (((__ns) * (__freq)) / NSEC_PER_SEC) @@ -159,5 +186,9 @@ bkt_mod(uint32_t rel_bkt, uint32_t nb_bkts) int timvf_timer_adapter_caps_get(const struct rte_eventdev *dev, uint64_t flags, uint32_t *caps, const struct rte_event_timer_adapter_ops **ops); +int timvf_timer_unreg_burst(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers); +int timvf_timer_reg_burst_mp(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers); #endif /* __TIMVF_EVDEV_H__ */ diff --git a/drivers/event/octeontx/timvf_worker.c b/drivers/event/octeontx/timvf_worker.c new file mode 100644 index 000000000..7a924fd11 --- /dev/null +++ b/drivers/event/octeontx/timvf_worker.c @@ -0,0 +1,53 @@ +/* + * SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2017 Cavium, Inc + */ + +#include "timvf_worker.h" + +int +timvf_timer_unreg_burst(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers) +{ + RTE_SET_USED(adptr); + int ret; + uint16_t index; + + for (index = 0; index < nb_timers; index++) { + ret = timvf_rem_entry(tim[index]); + if (ret) { + rte_errno = -ret; + break; + } + } + return index; +} + +int +timvf_timer_reg_burst_mp(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers) +{ + int ret; + uint16_t index; + struct tim_mem_entry entry; + struct timvf_ring *timr = adptr->data->adapter_priv; + for (index = 0; index < nb_timers; index++) { + if (unlikely(tim[index]->timeout_ticks > timr->meta.nb_bkts)) { + tim[index]->state = RTE_EVENT_TIMER_ERROR_TOOLATE; + rte_errno = -EINVAL; + break; + } + + entry.w0 = (tim[index]->ev.event & 0xFFC000000000) >> 6 | + (tim[index]->ev.event & 0xFFFFFFFFF); + entry.wqe = tim[index]->ev.u64; + ret = timvf_add_entry_mp(timr, tim[index]->timeout_ticks, + tim[index], &entry); + if (unlikely(ret)) { + rte_errno = -ret; + break; + } + } + + return index; +} diff --git a/drivers/event/octeontx/timvf_worker.h b/drivers/event/octeontx/timvf_worker.h new file mode 100644 index 000000000..b63dd763c --- /dev/null +++ b/drivers/event/octeontx/timvf_worker.h @@ -0,0 +1,278 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2017 Cavium, Inc + */ + +#include +#include + +#include "timvf_evdev.h" + +static inline int16_t +timr_bkt_fetch_rem(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_CHUNK_REMAINDER) & + TIM_BUCKET_W1_M_CHUNK_REMAINDER; +} + +static inline int16_t +timr_bkt_get_rem(struct tim_mem_bucket *bktp) +{ + return __atomic_load_n((int16_t *)&bktp->chunk_remainder, + __ATOMIC_ACQUIRE); +} + +static inline void +timr_bkt_set_rem(struct tim_mem_bucket *bktp, uint16_t v) +{ + __atomic_store_n((int16_t *)&bktp->chunk_remainder, v, + __ATOMIC_RELEASE); +} + +static inline void +timr_bkt_sub_rem(struct tim_mem_bucket *bktp, uint16_t v) +{ + __atomic_fetch_sub((int16_t *)&bktp->chunk_remainder, v, + __ATOMIC_RELEASE); +} + +static inline uint8_t +timr_bkt_get_sbt(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_SBT) & TIM_BUCKET_W1_M_SBT; +} + +static inline uint64_t +timr_bkt_set_sbt(struct tim_mem_bucket *bktp) +{ + const uint64_t v = TIM_BUCKET_W1_M_SBT << TIM_BUCKET_W1_S_SBT; + return __atomic_fetch_or((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline uint64_t +timr_bkt_clr_sbt(struct tim_mem_bucket *bktp) +{ + const uint64_t v = ~(TIM_BUCKET_W1_M_SBT << TIM_BUCKET_W1_S_SBT); + return __atomic_fetch_and((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline uint8_t +timr_bkt_get_shbt(uint64_t w1) +{ + return ((w1 >> TIM_BUCKET_W1_S_HBT) & TIM_BUCKET_W1_M_HBT) | + ((w1 >> TIM_BUCKET_W1_S_SBT) & TIM_BUCKET_W1_M_SBT); +} + +static inline uint8_t +timr_bkt_hbt_chk(struct tim_mem_bucket *bktp) +{ + return (__atomic_load_n((uint64_t *)&bktp->w1, __ATOMIC_ACQ_REL) + >> TIM_BUCKET_W1_S_HBT) & TIM_BUCKET_W1_M_HBT; +} + +static inline uint8_t +timr_bkt_sbt_chk(struct tim_mem_bucket *bktp) +{ + return (__atomic_load_n((uint64_t *)&bktp->w1, __ATOMIC_ACQ_REL) + >> TIM_BUCKET_W1_S_SBT) & TIM_BUCKET_W1_M_SBT; +} + +static inline uint8_t +timr_bkt_get_bsk(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_BSK) & TIM_BUCKET_W1_M_BSK; +} + +static inline uint64_t +timr_bkt_clr_bsk(struct tim_mem_bucket *bktp) +{ + /*Clear everything except lock. */ + const uint64_t v = TIM_BUCKET_W1_M_LOCK << TIM_BUCKET_W1_S_LOCK; + return __atomic_fetch_and((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline uint64_t +timr_bkt_fetch_sema_lock(struct tim_mem_bucket *bktp) +{ + return __atomic_fetch_add((uint64_t *)&bktp->w1, TIM_BUCKET_SEMA_WLOCK, + __ATOMIC_ACQ_REL); +} + +static inline uint64_t +timr_bkt_fetch_sema(struct tim_mem_bucket *bktp) +{ + return __atomic_fetch_add((uint64_t *)&bktp->w1, TIM_BUCKET_SEMA, + __ATOMIC_ACQ_REL); +} + +static inline uint64_t +timr_bkt_inc_lock(struct tim_mem_bucket *bktp) +{ + const uint64_t v = 1ull << TIM_BUCKET_W1_S_LOCK; + return __atomic_fetch_add((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline void +timr_bkt_dec_lock(struct tim_mem_bucket *bktp) +{ + __atomic_add_fetch((uint8_t *)&bktp->lock, 0xff, __ATOMIC_ACQ_REL); +} + +static inline uint32_t +timr_bkt_get_nent(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_NUM_ENTRIES) & + TIM_BUCKET_W1_M_NUM_ENTRIES; +} + +static inline void +timr_bkt_inc_nent(struct tim_mem_bucket *bktp) +{ + __atomic_add_fetch((uint32_t *)&bktp->nb_entry, 1, __ATOMIC_ACQ_REL); +} + +static inline void +timr_bkt_add_nent(struct tim_mem_bucket *bktp, uint32_t v) +{ + __atomic_add_fetch((uint32_t *)&bktp->nb_entry, v, __ATOMIC_ACQ_REL); +} + +static inline uint64_t +timr_bkt_clr_nent(struct tim_mem_bucket *bktp) +{ + const uint64_t v = ~(TIM_BUCKET_W1_M_NUM_ENTRIES << + TIM_BUCKET_W1_S_NUM_ENTRIES); + return __atomic_and_fetch((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline __hot struct tim_mem_entry* +timr_clr_bkt(struct timvf_ring *timr, struct tim_mem_bucket *bkt) +{ + struct tim_mem_entry *chunk; + struct tim_mem_entry *pnext; + chunk = ((struct tim_mem_entry *)bkt->first_chunk); + chunk = (struct tim_mem_entry *)(chunk + nb_chunk_slots)->w0; + + while (chunk) { + pnext = (struct tim_mem_entry *)((chunk + nb_chunk_slots)->w0); + rte_mempool_put(timr->meta.chunk_pool, chunk); + chunk = pnext; + } + return (struct tim_mem_entry *)bkt->first_chunk; +} + +static inline int __hot +timvf_rem_entry(struct rte_event_timer *tim) +{ + uint64_t lock_sema; + struct tim_mem_entry *entry; + struct tim_mem_bucket *bkt; + if (tim->impl_opaque[1] == 0 || + tim->impl_opaque[0] == 0) + return -ENOENT; + + entry = (struct tim_mem_entry *)tim->impl_opaque[0]; + if (entry->wqe != tim->ev.u64) { + tim->impl_opaque[1] = tim->impl_opaque[0] = 0; + return -ENOENT; + } + bkt = (struct tim_mem_bucket *)tim->impl_opaque[1]; + lock_sema = timr_bkt_inc_lock(bkt); + if (timr_bkt_get_shbt(lock_sema) + || !timr_bkt_get_nent(lock_sema)) { + timr_bkt_dec_lock(bkt); + tim->impl_opaque[1] = tim->impl_opaque[0] = 0; + return -ENOENT; + } + + entry->w0 = entry->wqe = 0; + timr_bkt_dec_lock(bkt); + + tim->state = RTE_EVENT_TIMER_CANCELED; + tim->impl_opaque[1] = tim->impl_opaque[0] = 0; + return 0; +} + +/* Multi producer functions. */ +static inline int __hot +timvf_add_entry_mp(struct timvf_ring *timr, const uint32_t rel_bkt, + struct rte_event_timer *tim, const struct tim_mem_entry *pent) +{ + uint8_t lock_cnt; + int16_t rem; + uint32_t bucket; + uint32_t tbkt_id; + const uint32_t nb_bkts = timr->meta.nb_bkts; + uint64_t lock_sema; + uint64_t pos_reg; + const uint64_t start = timr->meta.ring_start_cyc; + struct tim_mem_bucket *bkt; + struct tim_mem_entry *chunk; + +__retry: + pos_reg = (rte_rdtsc() - start); + bucket = rte_reciprocal_divide_u64(pos_reg, + &timr->meta.fast_div) + rel_bkt; + tbkt_id = timr->meta.get_target_bkt(bucket, nb_bkts); + bkt = &timr->meta.bkt[tbkt_id]; + while (1) { + /* Bucket related checks. */ + + /*Get Bucket sema*/ + lock_sema = timr_bkt_fetch_sema_lock(bkt); + if (unlikely(timr_bkt_get_shbt(lock_sema))) { + timr_bkt_dec_lock(bkt); + goto __retry; + } + + RTE_SET_USED(lock_cnt); + /* Insert the work. */ + rem = timr_bkt_fetch_rem(lock_sema); + + if (rem < 0) { + /* goto diff bucket. */ + timr_bkt_dec_lock(bkt); + goto __retry; + } else if (!rem) { + /*Only one thread can be here*/ + if (bkt->nb_entry || !bkt->first_chunk) { + if (unlikely(rte_mempool_get( + timr->meta.chunk_pool, + (void **)&chunk))) { + timr_bkt_set_rem(bkt, 0); + timr_bkt_dec_lock(bkt); + tim->impl_opaque[0] = + tim->impl_opaque[1] = 0; + tim->state = RTE_EVENT_TIMER_ERROR; + return -ENOMEM; + } + if (bkt->nb_entry) { + *(uint64_t *)((struct tim_mem_entry *) + bkt->current_chunk + + nb_chunk_slots) = + (uint64_t) chunk; + } else { + bkt->first_chunk = (uint64_t) chunk; + } + *(uint64_t *)(chunk + nb_chunk_slots) = 0; + } else { + chunk = timr_clr_bkt(timr, bkt); + *(uint64_t *)(chunk + nb_chunk_slots) = 0; + bkt->first_chunk = (uint64_t) chunk; + } + bkt->current_chunk = (uint64_t) chunk; + timr_bkt_set_rem(bkt, nb_chunk_slots - 1); + } else { + chunk = (struct tim_mem_entry *)bkt->current_chunk; + chunk += nb_chunk_slots - rem; + } + /* Copy work entry. */ + *chunk = *pent; + timr_bkt_inc_nent(bkt); + timr_bkt_dec_lock(bkt); + break; + } + tim->impl_opaque[0] = (uint64_t)chunk; + tim->impl_opaque[1] = (uint64_t)bkt; + tim->state = RTE_EVENT_TIMER_ARMED; + return 0; +} -- 2.16.1