From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-bl2nam02on0082.outbound.protection.outlook.com [104.47.38.82]) by dpdk.org (Postfix) with ESMTP id 358D9A49B for ; Wed, 14 Mar 2018 14:53:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=zp3wjJw/KNbB2zK5t5YwYjDoqw086zBHdiJl+e/cRGs=; b=LhSTRGJAZYoiolo71iQVD68bmqdRI2ZC693BoAiB30mVz1dmb2nywi3ipA+oCd5i2Xjmfk1AzqmEievYl2DaMiH+2KNUYkx1Pqn/qb7vvcrk59FrcuY0Raa8CCbhWLdDbzlECuJRw2t7NQ2g538/n5ejWKOr/ZeHV0nDBDoZkU4= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Pavan.Bhagavatula@cavium.com; Received: from ltp-pvn.caveonetworks.com (111.93.218.67) by MWHPR07MB3470.namprd07.prod.outlook.com (2603:10b6:301:63::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.567.14; Wed, 14 Mar 2018 13:53:19 +0000 From: Pavan Nikhilesh To: jerin.jacob@caviumnetworks.com, santosh.shukla@caviumnetworks.com, erik.g.carrillo@intel.com Cc: dev@dpdk.org, Pavan Nikhilesh Date: Wed, 14 Mar 2018 19:22:27 +0530 Message-Id: <20180314135233.31282-6-pbhagavatula@caviumnetworks.com> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180314135233.31282-1-pbhagavatula@caviumnetworks.com> References: <20180216213700.3415-1-pbhagavatula@caviumnetworks.com> <20180314135233.31282-1-pbhagavatula@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: DM5PR20CA0005.namprd20.prod.outlook.com (2603:10b6:3:93::15) To MWHPR07MB3470.namprd07.prod.outlook.com (2603:10b6:301:63::21) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4191d931-b3e8-4ba3-a443-08d589b2f324 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(2017052603328)(7153060)(7193020); SRVR:MWHPR07MB3470; X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3470; 3:vKas3QBDSGt8vO+JBFZkJzVggEGhdUKhhGk0Yv+4wNBZOxuZoovWehxolljU3ulGyz8E4Ycs0WS88F+PNuBkZpuxCtR8nwpv9nrtQDIivGstQtxsjOySUXMJ8/NZuBmUILgY7xQwPIBjP+JmFs99GhhIJUQyG1Bz1K+KBO5dV6QHmv4fzss561qCyudHWoeYs4UEseZIVnRGFYrOYGbS4iRCiclNzDTl/TgkAm9N2viodp4AOXgW22VwNZVufQtP; 25:ZKQ+nbIyOba2FUiN8+V5exYu0GdbGE6uBz2qpNH01fRZxr/0b4xT0NZlUM8/NYthvUpa/CnGSG5Q1idhQk54NaAThfVEFJEYWpvE44Z4SbBjkF45B6MCPRvWhZKcQz+2cpkEB7rvGYuJrPFvlQiNhyFUheXFAQOw6jHEks2N8bIv81Hw77EuUrO90Dcr4E14G2/WZPoD0qWPzIxxX/hSwQndmHayeoYOluoDO1gHHDHERU8HSjiTxk0oMk3Sp+diRh9xoPQQzbJvjkB7ZFWmRuyLcZDBqFhNuvZUovLxbSZCsu6AifnW6zUbVIxSkvGnDQ5qstFM9WOSAPKrZcKVPA==; 31:Xtnshxc7Fm/Emavyf6tUGkb3V22xgdRAdmQBihJNXkw8J2SF/PtaT47AFt+RVi7MMnxPlGQTkbglMcko1mP6FHswi8EhpTPvxO7M0WJK+KcZkIoqxrO8DDwIZdl6gEjHXSWwUcNH5uFS94zrvMrE+2tDO5imaBrMhuFHO47Mcw0paSocglXApMQO+Sn4L/C9SHbTnLFHYqUAlBu5prK7eeyWotnHaynGN7y9CyVwQB8= X-MS-TrafficTypeDiagnostic: MWHPR07MB3470: X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3470; 20:CDu+KkCMULYNcF/OBEFrtjk4MDBTqKGIHMmheSaOhGgUtwEsv707cSOrMa8QBTzLvN/lBA2vN5OKNabf3tFfCG3BH+dVQ3GyJgt+HzXfVA90x7Wh19VsXc7tOkF+LA1IQDXd4v0Fo6rZtTkpLAYIPpF4q73WdHPzhNS/z4R3nMoBFrNO6aLcUWBAjYyFIbpiSQ5uvb3C0YUWuceclW+gfV7YGWdLsPBRfwvYi6lF/tPmwbq2xCDmvPfnqqugN35zNzsHHjPetuHBwoW8hj+JQPURiGBoSxNYOKF5pIrzvUYC2GAA75WKtlmzo8LG6r3FksoSfj2gBP3oS8jCNUGiWpd3hr2hFvqaZykpxPbb2DdwC8HfZjJjEJ2UH2b31das9d7BlQMbvRK6x58YS/GLjOrsEytxDpWLpDE0hgdDQpWf6gqFVJxvvEaYTkelykGKvPS305EIbEWXVgx7S8jyxGNky9dF93Mse+Eg7Nl4uepcUxXrsuYW5RnJYdHAO2bNOPfnwAj1EF26YAQ0iEhh+oBuWdBjcCum/jjyieKBGtJfLGvR4JLZynBY7w+5vVt6vduXgyibiwXDfLpH+fooNkSDjDvG3YiDPKF58mtfVWI=; 4:7a5WkiFhtKeU6nxkgyzfZMsoaA4fYcf1vwBBe03mZ1pShaXXKIUAhTkDQ3wW/B0e0G0FQqprq23zmgZYYJ6Nlugg3cntOIZFK8qGv2A42DF0eZc1qIk9zGo0JqtytZ7nnqd1rVbfaTy0RXOWQmRS9HtSJAbNQcpLyTE7lf2BAtkz/gP6Enm+BcuRBbaJaBUYQG2PIg7gvcmSLIgFdvHDYOmznZ3hlhqrEjq+1KEoqCrGhhs53OZXnN20v3aLlX7mAXXCHrnBNw8dKFaLeK+wkg== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(3231221)(944501244)(52105095)(3002001)(93006095)(10201501046)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123562045)(20161123560045)(20161123558120)(6072148)(201708071742011); SRVR:MWHPR07MB3470; BCL:0; PCL:0; RULEID:; SRVR:MWHPR07MB3470; X-Forefront-PRVS: 0611A21987 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(376002)(39860400002)(39380400002)(366004)(346002)(396003)(189003)(199004)(53936002)(6512007)(3846002)(4326008)(6666003)(6506007)(48376002)(26005)(105586002)(68736007)(97736004)(47776003)(2950100002)(386003)(1076002)(6486002)(106356001)(52116002)(305945005)(107886003)(7736002)(69596002)(76176011)(66066001)(5660300001)(59450400001)(51416003)(25786009)(36756003)(50466002)(316002)(16586007)(72206003)(53416004)(1857600001)(6116002)(478600001)(8676002)(81156014)(81166006)(42882007)(2906002)(5009440100003)(50226002)(16526019)(8936002)(42262002); DIR:OUT; SFP:1101; SCL:1; SRVR:MWHPR07MB3470; H:ltp-pvn.caveonetworks.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR07MB3470; 23:wiYItMXElgTBWTEFdNBaRmEYMcIHgosl9n60zUwsu?= =?us-ascii?Q?gubKutQWcdwkKOniDjZQFKprDdlAyHe9XB54E0uYEUm5CO1a695UGxAEl6vB?= =?us-ascii?Q?u538EAcF1NN5xiGuN98GDktVafCEPMIedI/Z1pHPjhTyrF0yPL0Q4Cs88vEZ?= =?us-ascii?Q?U8Fga6Z8EAv1dhMnBDNhCOSj2gJQoJUmE+3yAwDUoJdFJl1hIeSw8C5+JFa7?= =?us-ascii?Q?5KfYc4i6G6Y7Pi3nWoBVi7eVX8HGIteRG5psx3Xm+4fxvK3Djf1dSTk8Ig2T?= =?us-ascii?Q?92XkwpRoIvS3xGDfX3TKHYKhxRPx7eEo/SUboVWDgjcd1ujXu1tgRQlISqlE?= =?us-ascii?Q?c327SHeOIFt9OPV4MkNZ+FguG0XxVfhJ5eeEWcyrjubwPNAqITI+/m7JAPnS?= =?us-ascii?Q?c8FLBW/iTLZQIAuTqhnPfgIUctX11QEHUZl0wAHVvhCagft+GGBY4gzUna82?= =?us-ascii?Q?pH2kP+4ijeHFFixlOxu2lwzghsiuUiZJAsNY1wldz8xz159HrRDHlODswB0I?= =?us-ascii?Q?m3RgS7SSnk9pR9FuCxBrw7nyl7vkfVTLH7UsSl5E77WM76ZNK9CAKBioU4lJ?= =?us-ascii?Q?oS0KoNXHtDrI+vxv8Nk4KcHg0bYiwmxoWpEQKUN7L2D1oNyjEPA92kc4XIaD?= =?us-ascii?Q?PoF4nbS+BQoPGrXv/+V+WDX3EU3oa5gmcSqbdZ09F2hlHMdlmAOkKdbU9MhF?= =?us-ascii?Q?DZhItm339Byvys3cbEaA24OXFHlCjJzKlXFhfA2RpBhpZHBI4Y/hEEKupXGI?= =?us-ascii?Q?CGNYwdD6/Lt6Tdhgfayaa3dxOZ8aOpwogtW88DlX5Uth6OnZm5F08hA4OHCu?= =?us-ascii?Q?NrBEIpJ8kB4feIOZ0ZijE3M2qar7pyOhaBcDgKayRALNJNsRnduT8/y4ptpy?= =?us-ascii?Q?UPtfMfpOQ7loeh4sXilg1RBYeo/2N/KUxPI8+9u1CWYI2KmnYjcQepYs0/mX?= =?us-ascii?Q?n01Nf8ui1RfagcH2BieDBJR2pZyday7w+R6pE473dvzIRBxpg9BBiwOnvNNC?= =?us-ascii?Q?co25oVRQi3NYkMFEWnF4fCyDvDAGSJ4dzrRlloyF1f0Bk6KUKA2fSLalDDcZ?= =?us-ascii?Q?gJKPb/utTaMaLuMmb3kh6Wn903dq9pYRH9jOdg7DhR4gImHFgJ9SpMgvVHKo?= =?us-ascii?Q?ZSyPjKzyN0a91wt2QBBERNVOCzBGVsW0wN1FzhJj1SpQLgmfRGJauYEQS2YU?= =?us-ascii?Q?lMcxWA0rx9TVrs5r/U9kcETTxFQv/OzsF4QAzFyJ2buoZkSeEoHty/vRBUWn?= =?us-ascii?Q?g4YvozpmUklVdqoO64gk8eQvXjoXqExZ7TG16lJ?= X-Microsoft-Antispam-Message-Info: wRNb6HEG6w57eW8FZoLIWuose2j8LCoNQ/W5Y389Fl8IQ8568x7Q7vrX0U8Aei8UA7L2jWdwKuPzBvcxTH9Yd4N1FYyR9GNcs2zgpjpkzyeW1I19YzPvilz/UAxeHt5ENbDJjko6r1aZPUo5WhGPcF40GmPBkFjtoL/pJSajtKYmny3wGNaQcV9YWD5ZpzQH X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3470; 6:0qMgpBIDIgcdkyrr8ESBbE58g17WzvgAMK4eX/FybYTc0BJqngqaNKyduFaZjGjnXxwpBmWhBu/F2gG/QAte8nyeYf1rd4jXd9jxUC8FEvsko++gokE7u5TNO2elAf/YcLi/A0BqsB8/496bDzTKSX0+6nlcVD13KWPsxxXJRoEvBJQ8t/v5D1TvtzyrM1DtW09TRAefm8LZ0K+nlTy8K+XAGJO9Rp3cFNjUCXPgmyt3ScZQ9PUiliFhkpUmj6W55OllEAXAphO2CM+YmhXHeQkA95dGlK6CA/56dG6cgzVrd4KayZgk0/2sIF0LS+5hnOvjlR+JTiuxl4fXUF3F9rhxscZvotrBKcVeZOaOdP4=; 5:0Mtnd0k4k0tWgIIFZWFux6XE5e9WCnmbqUPO8t6PrZ97uzSxrXY7xWvbDGS+Vc62wtJvtvsFW5bMY5uTfxLGPg+16F8XljUBTDWYzrHfjf5dPp7RA84mNAuskuh7sgCyRP2QgIo5mnkmuuUNCim5ANl9cMxwjFPJQ4Rgefvd8+Q=; 24:WM4l1TO64oUKaOLTgJuVd4R0PT17eoaC5AQXk9yJvAuekRO0t6xY6UoTMUOVap1Be0AvNa6TM+BcacI0+BZNumrMX8WC7+ex5RVbPT52YVo=; 7:EKdv/jbUWiMbxbZNhI+5oygsmUBEw3Sb+2hr91Ke76+m2+TXkrRQnlsc5dJRVfDO3OtkYzAbPBnjTwOOZClhV+XLHBeNqppqaejiuK+myEy38RvJ9OWpfiqNzaHfG+D4lc4LEVx3CBO+E7TlI9oT92UN7RW90fXOMz+kVQImBZO/YIg2/Wx4c4zFDLBd2l7dCrX3ne8x5O5uVvMvQBRl+B5H0CRux0B8XrLxkKguZ/hynmHf9XBMUo6WPTNdJkx0 SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Mar 2018 13:53:19.9397 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4191d931-b3e8-4ba3-a443-08d589b2f324 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR07MB3470 Subject: [dpdk-dev] [PATCH v2 05/11] event/octeontx: add multiproducer timer arm and cancel X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Mar 2018 13:53:25 -0000 Signed-off-by: Pavan Nikhilesh --- drivers/event/octeontx/Makefile | 5 + drivers/event/octeontx/meson.build | 1 + drivers/event/octeontx/timvf_evdev.c | 7 +- drivers/event/octeontx/timvf_evdev.h | 32 ++++ drivers/event/octeontx/timvf_worker.c | 59 +++++++ drivers/event/octeontx/timvf_worker.h | 280 ++++++++++++++++++++++++++++++++++ 6 files changed, 383 insertions(+), 1 deletion(-) create mode 100644 drivers/event/octeontx/timvf_worker.c create mode 100644 drivers/event/octeontx/timvf_worker.h diff --git a/drivers/event/octeontx/Makefile b/drivers/event/octeontx/Makefile index 570a79412..58fa6af88 100644 --- a/drivers/event/octeontx/Makefile +++ b/drivers/event/octeontx/Makefile @@ -28,19 +28,24 @@ LIBABIVER := 1 SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_worker.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_evdev.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_evdev_selftest.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += timvf_worker.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += timvf_evdev.c ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y) CFLAGS_ssovf_worker.o += -fno-prefetch-loop-arrays +CFLAGS_timvf_worker.o += -fno-prefetch-loop-arrays ifeq ($(shell test $(GCC_VERSION) -ge 46 && echo 1), 1) CFLAGS_ssovf_worker.o += -Ofast +CFLAGS_timvf_worker.o += -Ofast else CFLAGS_ssovf_worker.o += -O3 -ffast-math +CFLAGS_timvf_worker.o += -O3 -ffast-math endif else CFLAGS_ssovf_worker.o += -Ofast +CFLAGS_timvf_worker.o += -Ofast endif include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/event/octeontx/meson.build b/drivers/event/octeontx/meson.build index 8941f8a56..efac94cba 100644 --- a/drivers/event/octeontx/meson.build +++ b/drivers/event/octeontx/meson.build @@ -4,6 +4,7 @@ sources = files('ssovf_worker.c', 'ssovf_evdev.c', 'ssovf_evdev_selftest.c', + 'timvf_worker.c', 'timvf_evdev.c', ) diff --git a/drivers/event/octeontx/timvf_evdev.c b/drivers/event/octeontx/timvf_evdev.c index ccf724115..852e8811c 100644 --- a/drivers/event/octeontx/timvf_evdev.c +++ b/drivers/event/octeontx/timvf_evdev.c @@ -126,6 +126,7 @@ timvf_ring_start(const struct rte_event_timer_adapter *adptr) timvf_write64((uint64_t)timr->meta.bkt, (uint8_t *)timr->vbar0 + TIM_VRING_BASE); + timvf_set_chunk_refill(timr); if (timvf_ring_conf_set(&rctrl, timr->tim_ring_id)) { ret = -EACCES; goto error; @@ -295,7 +296,11 @@ timvf_timer_adapter_caps_get(const struct rte_eventdev *dev, uint64_t flags, { RTE_SET_USED(dev); RTE_SET_USED(flags); + + timvf_ops.arm_burst = timvf_timer_reg_burst_mp; + timvf_ops.cancel_burst = timvf_timer_unreg_burst; + *caps = RTE_EVENT_TIMER_ADAPTER_CAP_INTERNAL_PORT; *ops = &timvf_ops; - return -EINVAL; + return 0; } diff --git a/drivers/event/octeontx/timvf_evdev.h b/drivers/event/octeontx/timvf_evdev.h index e155b6ce2..c717c19bc 100644 --- a/drivers/event/octeontx/timvf_evdev.h +++ b/drivers/event/octeontx/timvf_evdev.h @@ -75,6 +75,33 @@ #define TIM_VRING_AURA (0x108) #define TIM_VRING_REL (0x110) +#define TIM_CTL1_W0_S_BUCKET 20 +#define TIM_CTL1_W0_M_BUCKET ((1ull << (40 - 20)) - 1) + +#define TIM_BUCKET_W1_S_NUM_ENTRIES (0) /*Shift*/ +#define TIM_BUCKET_W1_M_NUM_ENTRIES ((1ull << (32 - 0)) - 1) +#define TIM_BUCKET_W1_S_SBT (32) +#define TIM_BUCKET_W1_M_SBT ((1ull << (33 - 32)) - 1) +#define TIM_BUCKET_W1_S_HBT (33) +#define TIM_BUCKET_W1_M_HBT ((1ull << (34 - 33)) - 1) +#define TIM_BUCKET_W1_S_BSK (34) +#define TIM_BUCKET_W1_M_BSK ((1ull << (35 - 34)) - 1) +#define TIM_BUCKET_W1_S_LOCK (40) +#define TIM_BUCKET_W1_M_LOCK ((1ull << (48 - 40)) - 1) +#define TIM_BUCKET_W1_S_CHUNK_REMAINDER (48) +#define TIM_BUCKET_W1_M_CHUNK_REMAINDER ((1ull << (64 - 48)) - 1) + +#define TIM_BUCKET_SEMA \ + (TIM_BUCKET_CHUNK_REMAIN) + +#define TIM_BUCKET_CHUNK_REMAIN \ + (TIM_BUCKET_W1_M_CHUNK_REMAINDER << TIM_BUCKET_W1_S_CHUNK_REMAINDER) + +#define TIM_BUCKET_LOCK \ + (TIM_BUCKET_W1_M_LOCK << TIM_BUCKET_W1_S_LOCK) + +#define TIM_BUCKET_SEMA_WLOCK \ + (TIM_BUCKET_CHUNK_REMAIN | (1ull << TIM_BUCKET_W1_S_LOCK)) #define NSEC_PER_SEC 1E9 #define NSEC2CLK(__ns, __freq) (((__ns) * (__freq)) / NSEC_PER_SEC) @@ -166,5 +193,10 @@ bkt_mod(const uint32_t rel_bkt, const uint32_t nb_bkts) int timvf_timer_adapter_caps_get(const struct rte_eventdev *dev, uint64_t flags, uint32_t *caps, const struct rte_event_timer_adapter_ops **ops); +int timvf_timer_unreg_burst(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers); +int timvf_timer_reg_burst_mp(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers); +void timvf_set_chunk_refill(struct timvf_ring * const timr); #endif /* __TIMVF_EVDEV_H__ */ diff --git a/drivers/event/octeontx/timvf_worker.c b/drivers/event/octeontx/timvf_worker.c new file mode 100644 index 000000000..113603616 --- /dev/null +++ b/drivers/event/octeontx/timvf_worker.c @@ -0,0 +1,59 @@ +/* + * SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2017 Cavium, Inc + */ + +#include "timvf_worker.h" + +int +timvf_timer_unreg_burst(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers) +{ + RTE_SET_USED(adptr); + int ret; + uint16_t index; + + for (index = 0; index < nb_timers; index++) { + ret = timvf_rem_entry(tim[index]); + if (ret) { + rte_errno = -ret; + break; + } + } + return index; +} + +int +timvf_timer_reg_burst_mp(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers) +{ + int ret; + uint16_t index; + struct tim_mem_entry entry; + struct timvf_ring *timr = adptr->data->adapter_priv; + for (index = 0; index < nb_timers; index++) { + if (unlikely(tim[index]->timeout_ticks > timr->meta.nb_bkts)) { + tim[index]->state = RTE_EVENT_TIMER_ERROR_TOOLATE; + rte_errno = EINVAL; + break; + } + + entry.w0 = (tim[index]->ev.event & 0xFFC000000000) >> 6 | + (tim[index]->ev.event & 0xFFFFFFFFF); + entry.wqe = tim[index]->ev.u64; + ret = timvf_add_entry_mp(timr, tim[index]->timeout_ticks, + tim[index], &entry); + if (unlikely(ret)) { + rte_errno = -ret; + break; + } + } + + return index; +} + +void +timvf_set_chunk_refill(struct timvf_ring * const timr) +{ + timr->meta.refill_chunk = timvf_refill_chunk_generic; +} diff --git a/drivers/event/octeontx/timvf_worker.h b/drivers/event/octeontx/timvf_worker.h new file mode 100644 index 000000000..bba4985ab --- /dev/null +++ b/drivers/event/octeontx/timvf_worker.h @@ -0,0 +1,280 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2017 Cavium, Inc + */ + +#include +#include + +#include "timvf_evdev.h" + +static inline int16_t +timr_bkt_fetch_rem(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_CHUNK_REMAINDER) & + TIM_BUCKET_W1_M_CHUNK_REMAINDER; +} + +static inline int16_t +timr_bkt_get_rem(struct tim_mem_bucket *bktp) +{ + return __atomic_load_n((int16_t *)&bktp->chunk_remainder, + __ATOMIC_ACQUIRE); +} + +static inline void +timr_bkt_set_rem(struct tim_mem_bucket *bktp, uint16_t v) +{ + __atomic_store_n((int16_t *)&bktp->chunk_remainder, v, + __ATOMIC_RELEASE); +} + +static inline void +timr_bkt_sub_rem(struct tim_mem_bucket *bktp, uint16_t v) +{ + __atomic_fetch_sub((int16_t *)&bktp->chunk_remainder, v, + __ATOMIC_RELEASE); +} + +static inline uint8_t +timr_bkt_get_sbt(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_SBT) & TIM_BUCKET_W1_M_SBT; +} + +static inline uint64_t +timr_bkt_set_sbt(struct tim_mem_bucket *bktp) +{ + const uint64_t v = TIM_BUCKET_W1_M_SBT << TIM_BUCKET_W1_S_SBT; + return __atomic_fetch_or((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline uint64_t +timr_bkt_clr_sbt(struct tim_mem_bucket *bktp) +{ + const uint64_t v = ~(TIM_BUCKET_W1_M_SBT << TIM_BUCKET_W1_S_SBT); + return __atomic_fetch_and((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline uint8_t +timr_bkt_get_shbt(uint64_t w1) +{ + return ((w1 >> TIM_BUCKET_W1_S_HBT) & TIM_BUCKET_W1_M_HBT) | + ((w1 >> TIM_BUCKET_W1_S_SBT) & TIM_BUCKET_W1_M_SBT); +} + +static inline uint8_t +timr_bkt_get_hbt(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_HBT) & TIM_BUCKET_W1_M_HBT; +} + +static inline uint8_t +timr_bkt_get_bsk(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_BSK) & TIM_BUCKET_W1_M_BSK; +} + +static inline uint64_t +timr_bkt_clr_bsk(struct tim_mem_bucket *bktp) +{ + /*Clear everything except lock. */ + const uint64_t v = TIM_BUCKET_W1_M_LOCK << TIM_BUCKET_W1_S_LOCK; + return __atomic_fetch_and((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline uint64_t +timr_bkt_fetch_sema_lock(struct tim_mem_bucket *bktp) +{ + return __atomic_fetch_add((uint64_t *)&bktp->w1, TIM_BUCKET_SEMA_WLOCK, + __ATOMIC_ACQ_REL); +} + +static inline uint64_t +timr_bkt_fetch_sema(struct tim_mem_bucket *bktp) +{ + return __atomic_fetch_add((uint64_t *)&bktp->w1, TIM_BUCKET_SEMA, + __ATOMIC_RELAXED); +} + +static inline uint64_t +timr_bkt_inc_lock(struct tim_mem_bucket *bktp) +{ + const uint64_t v = 1ull << TIM_BUCKET_W1_S_LOCK; + return __atomic_fetch_add((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline void +timr_bkt_dec_lock(struct tim_mem_bucket *bktp) +{ + __atomic_add_fetch((uint8_t *)&bktp->lock, 0xff, __ATOMIC_ACQ_REL); +} + +static inline uint32_t +timr_bkt_get_nent(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_NUM_ENTRIES) & + TIM_BUCKET_W1_M_NUM_ENTRIES; +} + +static inline void +timr_bkt_inc_nent(struct tim_mem_bucket *bktp) +{ + __atomic_add_fetch((uint32_t *)&bktp->nb_entry, 1, __ATOMIC_RELAXED); +} + +static inline void +timr_bkt_add_nent(struct tim_mem_bucket *bktp, uint32_t v) +{ + __atomic_add_fetch((uint32_t *)&bktp->nb_entry, v, __ATOMIC_RELAXED); +} + +static inline uint64_t +timr_bkt_clr_nent(struct tim_mem_bucket *bktp) +{ + const uint64_t v = ~(TIM_BUCKET_W1_M_NUM_ENTRIES << + TIM_BUCKET_W1_S_NUM_ENTRIES); + return __atomic_and_fetch((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline struct tim_mem_entry * +timr_clr_bkt(struct timvf_ring * const timr, struct tim_mem_bucket * const bkt) +{ + struct tim_mem_entry *chunk; + struct tim_mem_entry *pnext; + chunk = ((struct tim_mem_entry *)bkt->first_chunk); + chunk = (struct tim_mem_entry *)(chunk + nb_chunk_slots)->w0; + + while (chunk) { + pnext = (struct tim_mem_entry *)((chunk + nb_chunk_slots)->w0); + rte_mempool_put(timr->meta.chunk_pool, chunk); + chunk = pnext; + } + return (struct tim_mem_entry *)bkt->first_chunk; +} + +static inline int +timvf_rem_entry(struct rte_event_timer *tim) +{ + uint64_t lock_sema; + struct tim_mem_entry *entry; + struct tim_mem_bucket *bkt; + if (tim->impl_opaque[1] == 0 || + tim->impl_opaque[0] == 0) + return -ENOENT; + + entry = (struct tim_mem_entry *)tim->impl_opaque[0]; + if (entry->wqe != tim->ev.u64) { + tim->impl_opaque[1] = tim->impl_opaque[0] = 0; + return -ENOENT; + } + bkt = (struct tim_mem_bucket *)tim->impl_opaque[1]; + lock_sema = timr_bkt_inc_lock(bkt); + if (timr_bkt_get_shbt(lock_sema) + || !timr_bkt_get_nent(lock_sema)) { + timr_bkt_dec_lock(bkt); + tim->impl_opaque[1] = tim->impl_opaque[0] = 0; + return -ENOENT; + } + + entry->w0 = entry->wqe = 0; + timr_bkt_dec_lock(bkt); + + tim->state = RTE_EVENT_TIMER_CANCELED; + tim->impl_opaque[1] = tim->impl_opaque[0] = 0; + return 0; +} + +static inline struct tim_mem_entry * +timvf_refill_chunk_generic(struct tim_mem_bucket * const bkt, + struct timvf_ring * const timr) +{ + struct tim_mem_entry *chunk; + + if (bkt->nb_entry || !bkt->first_chunk) { + if (unlikely(rte_mempool_get(timr->meta.chunk_pool, + (void **)&chunk))) { + return NULL; + } + if (bkt->nb_entry) { + *(uint64_t *)((struct tim_mem_entry *) + bkt->current_chunk + + nb_chunk_slots) = + (uint64_t) chunk; + } else { + bkt->first_chunk = (uint64_t) chunk; + } + } else { + chunk = timr_clr_bkt(timr, bkt); + bkt->first_chunk = (uint64_t) chunk; + } + *(uint64_t *)(chunk + nb_chunk_slots) = 0; + + return chunk; +} + +static inline struct tim_mem_bucket * +timvf_get_target_bucket(struct timvf_ring * const timr, const uint32_t rel_bkt) +{ + const uint64_t bkt_cyc = rte_rdtsc() - timr->meta.ring_start_cyc; + const uint32_t bucket = rte_reciprocal_divide_u64(bkt_cyc, + &timr->meta.fast_div) + rel_bkt; + const uint32_t tbkt_id = timr->meta.get_target_bkt(bucket, + timr->meta.nb_bkts); + return &timr->meta.bkt[tbkt_id]; +} + +/* Multi producer functions. */ +static inline int +timvf_add_entry_mp(struct timvf_ring * const timr, const uint32_t rel_bkt, + struct rte_event_timer * const tim, + const struct tim_mem_entry * const pent) +{ + uint8_t lock_cnt; + int16_t rem; + uint64_t lock_sema; + struct tim_mem_bucket *bkt; + struct tim_mem_entry *chunk; + +__retry: + bkt = timvf_get_target_bucket(timr, rel_bkt); + /* Bucket related checks. */ + /*Get Bucket sema*/ + lock_sema = timr_bkt_fetch_sema_lock(bkt); + if (unlikely(timr_bkt_get_shbt(lock_sema))) { + timr_bkt_dec_lock(bkt); + goto __retry; + } + + RTE_SET_USED(lock_cnt); + rem = timr_bkt_fetch_rem(lock_sema); + + if (rem < 0) { + /* goto diff bucket. */ + timr_bkt_dec_lock(bkt); + goto __retry; + } else if (!rem) { + /*Only one thread can be here*/ + chunk = timr->meta.refill_chunk(bkt, timr); + if (unlikely(chunk == NULL)) { + timr_bkt_set_rem(bkt, 0); + timr_bkt_dec_lock(bkt); + tim->impl_opaque[0] = tim->impl_opaque[1] = 0; + tim->state = RTE_EVENT_TIMER_ERROR; + return -ENOMEM; + } + bkt->current_chunk = (uint64_t) chunk; + timr_bkt_set_rem(bkt, nb_chunk_slots - 1); + } else { + chunk = (struct tim_mem_entry *)bkt->current_chunk; + chunk += nb_chunk_slots - rem; + } + /* Copy work entry. */ + *chunk = *pent; + timr_bkt_inc_nent(bkt); + timr_bkt_dec_lock(bkt); + + tim->impl_opaque[0] = (uint64_t)chunk; + tim->impl_opaque[1] = (uint64_t)bkt; + tim->state = RTE_EVENT_TIMER_ARMED; + return 0; +} -- 2.16.2