From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-by2nam01on0052.outbound.protection.outlook.com [104.47.34.52]) by dpdk.org (Postfix) with ESMTP id 3B0F01BA35 for ; Mon, 9 Apr 2018 23:01:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=tuf5Uum+W6Q99MA/KWyWVAaviop/Z7Ke73W515fsh+M=; b=aw8juhMo6o+pnAhLgoz/+nqcE8jvRS6D8vIof0ZHpxmOEddj7+6k6LZmEbJFE6MCKfufxTzf8uD07wQXX9Y7yC5ZtwqHrFJk6Yavb2PQ5FYisWA/GE1fL81c00URI594N5+A3cjXPKHNH1nV+alckGAVd1TRQK6EmaqCuqKFL50= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Pavan.Bhagavatula@cavium.com; Received: from localhost.localdomain (111.93.218.67) by MWHPR07MB3470.namprd07.prod.outlook.com (2603:10b6:301:63::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.653.12; Mon, 9 Apr 2018 21:01:32 +0000 From: Pavan Nikhilesh To: jerin.jacob@caviumnetworks.com, santosh.shukla@caviumnetworks.com, erik.g.carrillo@intel.com Cc: dev@dpdk.org, Pavan Nikhilesh Date: Tue, 10 Apr 2018 02:30:30 +0530 Message-Id: <20180409210035.23278-7-pbhagavatula@caviumnetworks.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180409210035.23278-1-pbhagavatula@caviumnetworks.com> References: <20180216213700.3415-1-pbhagavatula@caviumnetworks.com> <20180409210035.23278-1-pbhagavatula@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: BM1PR01CA0088.INDPRD01.PROD.OUTLOOK.COM (2603:1096:b00:1::28) To MWHPR07MB3470.namprd07.prod.outlook.com (2603:10b6:301:63::21) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 689f04b0-004d-4e50-d607-08d59e5d13c6 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(2017052603328)(7153060)(7193020); SRVR:MWHPR07MB3470; X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3470; 3:+/7eJsAW2G7+clwMapRkWQmMJbIJ+S818kpv29Kz09HaPZMD2SLPH1Nz98dyUySJZYzUNNS+D7sSnoOF9UDUjrBpk+fOcdPwVs9Tj/OJ/w7VIGSNZcE4wRjkh586DByJcjB+3etrASHPj2bLslSthLIwQn4IsA38itwn6+lZcttvnwhKaruS4RlLlVWJa12X0zWjs95iq1Xz4+jvWRQpWf6Izi716ezD0LkzRHmDMkbHHkt/bDfePCVdXQQgPaKI; 25:U5asbtLSgQpMkdk790SIYkNJ6yObywsEttFWW+5zmyov95t7EPHYpWc4Y/DcI9FTwdt1ycl+D6QbUnD/Log9Q2esgdTciZZNfF6uxvFMZli4uEemd5V/MeGuxZ1wfzMk4UDc/fITfEq+T0BRtyacUJRnpSmYezKXfD4uGs7zw9KpYbGUBiKt572o4iXndNN7LCkrJlGuHZE4xjVqS/yE9kGQkPevnsDiRa8IRWlWQMuoUOqBXIedgmYUKzWm2PrcdoU81FVHVtUh8fj9loiMZ+9Hn3dI1weMVxCePW1RvOxTmd2uQdOKC0S7bQDyxhjy5jp0WfJddaSPJCBuUHVfCQ==; 31:3eGR0A4OzBP4oK492hPqopaD4qpd0TejWXyQ0N/415a80hYYy/P4MT/HyzAuNrVl5bgWQ/YDEdv8IOGvlQ6CU8omEcQHjCFpqnaAypwnwsVjpiX5oG4ELnPxlrw7EgW66pMOruo9AD38oigF9JvB9LUbqKjAqt3Lddr1LvouU5cWfIKhmpuV9AKkoNs8qsU2ur4/hXF/WbepJt/A6zOmJ5jraiiX+dLGnReWAVTl+ok= X-MS-TrafficTypeDiagnostic: MWHPR07MB3470: X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3470; 20:wl1ongT7HN9nYVWr1DE/+R+FlUPHBmNyf67bDGsZUhxdONarnjz0d7E3UasO0/zW6cyYJDKofrL1Ral58W5IOQhiS8enwl5Onm2mSYYjlRoINJmbbDhnrP8c8s9znCrkrWy0sCPAEzjzFd4o3itJ4SH+UAgERhDxlMz6sAP1hvn0S+Y7CJon1NTKwFVcCKHeatd5dpNTaXbW9PjdTN3ot3pRjp0WxeyQ1gpuPrgqRzC+mla/2tPsWOMaIsi6UW/4G9XH4R5WgS50LBoyJel3YtPlSdmaENJ9e82prxLd9Cow4T+zO20CHY075Lbm5tWqq05X5+OfOs5xsHsPRaSDSlJIkLh1Hv3t0cd2bJPHEGPa/RWfpGJ19iEEiM5kzJPiJc1x3YqXqBaa9tXMvEqNnjYXC2uSX3P9LDa8D1AFrDjLnaxHGA2HGbL0mdf2yCNr5LPogcj3gsb+Ke8C6oLpTYlWPr2D8Szv0YpFl54H7Vcu82OweUJ4sL/ODy+2pXJFQcp0I5UH2yFSgpJz2SDWjdjlrOqjmJuDWqLZ7D9Rz1YNXnVG8LDE4q9BaY37+tyS6dXUhkv0G6uR3vk411tYJHbsAqUdF++0S7w0F/zpNNU=; 4:rMJLbGQxGhkF1rZAqh6c2IFeuyCyq4gHVoEfNOt+ZcV8M4KSEZesjKIYxxveQkskxUDmcCzz/WxNPuT5h7ZH2oF3OrLyTRM8NBW7fn26/fbPV8YxhMo9rmidYXjqXpNoGROMP5hH9g1eWXO3CY7dy3X11DM5ToJGTAU7yVo4ONka8HWOFU7ZxPh56QdsqYF0XgVP13h+CP5MKD1A91CBeJLjiKbaHKiGY2oTMvaY7kG7eqa6NsTGoui8Bu8yq6u1rf/Sb68nGNByjcbB/KADPw== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(3231221)(944501327)(52105095)(3002001)(10201501046)(93006095)(6041310)(20161123558120)(20161123562045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(6072148)(201708071742011); SRVR:MWHPR07MB3470; BCL:0; PCL:0; RULEID:; SRVR:MWHPR07MB3470; X-Forefront-PRVS: 0637FCE711 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6069001)(366004)(376002)(346002)(39860400002)(396003)(39380400002)(189003)(199004)(107886003)(48376002)(446003)(1857600001)(50466002)(97736004)(575784001)(66066001)(2906002)(53936002)(6512007)(42882007)(59450400001)(81166006)(956004)(81156014)(25786009)(16526019)(6116002)(2616005)(476003)(5660300001)(6486002)(3846002)(11346002)(8936002)(26005)(50226002)(8676002)(486006)(47776003)(1076002)(478600001)(105586002)(76176011)(68736007)(72206003)(7736002)(106356001)(5009440100003)(305945005)(4326008)(6506007)(316002)(16586007)(386003)(51416003)(52116002)(36756003)(42262002); DIR:OUT; SFP:1101; SCL:1; SRVR:MWHPR07MB3470; H:localhost.localdomain; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR07MB3470; 23:xshc/BZtseKi69REl0HkJ65Pi6q6mU/1XZGQYMIZu?= =?us-ascii?Q?HLzyyKHMqLmltpJL7EBbjb4aftohxu6i4RORUpGgtCsiM6vREIludJwRuiXI?= =?us-ascii?Q?M5FuIBzxlXuORjrN16sQyEfZN7s1W2PJ2p+Q883Bu4aq8C6Rd5OPyvaUZe//?= =?us-ascii?Q?B4zHgfepAiAlg0IMXCHO251YN3p5AFyJihIVVtIRfagiHVEECQwkKA7svsNk?= =?us-ascii?Q?eXX0CVQfPDCjvTDrMovvdhnS9UYhVTBsS4Y0ZPRqE2GsUgLzy6zCXi6Ne/uD?= =?us-ascii?Q?6tsXk6LY8GdGLBlNJyCSf9IxvxtFblLqAfV2ltFSnnwNlyocQ/JLoTeIce7E?= =?us-ascii?Q?hNK8OcWovpHs+s7cniASBMJ/eLjVBCap7HXpbPTzMWrZZO/UeBsqz+AYFDRo?= =?us-ascii?Q?wuZGJmj7Gt1ToTqn20f1865XjNeIUfG6LUrgZQ+MxnFgu8jMsYnpHIpH0iWn?= =?us-ascii?Q?Of1qKXCtQIPwJX0kgsP5k1ycTvoWNZZxSqIbjd48I3S2WXYWPf6R2VnzMxB7?= =?us-ascii?Q?ni+FDcs2XgfgbC66s5qioTnH2OUzanxR2l67Cyn20Sf0oq44d2CtPF55SQOs?= =?us-ascii?Q?lC6qLan7o9tQwoq96w3UukZzx2E5lAEoDGfJkzpBA/J3cbwgXWzlobCh+wiO?= =?us-ascii?Q?AdE0u3XVaer3ysb/4U/Biru4tQD0uMatguCrTAfThVzPGuhWSbBTePKt2H3J?= =?us-ascii?Q?Y57J2NB6QITE3kTOeMH9WRQcA80od/dlT6cIhWyjjP9+6pBUSjoPPqiRKVor?= =?us-ascii?Q?qrDkG+jtXNCeov8T4yzeawBN2NzTnuUPvLETX96ql4BzkedKipvurBV9sedP?= =?us-ascii?Q?KOOQgYkJMJnnW0la2Dh9rBq37oEVa83K8Cir6306Ljqo50zVmEVtTpG0iKX7?= =?us-ascii?Q?kgr1+mBNtfh0uor6p1CR1DhxheyIpI7OSSKTxzKfoOMyqr+3E5G3zneMi+dF?= =?us-ascii?Q?7Ox0IyMjwCtOi/oFvp0nopVjclvdK36YNyUKaK9atR2RHaO2dgxgHtRiZukH?= =?us-ascii?Q?ofF929Q8lA/XiiDGmlJLtshNVzkylypsGi1+wLiLrGc3x55Q0hvFb1EybEW/?= =?us-ascii?Q?ok02TLUqG6tu7vwh7DqGlKNC+nDRMfd25DJCPUtZuHqBQ55J5Coepm+3AeRt?= =?us-ascii?Q?9xiBMljm7vmtbxHOM/UAqGL4FbF6LRWJugolV1T73hL7ywRMcpmQB5+qWj0s?= =?us-ascii?Q?LaA1l7UCnYPu1WF780hXH4zXmrr5zm930fAT5AVHjijrwiatU2jua/PBiDfq?= =?us-ascii?Q?GB/LgyiXztgjJOtyWRFCXEUo9XbKvHNRubpSfpffKhnl0ZlvN1sS4sJVGC32?= =?us-ascii?Q?7R57G+bQ9XBIuE6T3yo99btfVGikMm/Ubnd0QYaaNn9?= X-Microsoft-Antispam-Message-Info: MQMNH/lEMDajWJmHtZKWrb4wcdkOIfR6WugfborjSHns5AU3ErZwZUYP/t1g1q0BxjOOktP6zMsOed5qJLC+n1cHhmWNschLfYpXs37GEvD0R1qEQfZxm9p2J/47iidBcCZ0BVSPDKpZbnUJyttIvdqB2D+WNvoyJNRiXVhWndkoXw3tnlv/GSaR3hYmeawL X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3470; 6:+vuVlMUXU8GBt9i7pDRxUpds7451xysbmRhkhosTAq3IgjGCbdMVF6YjqqqZTJdjTWKGUDtfu5VlM0V2DeX/iYypMZ54sQAlNj1L4zgWL9J5xZI6kObPBbQYgEiDXrDYSLclTt4WU3944B+wgXeILx3iwsl72TIKDRr7g0WyUvuny0W1UqhAbCAMYDTQzz4NjTVrjv2NqQ4YB4NnicSMzjrUQISEjNCqhzy8C0mxOtwNyLDAbcwcRke7hwKHQ3g3zYcNJa+XJh2+9199XsLnev8nFRYP3d5h1kKNzm02h0tpxsiEfgwIzYo2IqDwsOS2nnCxMGM3Qe67IioahI/sOdy6YRpO1AgSg82mjJ0MLwEa3Q63dIX6G6k33VAB+sN/Ydp6XrrKVDPKR1BGa2BYw8q+x09P6Ar1Z/DI6NKYVSUMEnC4qN5ScsIIvca4WptTgvKPRX1ztbOYW1e6xCgIqw==; 5:bY1KnN48dlv6LzDHDNJekVsWlmGJMGwS9FZvp+hSDa+lBMxJIwgpHe0Fhj24trxrLQSuNAP4dy0UHDDbD3L8Pb96KrLPRkd3kg6nLjYGkJ5b3+pvWQP65TVOkPupwzRHgvOVDGFUEV0DcV8YQ5YrZYZie1rx1iceKGajbVjxgAc=; 24:94bJ5bOvzz6QmLbmMGGkXPM4cTKIxrmWbpTjdkBkvIx9iCJSrNCP5rf4ibtsb5YvPeYBC5wUxI7LIu/Bcw63ymIuRYlqn2FK0SFKpycgHlA= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3470; 7:1kcU8FQvB901gfcNSuICc5appYfBYJ8nN0UE7ABdKYDBj1PVqCcsvhk4aPTEQeCJPPW/8yYkmzp3WAhiyB45kOypmmBWyudbkL7MqikjcAlu73Vshep+MUE/l1skB4lpmG1dJbibY4PXBD4sAD/VuXMYV7N119QSpe21G5lUJv7lMHQBwBxOdZB4Mr6hIx3Ec59ba4ETfoeJXbrwQirCenzDGxGFbCUZWK9Ahqki2JIg/lZDHPIkEX7B0FW0OHCW X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Apr 2018 21:01:32.3113 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 689f04b0-004d-4e50-d607-08d59e5d13c6 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR07MB3470 Subject: [dpdk-dev] [PATCH v4 06/11] event/octeontx: add multiproducer timer arm and cancel X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Apr 2018 21:01:37 -0000 Signed-off-by: Pavan Nikhilesh Acked-by: Jerin Jacob --- drivers/event/octeontx/Makefile | 5 + drivers/event/octeontx/meson.build | 1 + drivers/event/octeontx/timvf_evdev.c | 9 +- drivers/event/octeontx/timvf_evdev.h | 35 ++++ drivers/event/octeontx/timvf_worker.c | 107 ++++++++++ drivers/event/octeontx/timvf_worker.h | 279 ++++++++++++++++++++++++++ 6 files changed, 435 insertions(+), 1 deletion(-) create mode 100644 drivers/event/octeontx/timvf_worker.c create mode 100644 drivers/event/octeontx/timvf_worker.h diff --git a/drivers/event/octeontx/Makefile b/drivers/event/octeontx/Makefile index 8dbfd88c5..90ad2217f 100644 --- a/drivers/event/octeontx/Makefile +++ b/drivers/event/octeontx/Makefile @@ -30,20 +30,25 @@ SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_worker.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_evdev.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_evdev_selftest.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_probe.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += timvf_worker.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += timvf_evdev.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += timvf_probe.c ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y) CFLAGS_ssovf_worker.o += -fno-prefetch-loop-arrays +CFLAGS_timvf_worker.o += -fno-prefetch-loop-arrays ifeq ($(shell test $(GCC_VERSION) -ge 46 && echo 1), 1) CFLAGS_ssovf_worker.o += -Ofast +CFLAGS_timvf_worker.o += -Ofast else CFLAGS_ssovf_worker.o += -O3 -ffast-math +CFLAGS_timvf_worker.o += -O3 -ffast-math endif else CFLAGS_ssovf_worker.o += -Ofast +CFLAGS_timvf_worker.o += -Ofast endif include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/event/octeontx/meson.build b/drivers/event/octeontx/meson.build index 90af4fd1a..041855330 100644 --- a/drivers/event/octeontx/meson.build +++ b/drivers/event/octeontx/meson.build @@ -5,6 +5,7 @@ sources = files('ssovf_worker.c', 'ssovf_evdev.c', 'ssovf_evdev_selftest.c', 'ssovf_probe.c', + 'timvf_worker.c', 'timvf_evdev.c', 'timvf_probe.c' ) diff --git a/drivers/event/octeontx/timvf_evdev.c b/drivers/event/octeontx/timvf_evdev.c index c66db437e..5ffb460a8 100644 --- a/drivers/event/octeontx/timvf_evdev.c +++ b/drivers/event/octeontx/timvf_evdev.c @@ -130,6 +130,7 @@ timvf_ring_start(const struct rte_event_timer_adapter *adptr) timvf_write64((uintptr_t)timr->bkt, (uint8_t *)timr->vbar0 + TIM_VRING_BASE); + timvf_set_chunk_refill(timr); if (timvf_ring_conf_set(&rctrl, timr->tim_ring_id)) { ret = -EACCES; goto error; @@ -312,7 +313,13 @@ timvf_timer_adapter_caps_get(const struct rte_eventdev *dev, uint64_t flags, timvf_ops.stats_reset = timvf_stats_reset; } + if (enable_stats) + timvf_ops.arm_burst = timvf_timer_arm_burst_mp_stats; + else + timvf_ops.arm_burst = timvf_timer_arm_burst_mp; + + timvf_ops.cancel_burst = timvf_timer_cancel_burst; *caps = RTE_EVENT_TIMER_ADAPTER_CAP_INTERNAL_PORT; *ops = &timvf_ops; - return -EINVAL; + return 0; } diff --git a/drivers/event/octeontx/timvf_evdev.h b/drivers/event/octeontx/timvf_evdev.h index e89a43531..cae949c73 100644 --- a/drivers/event/octeontx/timvf_evdev.h +++ b/drivers/event/octeontx/timvf_evdev.h @@ -75,6 +75,33 @@ #define TIM_VRING_AURA (0x108) #define TIM_VRING_REL (0x110) +#define TIM_CTL1_W0_S_BUCKET 20 +#define TIM_CTL1_W0_M_BUCKET ((1ull << (40 - 20)) - 1) + +#define TIM_BUCKET_W1_S_NUM_ENTRIES (0) /*Shift*/ +#define TIM_BUCKET_W1_M_NUM_ENTRIES ((1ull << (32 - 0)) - 1) +#define TIM_BUCKET_W1_S_SBT (32) +#define TIM_BUCKET_W1_M_SBT ((1ull << (33 - 32)) - 1) +#define TIM_BUCKET_W1_S_HBT (33) +#define TIM_BUCKET_W1_M_HBT ((1ull << (34 - 33)) - 1) +#define TIM_BUCKET_W1_S_BSK (34) +#define TIM_BUCKET_W1_M_BSK ((1ull << (35 - 34)) - 1) +#define TIM_BUCKET_W1_S_LOCK (40) +#define TIM_BUCKET_W1_M_LOCK ((1ull << (48 - 40)) - 1) +#define TIM_BUCKET_W1_S_CHUNK_REMAINDER (48) +#define TIM_BUCKET_W1_M_CHUNK_REMAINDER ((1ull << (64 - 48)) - 1) + +#define TIM_BUCKET_SEMA \ + (TIM_BUCKET_CHUNK_REMAIN) + +#define TIM_BUCKET_CHUNK_REMAIN \ + (TIM_BUCKET_W1_M_CHUNK_REMAINDER << TIM_BUCKET_W1_S_CHUNK_REMAINDER) + +#define TIM_BUCKET_LOCK \ + (TIM_BUCKET_W1_M_LOCK << TIM_BUCKET_W1_S_LOCK) + +#define TIM_BUCKET_SEMA_WLOCK \ + (TIM_BUCKET_CHUNK_REMAIN | (1ull << TIM_BUCKET_W1_S_LOCK)) #define NSEC_PER_SEC 1E9 #define NSEC2CLK(__ns, __freq) (((__ns) * (__freq)) / NSEC_PER_SEC) @@ -168,5 +195,13 @@ void *timvf_bar(uint8_t id, uint8_t bar); int timvf_timer_adapter_caps_get(const struct rte_eventdev *dev, uint64_t flags, uint32_t *caps, const struct rte_event_timer_adapter_ops **ops, uint8_t enable_stats); +uint16_t timvf_timer_cancel_burst(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers); +uint16_t timvf_timer_arm_burst_mp(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers); +uint16_t timvf_timer_arm_burst_mp_stats( + const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers); +void timvf_set_chunk_refill(struct timvf_ring * const timr); #endif /* __TIMVF_EVDEV_H__ */ diff --git a/drivers/event/octeontx/timvf_worker.c b/drivers/event/octeontx/timvf_worker.c new file mode 100644 index 000000000..38b16d1ce --- /dev/null +++ b/drivers/event/octeontx/timvf_worker.c @@ -0,0 +1,107 @@ +/* + * SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2017 Cavium, Inc + */ + +#include "timvf_worker.h" + +static inline int +timvf_timer_reg_checks(const struct timvf_ring * const timr, + struct rte_event_timer * const tim) +{ + if (unlikely(tim->state)) { + tim->state = RTE_EVENT_TIMER_ERROR; + rte_errno = EALREADY; + goto fail; + } + + if (unlikely(!tim->timeout_ticks || + tim->timeout_ticks >= timr->nb_bkts)) { + tim->state = tim->timeout_ticks ? RTE_EVENT_TIMER_ERROR_TOOLATE + : RTE_EVENT_TIMER_ERROR_TOOEARLY; + rte_errno = EINVAL; + goto fail; + } + + return 0; +fail: + return -EINVAL; +} + +static inline void +timvf_format_event(const struct rte_event_timer * const tim, + struct tim_mem_entry * const entry) +{ + entry->w0 = (tim->ev.event & 0xFFC000000000) >> 6 | + (tim->ev.event & 0xFFFFFFFFF); + entry->wqe = tim->ev.u64; +} + +uint16_t +timvf_timer_cancel_burst(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers) +{ + RTE_SET_USED(adptr); + int ret; + uint16_t index; + + for (index = 0; index < nb_timers; index++) { + if (tim[index]->state == RTE_EVENT_TIMER_CANCELED) { + rte_errno = EALREADY; + break; + } + + if (tim[index]->state != RTE_EVENT_TIMER_ARMED) { + rte_errno = EINVAL; + break; + } + ret = timvf_rem_entry(tim[index]); + if (ret) { + rte_errno = -ret; + break; + } + } + return index; +} + +uint16_t +timvf_timer_arm_burst_mp(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers) +{ + int ret; + uint16_t index; + struct tim_mem_entry entry; + struct timvf_ring *timr = adptr->data->adapter_priv; + for (index = 0; index < nb_timers; index++) { + if (timvf_timer_reg_checks(timr, tim[index])) + break; + timvf_format_event(tim[index], &entry); + ret = timvf_add_entry_mp(timr, tim[index]->timeout_ticks, + tim[index], &entry); + if (unlikely(ret)) { + rte_errno = -ret; + break; + } + } + + return index; +} + +uint16_t +timvf_timer_arm_burst_mp_stats(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers) +{ + uint16_t ret; + struct timvf_ring *timr = adptr->data->adapter_priv; + + ret = timvf_timer_arm_burst_mp(adptr, tim, nb_timers); + timr->tim_arm_cnt += ret; + + return ret; +} + +void +timvf_set_chunk_refill(struct timvf_ring * const timr) +{ + timr->refill_chunk = timvf_refill_chunk_generic; +} diff --git a/drivers/event/octeontx/timvf_worker.h b/drivers/event/octeontx/timvf_worker.h new file mode 100644 index 000000000..c20d4d0cd --- /dev/null +++ b/drivers/event/octeontx/timvf_worker.h @@ -0,0 +1,279 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2017 Cavium, Inc + */ + +#include +#include + +#include "timvf_evdev.h" + +static inline int16_t +timr_bkt_fetch_rem(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_CHUNK_REMAINDER) & + TIM_BUCKET_W1_M_CHUNK_REMAINDER; +} + +static inline int16_t +timr_bkt_get_rem(struct tim_mem_bucket *bktp) +{ + return __atomic_load_n(&bktp->chunk_remainder, + __ATOMIC_ACQUIRE); +} + +static inline void +timr_bkt_set_rem(struct tim_mem_bucket *bktp, uint16_t v) +{ + __atomic_store_n(&bktp->chunk_remainder, v, + __ATOMIC_RELEASE); +} + +static inline void +timr_bkt_sub_rem(struct tim_mem_bucket *bktp, uint16_t v) +{ + __atomic_fetch_sub(&bktp->chunk_remainder, v, + __ATOMIC_RELEASE); +} + +static inline uint8_t +timr_bkt_get_sbt(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_SBT) & TIM_BUCKET_W1_M_SBT; +} + +static inline uint64_t +timr_bkt_set_sbt(struct tim_mem_bucket *bktp) +{ + const uint64_t v = TIM_BUCKET_W1_M_SBT << TIM_BUCKET_W1_S_SBT; + return __atomic_fetch_or(&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline uint64_t +timr_bkt_clr_sbt(struct tim_mem_bucket *bktp) +{ + const uint64_t v = ~(TIM_BUCKET_W1_M_SBT << TIM_BUCKET_W1_S_SBT); + return __atomic_fetch_and(&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline uint8_t +timr_bkt_get_shbt(uint64_t w1) +{ + return ((w1 >> TIM_BUCKET_W1_S_HBT) & TIM_BUCKET_W1_M_HBT) | + ((w1 >> TIM_BUCKET_W1_S_SBT) & TIM_BUCKET_W1_M_SBT); +} + +static inline uint8_t +timr_bkt_get_hbt(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_HBT) & TIM_BUCKET_W1_M_HBT; +} + +static inline uint8_t +timr_bkt_get_bsk(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_BSK) & TIM_BUCKET_W1_M_BSK; +} + +static inline uint64_t +timr_bkt_clr_bsk(struct tim_mem_bucket *bktp) +{ + /*Clear everything except lock. */ + const uint64_t v = TIM_BUCKET_W1_M_LOCK << TIM_BUCKET_W1_S_LOCK; + return __atomic_fetch_and(&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline uint64_t +timr_bkt_fetch_sema_lock(struct tim_mem_bucket *bktp) +{ + return __atomic_fetch_add(&bktp->w1, TIM_BUCKET_SEMA_WLOCK, + __ATOMIC_ACQ_REL); +} + +static inline uint64_t +timr_bkt_fetch_sema(struct tim_mem_bucket *bktp) +{ + return __atomic_fetch_add(&bktp->w1, TIM_BUCKET_SEMA, + __ATOMIC_RELAXED); +} + +static inline uint64_t +timr_bkt_inc_lock(struct tim_mem_bucket *bktp) +{ + const uint64_t v = 1ull << TIM_BUCKET_W1_S_LOCK; + return __atomic_fetch_add(&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline void +timr_bkt_dec_lock(struct tim_mem_bucket *bktp) +{ + __atomic_add_fetch(&bktp->lock, 0xff, __ATOMIC_ACQ_REL); +} + +static inline uint32_t +timr_bkt_get_nent(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_NUM_ENTRIES) & + TIM_BUCKET_W1_M_NUM_ENTRIES; +} + +static inline void +timr_bkt_inc_nent(struct tim_mem_bucket *bktp) +{ + __atomic_add_fetch(&bktp->nb_entry, 1, __ATOMIC_RELAXED); +} + +static inline void +timr_bkt_add_nent(struct tim_mem_bucket *bktp, uint32_t v) +{ + __atomic_add_fetch(&bktp->nb_entry, v, __ATOMIC_RELAXED); +} + +static inline uint64_t +timr_bkt_clr_nent(struct tim_mem_bucket *bktp) +{ + const uint64_t v = ~(TIM_BUCKET_W1_M_NUM_ENTRIES << + TIM_BUCKET_W1_S_NUM_ENTRIES); + return __atomic_and_fetch(&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline struct tim_mem_entry * +timr_clr_bkt(struct timvf_ring * const timr, struct tim_mem_bucket * const bkt) +{ + struct tim_mem_entry *chunk; + struct tim_mem_entry *pnext; + chunk = ((struct tim_mem_entry *)(uintptr_t)bkt->first_chunk); + chunk = (struct tim_mem_entry *)(uintptr_t)(chunk + nb_chunk_slots)->w0; + + while (chunk) { + pnext = (struct tim_mem_entry *)(uintptr_t) + ((chunk + nb_chunk_slots)->w0); + rte_mempool_put(timr->chunk_pool, chunk); + chunk = pnext; + } + return (struct tim_mem_entry *)(uintptr_t)bkt->first_chunk; +} + +static inline int +timvf_rem_entry(struct rte_event_timer *tim) +{ + uint64_t lock_sema; + struct tim_mem_entry *entry; + struct tim_mem_bucket *bkt; + if (tim->impl_opaque[1] == 0 || + tim->impl_opaque[0] == 0) + return -ENOENT; + + entry = (struct tim_mem_entry *)(uintptr_t)tim->impl_opaque[0]; + if (entry->wqe != tim->ev.u64) { + tim->impl_opaque[1] = tim->impl_opaque[0] = 0; + return -ENOENT; + } + bkt = (struct tim_mem_bucket *)(uintptr_t)tim->impl_opaque[1]; + lock_sema = timr_bkt_inc_lock(bkt); + if (timr_bkt_get_shbt(lock_sema) + || !timr_bkt_get_nent(lock_sema)) { + timr_bkt_dec_lock(bkt); + tim->impl_opaque[1] = tim->impl_opaque[0] = 0; + return -ENOENT; + } + + entry->w0 = entry->wqe = 0; + timr_bkt_dec_lock(bkt); + + tim->state = RTE_EVENT_TIMER_CANCELED; + tim->impl_opaque[1] = tim->impl_opaque[0] = 0; + return 0; +} + +static inline struct tim_mem_entry * +timvf_refill_chunk_generic(struct tim_mem_bucket * const bkt, + struct timvf_ring * const timr) +{ + struct tim_mem_entry *chunk; + + if (bkt->nb_entry || !bkt->first_chunk) { + if (unlikely(rte_mempool_get(timr->chunk_pool, + (void **)&chunk))) { + return NULL; + } + if (bkt->nb_entry) { + *(uint64_t *)(((struct tim_mem_entry *)(uintptr_t) + bkt->current_chunk) + + nb_chunk_slots) = + (uintptr_t) chunk; + } else { + bkt->first_chunk = (uintptr_t) chunk; + } + } else { + chunk = timr_clr_bkt(timr, bkt); + bkt->first_chunk = (uintptr_t)chunk; + } + *(uint64_t *)(chunk + nb_chunk_slots) = 0; + + return chunk; +} + +static inline struct tim_mem_bucket * +timvf_get_target_bucket(struct timvf_ring * const timr, const uint32_t rel_bkt) +{ + const uint64_t bkt_cyc = rte_rdtsc() - timr->ring_start_cyc; + const uint32_t bucket = rte_reciprocal_divide_u64(bkt_cyc, + &timr->fast_div) + rel_bkt; + const uint32_t tbkt_id = timr->get_target_bkt(bucket, + timr->nb_bkts); + return &timr->bkt[tbkt_id]; +} + +/* Multi producer functions. */ +static inline int +timvf_add_entry_mp(struct timvf_ring * const timr, const uint32_t rel_bkt, + struct rte_event_timer * const tim, + const struct tim_mem_entry * const pent) +{ + int16_t rem; + uint64_t lock_sema; + struct tim_mem_bucket *bkt; + struct tim_mem_entry *chunk; + +__retry: + bkt = timvf_get_target_bucket(timr, rel_bkt); + /* Bucket related checks. */ + /*Get Bucket sema*/ + lock_sema = timr_bkt_fetch_sema_lock(bkt); + if (unlikely(timr_bkt_get_shbt(lock_sema))) { + timr_bkt_dec_lock(bkt); + goto __retry; + } + + rem = timr_bkt_fetch_rem(lock_sema); + + if (rem < 0) { + /* goto diff bucket. */ + timr_bkt_dec_lock(bkt); + goto __retry; + } else if (!rem) { + /*Only one thread can be here*/ + chunk = timr->refill_chunk(bkt, timr); + if (unlikely(chunk == NULL)) { + timr_bkt_set_rem(bkt, 0); + timr_bkt_dec_lock(bkt); + tim->impl_opaque[0] = tim->impl_opaque[1] = 0; + tim->state = RTE_EVENT_TIMER_ERROR; + return -ENOMEM; + } + bkt->current_chunk = (uintptr_t) chunk; + timr_bkt_set_rem(bkt, nb_chunk_slots - 1); + } else { + chunk = (struct tim_mem_entry *)(uintptr_t)bkt->current_chunk; + chunk += nb_chunk_slots - rem; + } + /* Copy work entry. */ + *chunk = *pent; + timr_bkt_inc_nent(bkt); + timr_bkt_dec_lock(bkt); + + tim->impl_opaque[0] = (uintptr_t)chunk; + tim->impl_opaque[1] = (uintptr_t)bkt; + tim->state = RTE_EVENT_TIMER_ARMED; + return 0; +} -- 2.17.0