From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-by2nam01on0056.outbound.protection.outlook.com [104.47.34.56]) by dpdk.org (Postfix) with ESMTP id CF6E94F90 for ; Tue, 3 Apr 2018 17:05:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=h5K1lnr4N2WPt9q89J2+v8SI5zGOK1L5Gm+EHywIlac=; b=WXVqlmZNWAv7H0d5RHkQWI+N5wBzN+y64TO3zU54dY5ku1K1OAyPIhvVqYrX9yt2j6hv8FaRlmw18AIUTSOEOVHT/+erlHlptqhbrd/oi9FlnuST0d+YjThwO+Kj0g9awHH2HC6Yc17E5G/cRts9XHtILoZMacg2SX+p0GwsETg= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Pavan.Bhagavatula@cavium.com; Received: from ltp-pvn.caveonetworks.com (111.93.218.67) by MWHPR07MB3470.namprd07.prod.outlook.com (2603:10b6:301:63::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.631.10; Tue, 3 Apr 2018 15:05:55 +0000 From: Pavan Nikhilesh To: jerin.jacob@caviumnetworks.com, santosh.shukla@caviumnetworks.com, erik.g.carrillo@intel.com Cc: dev@dpdk.org, Pavan Nikhilesh Date: Tue, 3 Apr 2018 20:35:08 +0530 Message-Id: <20180403150514.24201-7-pbhagavatula@caviumnetworks.com> X-Mailer: git-send-email 2.16.3 In-Reply-To: <20180403150514.24201-1-pbhagavatula@caviumnetworks.com> References: <20180216213700.3415-1-pbhagavatula@caviumnetworks.com> <20180403150514.24201-1-pbhagavatula@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: BM1PR0101CA0070.INDPRD01.PROD.OUTLOOK.COM (2603:1096:b00:19::32) To MWHPR07MB3470.namprd07.prod.outlook.com (2603:10b6:301:63::21) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c99b684a-8e6c-4ee5-e33c-08d59974673e X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(2017052603328)(7153060)(7193020); SRVR:MWHPR07MB3470; X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3470; 3:qPigmmAs8Fb3uC0BOh2fpCGBjF8TZsNwN++zP1w6ZNn+D7I2+1NjlAsRbx+cn5jDyar9lXIYFWDz3QtR9g3W88vsUvgGljA55wMCWQg65BOZpIAiYESoz+jnxtu7BTVMN9+DdewvXRc29Vf7riOC6bV49ZncA+6pmcSBRj+aPfrMQNZtWhVGsmBalPP3s9nR4yt8asdt53Wiv7aNMt39YjFZV7vN+T8saDyDxoPmkckROY6hPzIYxHik36w+dbGP; 25:sbIscdt+2Bx9K2tVwQxx91Lijp3iwDT3Z09V4jnZlxLVUWnmCam3yPeZbnb3tu9LS366JVdeGvrOVO63klDiDd2/PKA2zPJLLP6vy9MrP2qGCdbWevMoUmlfc+bWSJs9kDKOiCXu9W3vcLHnIcYRu2u7PftW6Tp2o7AR8b+VHUWga9w11FRoGhcdVu3PJlaxMprSGCjmDbWUng9v7C0yyRh3wchsSYoHm2peEkLvnUG8MheZ5hozXR60PBy+XyhWAxO1gJkPmB5knPpg2kd++5hSR/eEFaGa8SWMoqvntLBH5iUX8SCixgijfXgzIePyGgO+55NqmjDcgMTEwjujhQ==; 31:mfd8XTfwLF2sZas3wdX9/NCN1cuIljOTQEYjRVJSrPBHuD7ryFF9oXVzqL3+5aOFwOU0AeCf6drdhqh6T8l7bTPUFQsdBLN459R+J7nvBeVkUcyYpp/cTrPIksztNojN4DEJWpopaH5Pi06oexXOnCLDeE4nLIrCCJsirIT3ZO3g0TkhO1KeTJPqLiYg0nCTLb7urJ2fc351nEYfpa0VPeclqBX88WVJmjdUV+KxTo8= X-MS-TrafficTypeDiagnostic: MWHPR07MB3470: X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3470; 20:3tZd9owSb3+H6X6lgnEasyfeCzFoC5P+n3kJn5IY8gZNdsL4BbWLVPDNwRreLxRxQ5iRSUvebeQCUiQxryj1z8Fnj8hNPC2c6s+khzKzwarVkMp0ZpbUutxZ6wov2AXFVpV66Vcmliv6F2lb7ggrJ2hZFCMuIaOGnXf4O5U0GFoMrVzPSFlmukD5+DAvFGHctLLVyA1LaWFI+Jo01ZBpyJTfud9YUW4DaqpySyHHxpy4CO2+iRSwe0POQYmDFOq0BODnk3IpTcn21qukWH7BcZ0fluoU6r7Z9BMzmUEVDlotC/zFdrTCmHc1al0gSt0g0lUL6F1tCZh9zFYXHUhzbnApmxlpCM5lmsc8zhDuPI8U4T6NwThecWaPtieFhZsNwUg8ywYbbn/jr06wDOk6/W+f1FkBgqgE3JU67unkXOImROkqN9w7pUT0QVRZ9KPTZRLYln8ZXRSMBT8N/v0VUHwtVj+DPE8i2Ix8rwRp17uY6HhdTdrULgnW0Ei3T3SPEkDp9hwj0Jytd3WnPyMrB/yxDbYoMSoefBeKh1YTXlKzf0sm6HJvLu1U25KyL3OxkYKS0I7VNtr+mZitVdxIu56D/3YI9JfQuO4u0Gj7Wjw=; 4:E1HWoCP2nx8lO1eQnd1SfOMDO5bN3/Cby/08SFa1v+XYS6olGRKQcfR7/D+bgOrsG76RXjeRTugNlftcppLpykKwi/C52j+zyqkN8+WfL88DqqTrl9WUS3s63pKMx3HzVXTlswiFuiT8rX7dWzE5MF4MHjYJi2r1s3VvkWhoX2vVfllmXaBBFonPj4iZCjazT2PMVu1rGw5Rlt5H0bkGrFYE8zy6rWHvrWuItXvV8sAVVtGpPFE/xrwRxIWTWxvlDoWMDJQ9clitIJFI1ON4Zw== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(3231221)(944501327)(52105095)(10201501046)(3002001)(93006095)(6041310)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123558120)(20161123564045)(6072148)(201708071742011); SRVR:MWHPR07MB3470; BCL:0; PCL:0; RULEID:; SRVR:MWHPR07MB3470; X-Forefront-PRVS: 0631F0BC3D X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(396003)(39380400002)(346002)(39850400004)(376002)(366004)(189003)(199004)(47776003)(305945005)(66066001)(72206003)(386003)(6506007)(16586007)(478600001)(316002)(48376002)(7736002)(76176011)(59450400001)(36756003)(52116002)(51416003)(5009440100003)(50466002)(68736007)(106356001)(4326008)(25786009)(486005)(476003)(2906002)(107886003)(6512007)(11346002)(6116002)(2616005)(53416004)(26005)(16526019)(5660300001)(6486002)(3846002)(1076002)(81166006)(81156014)(446003)(956004)(486005)(69596002)(8676002)(6666003)(97736004)(42882007)(50226002)(105586002)(8936002)(1857600001)(53936002)(42262002); DIR:OUT; SFP:1101; SCL:1; SRVR:MWHPR07MB3470; H:ltp-pvn.caveonetworks.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR07MB3470; 23:0pINHqTVZzIsqTNYbFXM24kcegFcLgSAoDh9b+kgF?= =?us-ascii?Q?oOT0AKYZ8+qaZWvBIMcFPpfjWsI2rWk9QPYu8pd16Lkv7GZkKPSJGLySico2?= =?us-ascii?Q?lL+ktH3gEb13T/M4s6HHkz9mEGnd0Q1s8nrX6jRCA8IydP8BPYfvVieA2eCF?= =?us-ascii?Q?OLQyBAqaw7K5KNjfJQvH9YrHeZGWdTmodbmHYZZf9VfRpaju4Rmro8fiQR9+?= =?us-ascii?Q?M+p7gV1XYeEG4LXVr0m9E24sNthq0CMwn27DwQVikDepd5ES8caAtCwVczQq?= =?us-ascii?Q?WKbqi47u6UYz3x1E/f1dx4pnxKKqfiwk5VyJLN+U6AF/8FKgYasOdvJW7Avl?= =?us-ascii?Q?yLCHpZKxCcd1oVMpLFxgadiY5AeFPLZbcME7GCqtsMng2eFnkO06vSbA43nC?= =?us-ascii?Q?k5zRM3BpzCqAYKZELFmMjf+Ada5L2KSajKhCGbCBI8uswzb3eqT0539vBQFO?= =?us-ascii?Q?KLq4rOnyYXSJpkytAPH+UyL6aie/AU6YAoGwjca5GctevTD7uvF1UeLi935V?= =?us-ascii?Q?IQwRwwWSs2jdMfcrrd03sSVip7K2NFntk3UrXAGYQ8uUnQ0trLCZ2nX4/e6A?= =?us-ascii?Q?BswRVET1XE2ArO6VGm3yO4t4TS69DdRuoSRWw5sDIUYBQrbvJvphgxVW6A6E?= =?us-ascii?Q?fIpLiXKf+R61OTSRFfdEFQ75T/jgu9V1bfskrWW5mvVSaukhGqkmNk6B/51X?= =?us-ascii?Q?Mk/GOC48pi1Ms7cIeSSZHZYEi9lnK6lnmKcxAepeqIDF6DAEwGRSb63VnfB7?= =?us-ascii?Q?u/OedEML0k/+MtmGQRTsa0as5Rvjt5YAWwmQM76U0VfOKczVRiFCRLLFL5uT?= =?us-ascii?Q?BnH9EAwkDQ1r2I4FnARlsakzypROG3Mo2Iuo/4AHEaT/mVQDPILc6wmOZ2GH?= =?us-ascii?Q?EEtMKKfkc9uQ/BYJkcTxHgUmmYcr+R3hhRCiv9RcMNHvw0+k0jM1nilh4NNe?= =?us-ascii?Q?qsb0+F+PvOOBWsyAhkK2agZsHdLx3YXodClobnlWiaLA1dafY5aNafHo3ec/?= =?us-ascii?Q?PjTeLNsYQmxzOibi2KZADvLL129U/2ts0+vYKPSeqdyHMGGC0aYv+ljgwTcV?= =?us-ascii?Q?1sTGOaldIvnob+5EUq2ivr0oN8BNwVP6m11+GEcLNxFgppajnPys+c6qqSUn?= =?us-ascii?Q?MesFrOiqWFfMSnWgaOFaEX8ijLNHRZa4gsXtqHpzEtqdrBlegbJUXa/9HQie?= =?us-ascii?Q?MRXD9immEvqE3rW19cbEOcLBYFThJE53f4VO977JHskIpA8jWmyym8X30QmA?= =?us-ascii?Q?lboWNQjol644wqM7NDdDMmT5iVsMcS/Y90AsNSZVTOU+Vz9nTtz8rNOZqu8S?= =?us-ascii?Q?D3f+NuGAhKr/AP19GaR0sU+QpIKbxabp8U1qcKyKL1bRmkuipCV1gBmR/AtJ?= =?us-ascii?Q?atcH71ygas7ig4o8Hc8NodZ028=3D?= X-Microsoft-Antispam-Message-Info: 5x3krlnl6vicrDFFHDFdKFPwEgm4Z5bYj0FJk4VfG9NMd5bkEMUOwWOCqVfVY4m/PfKgiv5iWr1d463mrWbvD4Q+RP4a8FEfcJmVaEO7dDSaLkgM01WJnbr9x9EzidzwYMxXEX83aBQ+/kmKWOQWJMvwKiBJm5f5zIgRjBL45k4AOC12+pjO+X48boJ2qoRJ X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3470; 6:cbd2zgQF/D8zoOg0veHVMVDY6c8RX6mQT0DWTrrsPeTN+MI+0jsNv9B0kgFjF/GlMl8sigD47FCMduKjZkfzhHwnz7SBGW6DwlYECYBTxmtzEXW+gqQ4I1WbABWNEvzKVTNpmIK0JNywVXXu+6XU8+Id9SiKPrB+FRVKnh0XmY/8PAwwg1vr5gXoEguz+tsTewui3I4PcxiGsbQCIUwtZPf0nnY0+xl5Cp4P5UooRPjeILer2FSLCsBtSVdwfAouoNKVTPRXLxiKN/w7r862kPpD1EBJWNKPrhIk2IXfNK6t5SwlcSYTskVTW5FWIXzSPozOoKm50ci5sKxMoaqk7ZrsmR8azHblOGXr5ZIIwi2HQ7h1VYfp590afaHYTZcym+1F7W+WIu1AsO4e5w8I0cr2bX8ubM0l0eqQBPHqmLZXX2vqtDaedLzojvAQELIhnBn6qJFDAE09TZI/qC9eiw==; 5:/JHjJM9GTJxpmxwYJvIbBcap0Ww3PovVocfuz9b8mnUeYwBBJFDiV43Aa4P9fh9H7/wiHBo0hC9R8LUBk8OngYvCNGbf6ztXlgSdnZclPUjDZyi9+jnii34dsWr2j72dANcyUpadaADrCuSgc7a2gnuLyMVAjcci8kWUX3G4VeQ=; 24:UgDd29Q/9qbpuwZ0aj+tcXSH0yQ28+mqXV3p7lkkWfKkqgxU56U61tSiPHyA1mvnqPX5ZiI5LdPc0XOWJL1Xeax72BFdSfz+duHh0rvAOFY= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3470; 7:sFsCsetDkTMsOaUyZQtNlriIG79Y1N0fGpX9pEu9pvMefRSF8S72EtyUCilTbIkjBss7UmU7ujZJ81wV/qDBDvryywxX1s8N+7EiMujgiK7CimcRVfWwoiZATChAmITQwZP5VgInvmbtKktnXj/trDbmbMrgtUOlGJcVw2TDju81zC2jSZyjSzJ2kTwIdg04FZODpi4w3OTcEEOijIX92kwPN7hjyKH/hNfHCag7TSaOET+2otv+PWOqh+vBbEws X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2018 15:05:55.2411 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c99b684a-8e6c-4ee5-e33c-08d59974673e X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR07MB3470 Subject: [dpdk-dev] [PATCH v3 06/12] event/octeontx: add multiproducer timer arm and cancel X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Apr 2018 15:05:59 -0000 Signed-off-by: Pavan Nikhilesh --- drivers/event/octeontx/Makefile | 5 + drivers/event/octeontx/meson.build | 1 + drivers/event/octeontx/timvf_evdev.c | 7 +- drivers/event/octeontx/timvf_evdev.h | 32 ++++ drivers/event/octeontx/timvf_worker.c | 95 ++++++++++++ drivers/event/octeontx/timvf_worker.h | 280 ++++++++++++++++++++++++++++++++++ 6 files changed, 419 insertions(+), 1 deletion(-) create mode 100644 drivers/event/octeontx/timvf_worker.c create mode 100644 drivers/event/octeontx/timvf_worker.h diff --git a/drivers/event/octeontx/Makefile b/drivers/event/octeontx/Makefile index 570a79412..58fa6af88 100644 --- a/drivers/event/octeontx/Makefile +++ b/drivers/event/octeontx/Makefile @@ -28,19 +28,24 @@ LIBABIVER := 1 SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_worker.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_evdev.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += ssovf_evdev_selftest.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += timvf_worker.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF) += timvf_evdev.c ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y) CFLAGS_ssovf_worker.o += -fno-prefetch-loop-arrays +CFLAGS_timvf_worker.o += -fno-prefetch-loop-arrays ifeq ($(shell test $(GCC_VERSION) -ge 46 && echo 1), 1) CFLAGS_ssovf_worker.o += -Ofast +CFLAGS_timvf_worker.o += -Ofast else CFLAGS_ssovf_worker.o += -O3 -ffast-math +CFLAGS_timvf_worker.o += -O3 -ffast-math endif else CFLAGS_ssovf_worker.o += -Ofast +CFLAGS_timvf_worker.o += -Ofast endif include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/event/octeontx/meson.build b/drivers/event/octeontx/meson.build index 8941f8a56..efac94cba 100644 --- a/drivers/event/octeontx/meson.build +++ b/drivers/event/octeontx/meson.build @@ -4,6 +4,7 @@ sources = files('ssovf_worker.c', 'ssovf_evdev.c', 'ssovf_evdev_selftest.c', + 'timvf_worker.c', 'timvf_evdev.c', ) diff --git a/drivers/event/octeontx/timvf_evdev.c b/drivers/event/octeontx/timvf_evdev.c index 4db10cdd0..c8865e33c 100644 --- a/drivers/event/octeontx/timvf_evdev.c +++ b/drivers/event/octeontx/timvf_evdev.c @@ -126,6 +126,7 @@ timvf_ring_start(const struct rte_event_timer_adapter *adptr) timvf_write64((uint64_t)timr->meta.bkt, (uint8_t *)timr->vbar0 + TIM_VRING_BASE); + timvf_set_chunk_refill(timr); if (timvf_ring_conf_set(&rctrl, timr->tim_ring_id)) { ret = -EACCES; goto error; @@ -321,7 +322,11 @@ timvf_timer_adapter_caps_get(const struct rte_eventdev *dev, uint64_t flags, { RTE_SET_USED(dev); RTE_SET_USED(flags); + + timvf_ops.arm_burst = timvf_timer_reg_burst_mp; + timvf_ops.cancel_burst = timvf_timer_unreg_burst; + *caps = RTE_EVENT_TIMER_ADAPTER_CAP_INTERNAL_PORT; *ops = &timvf_ops; - return -EINVAL; + return 0; } diff --git a/drivers/event/octeontx/timvf_evdev.h b/drivers/event/octeontx/timvf_evdev.h index e98ab631a..ddf50643f 100644 --- a/drivers/event/octeontx/timvf_evdev.h +++ b/drivers/event/octeontx/timvf_evdev.h @@ -75,6 +75,33 @@ #define TIM_VRING_AURA (0x108) #define TIM_VRING_REL (0x110) +#define TIM_CTL1_W0_S_BUCKET 20 +#define TIM_CTL1_W0_M_BUCKET ((1ull << (40 - 20)) - 1) + +#define TIM_BUCKET_W1_S_NUM_ENTRIES (0) /*Shift*/ +#define TIM_BUCKET_W1_M_NUM_ENTRIES ((1ull << (32 - 0)) - 1) +#define TIM_BUCKET_W1_S_SBT (32) +#define TIM_BUCKET_W1_M_SBT ((1ull << (33 - 32)) - 1) +#define TIM_BUCKET_W1_S_HBT (33) +#define TIM_BUCKET_W1_M_HBT ((1ull << (34 - 33)) - 1) +#define TIM_BUCKET_W1_S_BSK (34) +#define TIM_BUCKET_W1_M_BSK ((1ull << (35 - 34)) - 1) +#define TIM_BUCKET_W1_S_LOCK (40) +#define TIM_BUCKET_W1_M_LOCK ((1ull << (48 - 40)) - 1) +#define TIM_BUCKET_W1_S_CHUNK_REMAINDER (48) +#define TIM_BUCKET_W1_M_CHUNK_REMAINDER ((1ull << (64 - 48)) - 1) + +#define TIM_BUCKET_SEMA \ + (TIM_BUCKET_CHUNK_REMAIN) + +#define TIM_BUCKET_CHUNK_REMAIN \ + (TIM_BUCKET_W1_M_CHUNK_REMAINDER << TIM_BUCKET_W1_S_CHUNK_REMAINDER) + +#define TIM_BUCKET_LOCK \ + (TIM_BUCKET_W1_M_LOCK << TIM_BUCKET_W1_S_LOCK) + +#define TIM_BUCKET_SEMA_WLOCK \ + (TIM_BUCKET_CHUNK_REMAIN | (1ull << TIM_BUCKET_W1_S_LOCK)) #define NSEC_PER_SEC 1E9 #define NSEC2CLK(__ns, __freq) (((__ns) * (__freq)) / NSEC_PER_SEC) @@ -167,5 +194,10 @@ bkt_mod(const uint32_t rel_bkt, const uint32_t nb_bkts) int timvf_timer_adapter_caps_get(const struct rte_eventdev *dev, uint64_t flags, uint32_t *caps, const struct rte_event_timer_adapter_ops **ops); +uint16_t timvf_timer_unreg_burst(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers); +uint16_t timvf_timer_reg_burst_mp(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers); +void timvf_set_chunk_refill(struct timvf_ring * const timr); #endif /* __TIMVF_EVDEV_H__ */ diff --git a/drivers/event/octeontx/timvf_worker.c b/drivers/event/octeontx/timvf_worker.c new file mode 100644 index 000000000..50794d37b --- /dev/null +++ b/drivers/event/octeontx/timvf_worker.c @@ -0,0 +1,95 @@ +/* + * SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2017 Cavium, Inc + */ + +#include "timvf_worker.h" + +static inline int +timvf_timer_reg_checks(const struct timvf_ring * const timr, + struct rte_event_timer * const tim) +{ + if (unlikely(tim->state)) { + tim->state = RTE_EVENT_TIMER_ERROR; + rte_errno = EALREADY; + goto fail; + } + + if (unlikely(!tim->timeout_ticks || + tim->timeout_ticks >= timr->meta.nb_bkts)) { + tim->state = tim->timeout_ticks ? RTE_EVENT_TIMER_ERROR_TOOLATE + : RTE_EVENT_TIMER_ERROR_TOOEARLY; + rte_errno = EINVAL; + goto fail; + } + + return 0; +fail: + return -EINVAL; +} + +static inline void +timvf_format_event(const struct rte_event_timer * const tim, + struct tim_mem_entry * const entry) +{ + entry->w0 = (tim->ev.event & 0xFFC000000000) >> 6 | + (tim->ev.event & 0xFFFFFFFFF); + entry->wqe = tim->ev.u64; +} + +uint16_t +timvf_timer_unreg_burst(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers) +{ + RTE_SET_USED(adptr); + int ret; + uint16_t index; + + for (index = 0; index < nb_timers; index++) { + if (tim[index]->state == RTE_EVENT_TIMER_CANCELED) { + rte_errno = EALREADY; + break; + } + + if (tim[index]->state != RTE_EVENT_TIMER_ARMED) { + rte_errno = EINVAL; + break; + } + ret = timvf_rem_entry(tim[index]); + if (ret) { + rte_errno = -ret; + break; + } + } + return index; +} + +uint16_t +timvf_timer_reg_burst_mp(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers) +{ + int ret; + uint16_t index; + struct tim_mem_entry entry; + struct timvf_ring *timr = adptr->data->adapter_priv; + for (index = 0; index < nb_timers; index++) { + if (timvf_timer_reg_checks(timr, tim[index])) + break; + timvf_format_event(tim[index], &entry); + ret = timvf_add_entry_mp(timr, tim[index]->timeout_ticks, + tim[index], &entry); + if (unlikely(ret)) { + rte_errno = -ret; + break; + } + } + timr->meta.tim_arm_cnt += index; + + return index; +} + +void +timvf_set_chunk_refill(struct timvf_ring * const timr) +{ + timr->meta.refill_chunk = timvf_refill_chunk_generic; +} diff --git a/drivers/event/octeontx/timvf_worker.h b/drivers/event/octeontx/timvf_worker.h new file mode 100644 index 000000000..bba4985ab --- /dev/null +++ b/drivers/event/octeontx/timvf_worker.h @@ -0,0 +1,280 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2017 Cavium, Inc + */ + +#include +#include + +#include "timvf_evdev.h" + +static inline int16_t +timr_bkt_fetch_rem(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_CHUNK_REMAINDER) & + TIM_BUCKET_W1_M_CHUNK_REMAINDER; +} + +static inline int16_t +timr_bkt_get_rem(struct tim_mem_bucket *bktp) +{ + return __atomic_load_n((int16_t *)&bktp->chunk_remainder, + __ATOMIC_ACQUIRE); +} + +static inline void +timr_bkt_set_rem(struct tim_mem_bucket *bktp, uint16_t v) +{ + __atomic_store_n((int16_t *)&bktp->chunk_remainder, v, + __ATOMIC_RELEASE); +} + +static inline void +timr_bkt_sub_rem(struct tim_mem_bucket *bktp, uint16_t v) +{ + __atomic_fetch_sub((int16_t *)&bktp->chunk_remainder, v, + __ATOMIC_RELEASE); +} + +static inline uint8_t +timr_bkt_get_sbt(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_SBT) & TIM_BUCKET_W1_M_SBT; +} + +static inline uint64_t +timr_bkt_set_sbt(struct tim_mem_bucket *bktp) +{ + const uint64_t v = TIM_BUCKET_W1_M_SBT << TIM_BUCKET_W1_S_SBT; + return __atomic_fetch_or((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline uint64_t +timr_bkt_clr_sbt(struct tim_mem_bucket *bktp) +{ + const uint64_t v = ~(TIM_BUCKET_W1_M_SBT << TIM_BUCKET_W1_S_SBT); + return __atomic_fetch_and((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline uint8_t +timr_bkt_get_shbt(uint64_t w1) +{ + return ((w1 >> TIM_BUCKET_W1_S_HBT) & TIM_BUCKET_W1_M_HBT) | + ((w1 >> TIM_BUCKET_W1_S_SBT) & TIM_BUCKET_W1_M_SBT); +} + +static inline uint8_t +timr_bkt_get_hbt(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_HBT) & TIM_BUCKET_W1_M_HBT; +} + +static inline uint8_t +timr_bkt_get_bsk(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_BSK) & TIM_BUCKET_W1_M_BSK; +} + +static inline uint64_t +timr_bkt_clr_bsk(struct tim_mem_bucket *bktp) +{ + /*Clear everything except lock. */ + const uint64_t v = TIM_BUCKET_W1_M_LOCK << TIM_BUCKET_W1_S_LOCK; + return __atomic_fetch_and((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline uint64_t +timr_bkt_fetch_sema_lock(struct tim_mem_bucket *bktp) +{ + return __atomic_fetch_add((uint64_t *)&bktp->w1, TIM_BUCKET_SEMA_WLOCK, + __ATOMIC_ACQ_REL); +} + +static inline uint64_t +timr_bkt_fetch_sema(struct tim_mem_bucket *bktp) +{ + return __atomic_fetch_add((uint64_t *)&bktp->w1, TIM_BUCKET_SEMA, + __ATOMIC_RELAXED); +} + +static inline uint64_t +timr_bkt_inc_lock(struct tim_mem_bucket *bktp) +{ + const uint64_t v = 1ull << TIM_BUCKET_W1_S_LOCK; + return __atomic_fetch_add((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline void +timr_bkt_dec_lock(struct tim_mem_bucket *bktp) +{ + __atomic_add_fetch((uint8_t *)&bktp->lock, 0xff, __ATOMIC_ACQ_REL); +} + +static inline uint32_t +timr_bkt_get_nent(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_NUM_ENTRIES) & + TIM_BUCKET_W1_M_NUM_ENTRIES; +} + +static inline void +timr_bkt_inc_nent(struct tim_mem_bucket *bktp) +{ + __atomic_add_fetch((uint32_t *)&bktp->nb_entry, 1, __ATOMIC_RELAXED); +} + +static inline void +timr_bkt_add_nent(struct tim_mem_bucket *bktp, uint32_t v) +{ + __atomic_add_fetch((uint32_t *)&bktp->nb_entry, v, __ATOMIC_RELAXED); +} + +static inline uint64_t +timr_bkt_clr_nent(struct tim_mem_bucket *bktp) +{ + const uint64_t v = ~(TIM_BUCKET_W1_M_NUM_ENTRIES << + TIM_BUCKET_W1_S_NUM_ENTRIES); + return __atomic_and_fetch((uint64_t *)&bktp->w1, v, __ATOMIC_ACQ_REL); +} + +static inline struct tim_mem_entry * +timr_clr_bkt(struct timvf_ring * const timr, struct tim_mem_bucket * const bkt) +{ + struct tim_mem_entry *chunk; + struct tim_mem_entry *pnext; + chunk = ((struct tim_mem_entry *)bkt->first_chunk); + chunk = (struct tim_mem_entry *)(chunk + nb_chunk_slots)->w0; + + while (chunk) { + pnext = (struct tim_mem_entry *)((chunk + nb_chunk_slots)->w0); + rte_mempool_put(timr->meta.chunk_pool, chunk); + chunk = pnext; + } + return (struct tim_mem_entry *)bkt->first_chunk; +} + +static inline int +timvf_rem_entry(struct rte_event_timer *tim) +{ + uint64_t lock_sema; + struct tim_mem_entry *entry; + struct tim_mem_bucket *bkt; + if (tim->impl_opaque[1] == 0 || + tim->impl_opaque[0] == 0) + return -ENOENT; + + entry = (struct tim_mem_entry *)tim->impl_opaque[0]; + if (entry->wqe != tim->ev.u64) { + tim->impl_opaque[1] = tim->impl_opaque[0] = 0; + return -ENOENT; + } + bkt = (struct tim_mem_bucket *)tim->impl_opaque[1]; + lock_sema = timr_bkt_inc_lock(bkt); + if (timr_bkt_get_shbt(lock_sema) + || !timr_bkt_get_nent(lock_sema)) { + timr_bkt_dec_lock(bkt); + tim->impl_opaque[1] = tim->impl_opaque[0] = 0; + return -ENOENT; + } + + entry->w0 = entry->wqe = 0; + timr_bkt_dec_lock(bkt); + + tim->state = RTE_EVENT_TIMER_CANCELED; + tim->impl_opaque[1] = tim->impl_opaque[0] = 0; + return 0; +} + +static inline struct tim_mem_entry * +timvf_refill_chunk_generic(struct tim_mem_bucket * const bkt, + struct timvf_ring * const timr) +{ + struct tim_mem_entry *chunk; + + if (bkt->nb_entry || !bkt->first_chunk) { + if (unlikely(rte_mempool_get(timr->meta.chunk_pool, + (void **)&chunk))) { + return NULL; + } + if (bkt->nb_entry) { + *(uint64_t *)((struct tim_mem_entry *) + bkt->current_chunk + + nb_chunk_slots) = + (uint64_t) chunk; + } else { + bkt->first_chunk = (uint64_t) chunk; + } + } else { + chunk = timr_clr_bkt(timr, bkt); + bkt->first_chunk = (uint64_t) chunk; + } + *(uint64_t *)(chunk + nb_chunk_slots) = 0; + + return chunk; +} + +static inline struct tim_mem_bucket * +timvf_get_target_bucket(struct timvf_ring * const timr, const uint32_t rel_bkt) +{ + const uint64_t bkt_cyc = rte_rdtsc() - timr->meta.ring_start_cyc; + const uint32_t bucket = rte_reciprocal_divide_u64(bkt_cyc, + &timr->meta.fast_div) + rel_bkt; + const uint32_t tbkt_id = timr->meta.get_target_bkt(bucket, + timr->meta.nb_bkts); + return &timr->meta.bkt[tbkt_id]; +} + +/* Multi producer functions. */ +static inline int +timvf_add_entry_mp(struct timvf_ring * const timr, const uint32_t rel_bkt, + struct rte_event_timer * const tim, + const struct tim_mem_entry * const pent) +{ + uint8_t lock_cnt; + int16_t rem; + uint64_t lock_sema; + struct tim_mem_bucket *bkt; + struct tim_mem_entry *chunk; + +__retry: + bkt = timvf_get_target_bucket(timr, rel_bkt); + /* Bucket related checks. */ + /*Get Bucket sema*/ + lock_sema = timr_bkt_fetch_sema_lock(bkt); + if (unlikely(timr_bkt_get_shbt(lock_sema))) { + timr_bkt_dec_lock(bkt); + goto __retry; + } + + RTE_SET_USED(lock_cnt); + rem = timr_bkt_fetch_rem(lock_sema); + + if (rem < 0) { + /* goto diff bucket. */ + timr_bkt_dec_lock(bkt); + goto __retry; + } else if (!rem) { + /*Only one thread can be here*/ + chunk = timr->meta.refill_chunk(bkt, timr); + if (unlikely(chunk == NULL)) { + timr_bkt_set_rem(bkt, 0); + timr_bkt_dec_lock(bkt); + tim->impl_opaque[0] = tim->impl_opaque[1] = 0; + tim->state = RTE_EVENT_TIMER_ERROR; + return -ENOMEM; + } + bkt->current_chunk = (uint64_t) chunk; + timr_bkt_set_rem(bkt, nb_chunk_slots - 1); + } else { + chunk = (struct tim_mem_entry *)bkt->current_chunk; + chunk += nb_chunk_slots - rem; + } + /* Copy work entry. */ + *chunk = *pent; + timr_bkt_inc_nent(bkt); + timr_bkt_dec_lock(bkt); + + tim->impl_opaque[0] = (uint64_t)chunk; + tim->impl_opaque[1] = (uint64_t)bkt; + tim->state = RTE_EVENT_TIMER_ARMED; + return 0; +} -- 2.16.3