From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-by2nam01on0043.outbound.protection.outlook.com [104.47.34.43]) by dpdk.org (Postfix) with ESMTP id D5CA91B2CB for ; Fri, 16 Feb 2018 22:37:49 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=ObwTAWKgnZ4uPbmLw7aEUbBecItnu+lQDm+p9+KCccI=; b=bFf9ZBUdX87enf7R9mOf2iqD9RyfsPkLno4Sro7USaE38R30eFa4pGKhrNk80C2X2AdJ8Q3LagVtduhkgC8Et5kuhJ1fv4JRh5IrSs46Dz7OVH0BGEFkPCdUmKfwLVFK7pO/SukC1T/dYZHXyrxvJwC9GfddiaDkZeuxjtUGfAQ= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Pavan.Bhagavatula@cavium.com; Received: from localhost.localdomain (111.93.218.67) by DM5PR07MB3467.namprd07.prod.outlook.com (2603:10b6:4:67::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.506.18; Fri, 16 Feb 2018 21:37:46 +0000 From: Pavan Nikhilesh To: jerin.jacob@caviumnetworks.com, santosh.shukla@caviumnetworks.com, erik.g.carrillo@intel.com Cc: dev@dpdk.org, Pavan Nikhilesh Date: Sat, 17 Feb 2018 03:06:56 +0530 Message-Id: <20180216213700.3415-7-pbhagavatula@caviumnetworks.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180216213700.3415-1-pbhagavatula@caviumnetworks.com> References: <20180216213700.3415-1-pbhagavatula@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: SG2PR01CA0090.apcprd01.prod.exchangelabs.com (2603:1096:3:15::16) To DM5PR07MB3467.namprd07.prod.outlook.com (2603:10b6:4:67::22) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 519cdb1f-d401-4f7e-a915-08d57585862b X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(2017052603307)(7153060)(7193020); SRVR:DM5PR07MB3467; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3467; 3:e1VlyBImKz3NORhJcJsCNFOsrBAPeFtla7QBzYFddXYJd+mbUlZkRlFcImXb33vqWxcGAZTYfYhH2g2R7EDdB86bk7eWaV6om73PQDM4A1rPNxPs4B/jhCaD1YOUr63tpqgLNnp/64aIbUYxB91xpXLZJOgq18iVb6bqqQVrmLNQFpeZ9Z5vYqTrl+g62vyPgiVUv1oXC1IwENU/a4Kh5o7ncC/yO/x2o0inDm6H0AnBjL4Y5EYTPuvjt8nIzNcn; 25:OMLVQvmhmdtY0TE4fRPV7jJdQ+ycOE2P5kicm/ESKbXu4WVofAzkNcj8Jai6hKQWs7QE1gvoowSJwG8RSNC2bw5FxzUEBdb4RVdugbiudGbsjziPd26fEXDbs8ASzMqF0QrYZ8lFWk9/vuCv2TLuHg6IBJA1KO+XHWaREP8ASgo71F9AdFer57OyB6+7kmJFn04nfI++heQnGLvpHhnlEq9dLgNKt8sR+JJZCvj2slOCZTif4HZqd5bL81JZFIlvHi1iuaENW80sar+Tq9+hroucObAOAGz/UxwFlk9BagnDTXa/nmbZuRN0o7yZBEeH0qKHYc/6JWIWYPwc3eyLGg==; 31:dE/gki+J8G4KyJ43LANrlF/mCuxJ0y+6Gl8adJiNWzqLcR9tTslc/y1VtZtMfd2eejZRM2uVB6ex9gxdVkg6hCgCL86fL0y5AyoPs8sKb81QBLmPIHZmMB8yEgeAx2kEOnLekzC4lTnWF0daq0L3vMSAMHqqcAuwTycf62OLmbn24DykEWxONyAzSTbZYRbA62Bt3RDaktkWxVD5qAEwBRcEOtyL5ubzrLPk1WWkWgY= X-MS-TrafficTypeDiagnostic: DM5PR07MB3467: X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3467; 20:l40xUKVkx125QSV/MZPw8X7ZQvTomCBydbD8V/YRTaguu6Cdxb3MkvysE4a4gvrHEQ6pQaRjIZ2MpXEi7FGVzcTHYgsdOlvFjqrGfVWFv4u03EnhLGX9Sn6CXmCrE1X04BfHO3S8oiIH7VxhuSzuc5CE2OiiiaoaGDA0Ixke3+Qygr6B9DfcmZhiUnAcyBN3KHCLgQ3inKB5rOfPcuW5vi5CzeeZ5uI9C6BdYr5rX1XIX5WAPm6PWsvkjdt5KsZ4jk7vqMLRgdrD8DEXyl/8ddV/CuSvlycZS8eSxCQMnavsIRpnLcTLa6aDwmCPAKoNQQUvAaii1hfo7MUYdSR0wrpX/VUOoWRxWI41GXSHDHXlvbIs5g5LovWspcQGBvR7QZ+M6H/on7ArOdfxgeVL/uOAjJNJxL9v/LhsvzLukH/YPOQ80NfZsKMtyqwiqxRY8GPVKpPv8XZUWeG22MTVS+fGjhItVx2/jvSDWd9C5SKxpfOwmjyxHUu2oi1NiZ5lZiZMUqx0gqSfGbP5rGxCIAl0CNUxA1rFXxesQwinI4zQxFRt91Kvd/5F/KYyNphGcrKCKER6riB8lQSS3hnzfnPDQijWxH/pK72arGZmEsM=; 4:0++uOJvFD7C8qVAYaLxdhiyTFUZLebjJgmDpFMMKJxIow1Kqvo7V8SXCSFvE7M7+F2Mv1W2cS2gFQ2F7NzRPnE/oQuLEaox76pbNUiZJKvvwTA9FdIRY0+ZXlKdjWdYtszovY31kyCLefpr4VPXqimJvPZrx1NoiwOYEVQWeWUpt3PkfaAQJO1h5Q4eNrMLPiV4x4pbooKHVExcJTHWdoH3qCRYjZJQerQOHL4/NZscZknKlLq3yRdyyv9euWE4rCu+QC/VdQ6/YRU8JTv5ypQ== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040501)(2401047)(8121501046)(5005006)(10201501046)(3231101)(944501161)(93006095)(3002001)(6041288)(20161123560045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(20161123558120)(6072148)(201708071742011); SRVR:DM5PR07MB3467; BCL:0; PCL:0; RULEID:; SRVR:DM5PR07MB3467; X-Forefront-PRVS: 0585417D7B X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6069001)(346002)(366004)(396003)(39860400002)(39380400002)(376002)(189003)(199004)(6486002)(25786009)(2950100002)(386003)(6506007)(53936002)(42882006)(5009440100003)(4326008)(107886003)(6666003)(59450400001)(6512007)(16586007)(106356001)(316002)(105586002)(36756003)(478600001)(2906002)(47776003)(6116002)(76176011)(3846002)(1076002)(97736004)(66066001)(8676002)(51416003)(50466002)(48376002)(81156014)(5660300001)(68736007)(8936002)(575784001)(52116002)(16526019)(26005)(305945005)(1857600001)(72206003)(81166006)(50226002)(7736002)(42262002); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR07MB3467; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM5PR07MB3467; 23:KiRlDF1/8ozu4z3G1uh7TVwtEknTZ7RERkOouWOBF?= =?us-ascii?Q?C5YYLLallAsXh4U6HWNqZRGrRDFUY/3X/PfiZjOuq9UVF2Dq49f2dUc3Bkrc?= =?us-ascii?Q?waXLhZ+0Mvh/xlA+h2P96sraQSdxAKWpFZUKoDvwPh7rseVQbK4aUISdCH9J?= =?us-ascii?Q?ub2fq2i3xDMBCm6vztl5RDoiqcHmpIZ+As6nHRsmOmX6DTE8GyLhCbgc8wtl?= =?us-ascii?Q?img4YDA59ws0NX2eoT2wgaD9qWSNdhZCgnJFoH19FDaPs8KL2sEkZKSeoJXa?= =?us-ascii?Q?cxb6XN15Y3f4cVyT8gMeTMwKtGMBznDDB/J1B8YasXWtnzw3Ki+7L5+m48lF?= =?us-ascii?Q?BMkt5h/BpTE3NXfPgCNiy8Gv+emXW8j6kcI3DyJo0fKJFToMttOgi7+zQIMZ?= =?us-ascii?Q?l0f6Pn6vTTKUnGWKBU1dn50viG6VxDUsnyE8yorx0oo3xebPbBa7o5wwUvY7?= =?us-ascii?Q?Xz5WmUcY5st2L3lhg38FAQCnzElmx6WF7vJs+1EC3/KN8g7qulh/6244uOm2?= =?us-ascii?Q?S3AjvufcG6V3hk6Cvmf+IutO0JgiZvW4YFODYmL323GqIS6aNTjA3UfUPHlM?= =?us-ascii?Q?kWK+uk4G0OYHN3AJ7GmYczt7NsjhTbRGtowh24NKbF5DWR0d/CYeSDtxMW9Z?= =?us-ascii?Q?6TdJH12sO/RKJs8Id50SqK8X/ROrsap2PUpID7yv026kzaX6iPZK3YpKwTU5?= =?us-ascii?Q?+tOCPh+wlYWJIDVUae0FUNAMKRsIUGcuSTeRdbQ/2qnRyehAcQPo7GV7RWD8?= =?us-ascii?Q?ohReWOdrICgyf3E2Z2y7xlp71B+NimZpY5iFDr7waRS7Ds6O2FnaqboU/3Ax?= =?us-ascii?Q?eZyg2GD9Mgg0KypLxrx4atnVmi28PErBzYY2QRrUUWfihw3HENQzu7ZQ57ZE?= =?us-ascii?Q?h5fnEuYMJEgUYtfxxhRqeQtxg0hgAdLrqWgs68NlAeBSsUvm4+fXxVY3DTkD?= =?us-ascii?Q?x3MHnEO7TMulBrXqlOxkdBkSY/u+Wj90zN7iS5nXXkKLXHjMfM933PxMrXJb?= =?us-ascii?Q?58UETqMLFPaEzwV+xUE0IzVAafNGSWpdlsd6PdWBkQpEa8wcdR0SUveYfG0J?= =?us-ascii?Q?7UBqfnGePCLHuEWLiPDHmO9dIe68pyM7QaR9G1VOuPqtscS89Ol5g4pMt+0W?= =?us-ascii?Q?du3IBGBwvtdWfdKn+O3CBRaZpdGmYmjVh+ugj/DNzSsnDzY6/n0nJWy1xVFH?= =?us-ascii?Q?3wuQPPXNGBDti4owTmAa265xWRnZdGSqgOeer2gB1p6LOp7mQVDXE5J9t2JJ?= =?us-ascii?Q?g3jorRL7xBQiLmWxLPwKRKaEAEb5J35bup42A5k?= X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3467; 6:dlE19imP7CszLMAD+p0Dbylt2oj8hLZ1UDHqP52YGHzX9Bo2oGS+WGvqqxmT388ggwwLZ2a2/BGO9YMbDKLMBQX1cYKnbVSPsQozZuy/wyjeuDeD8QU6cR0BoRe8q2IFvDZUPZxwfN1uiTHc9bINbgjphip/HzJ/5og2KAz+nCeIu6D5t7DT5Lu83EsecQUDgEg7hTrTvk03K4zb7rqnAdNKIEM0JmPCQzcSKpCxfxdY5wG9YUqJ0GZM7IMVtiFJEfAT2RfCJXayaVL8Ah5AWM0GgAaYYbzPsG889OM0F/FErCIIrjfvCBJaDuvU88Dh8N8qRlVgD2frC24BwFtkeLbFtBSX8981EKMvIaNyB3o=; 5:EOSBhUcGf83E7Im+D3Qruk/0+gaauwIS1FMJPjaLZC83apfyodFc4je8vOR77fomSq+1Ap49LhrWRaw0GQFL7ZurQzZFy1mjtCtvW5/tZmcV7eQSCiklJcCFI2nrrFUn8aAuYe7lUXK1ZQMB7xJBDU1krlrlmrlR4CZhuAoSG88=; 24:mkqOsSLARGjnmpslE0sfQcR/XvtikV7odoFpvmNQznQEQz/I/IQ8qPHTUySu2rKf1Dyu5NZCa4Nb5Askmo3C4Wc+ZanFomQgJ745glU0SO0=; 7:Rs8F17rn+B5XLBDOYvweP6shWrtCR/pbcpdnjSfbTBwuIvyVzcoEGfO2yIC93bx4so1cj5npIjNN/csUranhZpVQc0MUqkqECg6QiBhJJlcuCy4mWlnDPcBep2r4EyBSEmanmnkQjv6JHibgUAgSRWO2BLOCnzxbGh6lEZL10kpGWEJB2ASYlN9AQ1yZCtZQ5aa3eU8ZbRHS6cSj/BKgyCMAcp2J/Lo/AIgacxOdgjr4wNtZqvXJDAm9vPfOgK1Z SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2018 21:37:46.6464 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 519cdb1f-d401-4f7e-a915-08d57585862b X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR07MB3467 Subject: [dpdk-dev] [PATCH 06/10] event/octeontx: add single producer timer arm variant X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Feb 2018 21:37:50 -0000 When application creates the timer adapter by passing `RTE_EVENT_TIMER_ADAPTER_F_SP_PUT` flag, we can optimize the arm sequence by removing the locking overhead. Signed-off-by: Pavan Nikhilesh --- drivers/event/octeontx/timvf_evdev.c | 22 +++- drivers/event/octeontx/timvf_evdev.h | 5 + drivers/event/octeontx/timvf_worker.c | 65 ++++++++++++ drivers/event/octeontx/timvf_worker.h | 183 ++++++++++++++++++++++++++++++++++ 4 files changed, 270 insertions(+), 5 deletions(-) diff --git a/drivers/event/octeontx/timvf_evdev.c b/drivers/event/octeontx/timvf_evdev.c index d0ba42263..6cf5d4846 100644 --- a/drivers/event/octeontx/timvf_evdev.c +++ b/drivers/event/octeontx/timvf_evdev.c @@ -174,6 +174,7 @@ timvf_ring_create(struct rte_event_timer_adapter *adptr) struct rte_event_timer_adapter_conf *rcfg = &adptr->data->conf; struct timvf_ring *timr; struct octeontx_timvf_info tinfo; + unsigned int mp_flags = 0; if (octeontx_timvf_info(&tinfo) < 0) return -ENODEV; @@ -224,6 +225,11 @@ timvf_ring_create(struct rte_event_timer_adapter *adptr) timr->nb_chunks = nb_timers / nb_chunk_slots; + if (rcfg->flags & RTE_EVENT_TIMER_ADAPTER_F_SP_PUT) { + mp_flags = MEMPOOL_F_SP_PUT | MEMPOOL_F_SC_GET; + timvf_log_info("Using single producer mode"); + } + timr->meta.bkt = rte_zmalloc("octeontx_timvf_bucket", (timr->meta.nb_bkts) * sizeof(struct tim_mem_bucket), 0); @@ -261,8 +267,12 @@ timvf_ring_create(struct rte_event_timer_adapter *adptr) timvf_write64(0x7, (uint8_t *)timr->vbar0 + TIM_VF_NRSPERR_ENA_W1C); timvf_write64(0x7, (uint8_t *)timr->vbar0 + TIM_VF_NRSPERR_ENA_W1S); - adptr->arm_burst = timvf_timer_reg_burst_mp; - adptr->arm_tmo_tick_burst = NULL; + if (mp_flags) + adptr->arm_burst = timvf_timer_reg_burst_sp; + else + adptr->arm_burst = timvf_timer_reg_burst_mp; + + adptr->arm_tmo_tick_burst = timvf_timer_reg_brst; adptr->cancel_burst = timvf_timer_unreg_burst; return 0; @@ -297,11 +307,13 @@ timvf_timer_adapter_caps_get(const struct rte_eventdev *dev, uint64_t flags, uint32_t *caps, const struct rte_event_timer_adapter_ops **ops) { RTE_SET_USED(dev); - RTE_SET_USED(flags); - timvf_ops.arm_burst = timvf_timer_reg_burst_mp; - timvf_ops.arm_tmo_tick_burst = NULL; + if (flags & RTE_EVENT_TIMER_ADAPTER_F_SP_PUT) + timvf_ops.arm_burst = timvf_timer_reg_burst_sp; + else + timvf_ops.arm_burst = timvf_timer_reg_burst_mp; + timvf_ops.arm_tmo_tick_burst = timvf_timer_reg_brst; timvf_ops.cancel_burst = timvf_timer_unreg_burst; *caps = RTE_EVENT_TIMER_ADAPTER_CAP_INTERNAL_PORT; *ops = &timvf_ops; diff --git a/drivers/event/octeontx/timvf_evdev.h b/drivers/event/octeontx/timvf_evdev.h index c80e147e8..b5db233bb 100644 --- a/drivers/event/octeontx/timvf_evdev.h +++ b/drivers/event/octeontx/timvf_evdev.h @@ -186,8 +186,13 @@ bkt_mod(uint32_t rel_bkt, uint32_t nb_bkts) int timvf_timer_adapter_caps_get(const struct rte_eventdev *dev, uint64_t flags, uint32_t *caps, const struct rte_event_timer_adapter_ops **ops); +int timvf_timer_reg_brst(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint64_t timeout_tick, + const uint16_t nb_timers); int timvf_timer_unreg_burst(const struct rte_event_timer_adapter *adptr, struct rte_event_timer **tim, const uint16_t nb_timers); +int timvf_timer_reg_burst_sp(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers); int timvf_timer_reg_burst_mp(const struct rte_event_timer_adapter *adptr, struct rte_event_timer **tim, const uint16_t nb_timers); diff --git a/drivers/event/octeontx/timvf_worker.c b/drivers/event/octeontx/timvf_worker.c index 7a924fd11..3e48f3ca6 100644 --- a/drivers/event/octeontx/timvf_worker.c +++ b/drivers/event/octeontx/timvf_worker.c @@ -5,6 +5,42 @@ #include "timvf_worker.h" +int +timvf_timer_reg_brst(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint64_t timeout_tick, + const uint16_t nb_timers) +{ + int ret; + uint16_t set_timers = 0; + uint16_t idx; + uint16_t arr_idx = 0; + struct timvf_ring *timr = adptr->data->adapter_priv; + struct tim_mem_entry entry[TIMVF_MAX_BURST] __rte_cache_aligned; + + if (unlikely(timeout_tick > timr->meta.nb_bkts)) { + for (idx = 0; idx < nb_timers; idx++) + tim[idx]->state = RTE_EVENT_TIMER_ERROR_TOOLATE; + rte_errno = -EINVAL; + return 0; + } + + while (arr_idx < nb_timers) { + for (idx = 0; idx < TIMVF_MAX_BURST && (arr_idx < nb_timers); + idx++, arr_idx++) { + entry[idx].w0 = + (tim[arr_idx]->ev.event & 0xFFC000000000) >> 6 | + (tim[arr_idx]->ev.event & 0xFFFFFFFF); + entry[idx].wqe = tim[arr_idx]->ev.u64; + } + ret = timvf_add_entry_brst(timr, timeout_tick, &tim[set_timers], + entry, idx); + set_timers += ret; + if (ret != idx) + break; + } + return set_timers; +} + int timvf_timer_unreg_burst(const struct rte_event_timer_adapter *adptr, struct rte_event_timer **tim, const uint16_t nb_timers) @@ -23,6 +59,35 @@ timvf_timer_unreg_burst(const struct rte_event_timer_adapter *adptr, return index; } +int +timvf_timer_reg_burst_sp(const struct rte_event_timer_adapter *adptr, + struct rte_event_timer **tim, const uint16_t nb_timers) +{ + int ret; + uint16_t index; + struct tim_mem_entry entry; + struct timvf_ring *timr = adptr->data->adapter_priv; + for (index = 0; index < nb_timers; index++) { + if (unlikely(tim[index]->timeout_ticks > timr->meta.nb_bkts)) { + tim[index]->state = RTE_EVENT_TIMER_ERROR_TOOLATE; + rte_errno = -EINVAL; + break; + } + + entry.w0 = (tim[index]->ev.event & 0xFFC000000000) >> 6 | + (tim[index]->ev.event & 0xFFFFFFFF); + entry.wqe = tim[index]->ev.u64; + ret = timvf_add_entry_sp(timr, tim[index]->timeout_ticks, + tim[index], &entry); + if (unlikely(ret)) { + rte_errno = -ret; + break; + } + } + + return index; +} + int timvf_timer_reg_burst_mp(const struct rte_event_timer_adapter *adptr, struct rte_event_timer **tim, const uint16_t nb_timers) diff --git a/drivers/event/octeontx/timvf_worker.h b/drivers/event/octeontx/timvf_worker.h index b63dd763c..320eb6ac1 100644 --- a/drivers/event/octeontx/timvf_worker.h +++ b/drivers/event/octeontx/timvf_worker.h @@ -160,6 +160,118 @@ timr_clr_bkt(struct timvf_ring *timr, struct tim_mem_bucket *bkt) return (struct tim_mem_entry *)bkt->first_chunk; } +/* Burst mode functions */ +static inline int __hot +timvf_add_entry_brst(struct timvf_ring *timr, const uint16_t rel_bkt, + struct rte_event_timer **tim, const struct tim_mem_entry *ents, + const uint16_t nb_timers) +{ + int16_t rem; + int16_t crem = 0; + uint8_t lock_cnt; + uint16_t index = 0; + uint16_t chunk_remainder = 0; + uint32_t bucket; + uint32_t tbkt_id; + const uint32_t nb_bkts = timr->meta.nb_bkts; + const uint64_t start = timr->meta.ring_start_cyc; + uint64_t pos_reg; + uint64_t lock_sema; + struct tim_mem_bucket *bkt; + struct tim_mem_entry *chunk; + +__retry: + pos_reg = (rte_rdtsc() - start); + bucket = rte_reciprocal_divide_u64(pos_reg, + &timr->meta.fast_div) + rel_bkt; + tbkt_id = timr->meta.get_target_bkt(bucket, nb_bkts); + bkt = &timr->meta.bkt[tbkt_id]; + + /* Only one thread beyond this. */ + lock_sema = timr_bkt_inc_lock(bkt); + lock_cnt = (uint8_t) + ((lock_sema >> TIM_BUCKET_W1_S_LOCK) & TIM_BUCKET_W1_M_LOCK); + + if (lock_cnt) { + timr_bkt_dec_lock(bkt); + goto __retry; + } + + /* Bucket related checks. */ + if (unlikely(timr_bkt_get_shbt(lock_sema))) { + timr_bkt_dec_lock(bkt); + goto __retry; + } + + chunk_remainder = timr_bkt_fetch_rem(lock_sema); + rem = chunk_remainder - nb_timers; + if (rem < 0) { + crem = nb_chunk_slots - chunk_remainder; + if (chunk_remainder && crem) { + chunk = ((struct tim_mem_entry *)bkt->current_chunk) + + crem; + for (; index < chunk_remainder; index++) { + *chunk = *(ents + index); + tim[index]->impl_opaque[0] = (uint64_t)chunk++; + tim[index]->impl_opaque[1] = (uint64_t)bkt; + tim[index]->state = RTE_EVENT_TIMER_ARMED; + } + timr_bkt_sub_rem(bkt, chunk_remainder); + timr_bkt_add_nent(bkt, chunk_remainder); + } + rem = nb_timers - chunk_remainder; + ents = ents + chunk_remainder; + if (bkt->nb_entry || !bkt->first_chunk) { + if (unlikely(rte_mempool_get(timr->meta.chunk_pool, + (void **)&chunk))) { + /* + * No more chunks, return number of entries + * successfully copied. + */ + timr_bkt_dec_lock(bkt); + rte_errno = -ENOMEM; + tim[index]->state = RTE_EVENT_TIMER_ERROR; + return crem; + } + if (bkt->nb_entry) { + *(uint64_t *)( + (struct tim_mem_entry *)bkt->current_chunk + + nb_chunk_slots) = (uint64_t) chunk; + } else { + bkt->first_chunk = (uint64_t) chunk; + } + } else { + chunk = timr_clr_bkt(timr, bkt); + bkt->first_chunk = (uint64_t) chunk; + } + *(uint64_t *)(chunk + nb_chunk_slots) = 0; + bkt->current_chunk = (uint64_t) chunk; + + for (; index < nb_timers; index++) { + *chunk = *(ents + index); + tim[index]->impl_opaque[0] = (uint64_t)chunk++; + tim[index]->impl_opaque[1] = (uint64_t)bkt; + tim[index]->state = RTE_EVENT_TIMER_ARMED; + } + timr_bkt_set_rem(bkt, nb_chunk_slots - rem); + timr_bkt_add_nent(bkt, rem); + } else { + chunk = (struct tim_mem_entry *)bkt->current_chunk; + chunk += (nb_chunk_slots - chunk_remainder); + for (; index < nb_timers; index++) { + *chunk = *(ents + index); + tim[index]->impl_opaque[0] = (uint64_t)chunk++; + tim[index]->impl_opaque[1] = (uint64_t)bkt; + tim[index]->state = RTE_EVENT_TIMER_ARMED; + } + timr_bkt_sub_rem(bkt, nb_timers); + timr_bkt_add_nent(bkt, nb_timers); + } + + timr_bkt_dec_lock(bkt); + return nb_timers; +} + static inline int __hot timvf_rem_entry(struct rte_event_timer *tim) { @@ -192,6 +304,77 @@ timvf_rem_entry(struct rte_event_timer *tim) return 0; } +/* Single producer functions. */ +static inline int __hot +timvf_add_entry_sp(struct timvf_ring *timr, const uint32_t rel_bkt, + struct rte_event_timer *tim, const struct tim_mem_entry *pent) +{ + int16_t rem; + uint32_t bucket; + uint32_t tbkt_id; + const uint32_t nb_bkts = timr->meta.nb_bkts; + uint64_t lock_sema; + uint64_t pos_reg; + const uint64_t start = timr->meta.ring_start_cyc; + struct tim_mem_bucket *bkt; + struct tim_mem_entry *chunk; + + pos_reg = (rte_rdtsc() - start); + bucket = rte_reciprocal_divide_u64(pos_reg, + &timr->meta.fast_div) + rel_bkt; + tbkt_id = timr->meta.get_target_bkt(bucket, nb_bkts); + bkt = &timr->meta.bkt[tbkt_id]; +__retry: + /*Get Bucket sema*/ + lock_sema = timr_bkt_fetch_sema(bkt); + /* Bucket related checks. */ + if (unlikely(timr_bkt_get_shbt(lock_sema))) + goto __retry; + + /* Insert the work. */ + rem = timr_bkt_fetch_rem(lock_sema); + + if (!rem) { + /* SP mode will have only one thread. */ + if (bkt->nb_entry || !bkt->first_chunk) { + if (unlikely(rte_mempool_get(timr->meta.chunk_pool, + (void **)&chunk))) { + timr_bkt_set_rem(bkt, 0); + tim->impl_opaque[0] = + tim->impl_opaque[1] = 0; + tim->state = RTE_EVENT_TIMER_ERROR; + return -ENOMEM; + } + if (bkt->nb_entry) { + *(uint64_t *)((struct tim_mem_entry *) + bkt->current_chunk + + nb_chunk_slots) = + (uint64_t) chunk; + } else { + bkt->first_chunk = (uint64_t) chunk; + } + *(uint64_t *)(chunk + nb_chunk_slots) = 0; + } else { + chunk = timr_clr_bkt(timr, bkt); + *(uint64_t *)(chunk + nb_chunk_slots) = 0; + bkt->first_chunk = (uint64_t) chunk; + } + bkt->current_chunk = (uint64_t) chunk; + timr_bkt_set_rem(bkt, nb_chunk_slots - 1); + } else { + chunk = (struct tim_mem_entry *)bkt->current_chunk; + chunk += nb_chunk_slots - rem; + } + /* Copy work entry. */ + *chunk = *pent; + timr_bkt_inc_nent(bkt); + + tim->impl_opaque[0] = (uint64_t)chunk; + tim->impl_opaque[1] = (uint64_t)bkt; + tim->state = RTE_EVENT_TIMER_ARMED; + return 0; +} + /* Multi producer functions. */ static inline int __hot timvf_add_entry_mp(struct timvf_ring *timr, const uint32_t rel_bkt, -- 2.16.1