From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-BN3-obe.outbound.protection.outlook.com (mail-bn3nam01on0049.outbound.protection.outlook.com [104.47.33.49]) by dpdk.org (Postfix) with ESMTP id 5BD0C995C for ; Thu, 17 Aug 2017 18:12:22 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=uZCf9ieA8nXHRhjFA0zPg8ySAHMSBmTr9uvbMOnBryA=; b=D7q8OwsxMhV7t20WTwafcI3KkLTT5DXQYgiAL8SZ9PLOg+oJhFZq+e9fcQlhn3uziSambxpjGQDoru96HsVvTnJvineWtej088ykOmHPCWR/mv/wNqgh3DagnPKePJntcS+RsPlCPYLIUxwVzw0O+rw2VHlTcvcRZmD5V5cT0OI= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.JacobKollanukkaran@cavium.com; Received: from jerin.domain.name (106.200.242.126) by CO2PR07MB2517.namprd07.prod.outlook.com (10.166.200.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1341.21; Thu, 17 Aug 2017 16:12:15 +0000 From: Jerin Jacob To: dev@dpdk.org Cc: thomas@monjalon.net, bruce.richardson@intel.com, harry.van.haaren@intel.com, hemant.agrawal@nxp.com, gage.eads@intel.com, nipun.gupta@nxp.com, narender.vangati@intel.com, nikhil.rao@intel.com, pbhagavatula@caviumnetworks.com, jianbo.liu@linaro.org, rsanford@akamai.com, Jerin Jacob Date: Thu, 17 Aug 2017 21:41:03 +0530 Message-Id: <20170817161104.24293-1-jerin.jacob@caviumnetworks.com> X-Mailer: git-send-email 2.14.1 MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [106.200.242.126] X-ClientProxiedBy: BMXPR01CA0035.INDPRD01.PROD.OUTLOOK.COM (10.174.214.21) To CO2PR07MB2517.namprd07.prod.outlook.com (10.166.200.151) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1d29d762-cf9c-4f19-6d9c-08d4e58abceb X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:CO2PR07MB2517; X-Microsoft-Exchange-Diagnostics: 1; CO2PR07MB2517; 3:nslN5nrYzdcIQ1Z+rD0ug8cr1NjbL+jGhuJQzhac4JxjBH8BtGxtwU+6t7kl0ekPRGUKyBRTScurfHXuN0nh8YOFb9UBF/KkO68+xkelLhnuL4pFubRwjVvBkDztks7kqZOP7sKfQR3qcQYaAMrCTHJ6JJZyF6Btloi+U6cpfvG70E4qSTJ1z7xUmHzD9fWgHMq+8maHZqio6HatQI0uNBWHRl+dbnYfzpZn3Z2PR5ViRNB0zttSuFaY+uM9HgF7; 25:+/80iFdUJDFCfe3yFlp0JgrU5XZyvBc0sHfl2hjH3WZb40eeOVDWT2xRVFp9peyW70hbdvjDmQdwRITt2ylpLnvLzCOLaznHPp5tMzIpL/sK8c4Mvy/v63/4TzaP+EQSMP9kDaCPbyn7G3VuWe+SQ6w0BwIb8zpHePcLTEkJE8aomCdfEk/qVjiblEv3wAe6yrvsZyuKr4Ra9+egmxeRAO0S0+yzL2vV4iwh6TtUWeyJNGM0hotMbDfuAUvQAU62gx0fyyrl+moqln5BDmRJoRlv6S0UiX4RHrUsY09UAZTVFy0GVAuS2cUS91WCcM1M8PiQW0soGwD/tX/PRR6C3g==; 31:dN+rlec8hLRxLzlHva07NdpdIRuk4Z2o5nuuCYSTaByDL+pCv/pSiF8zonXv4y9Ws8eKZ8lNeDwmJNdUF8nWq2NSD81GYgdQg1pBnI3ywgVKrxKBDKDpmxiRp3ekQjOKaN9o1H61WAQzDEmtO0RAw+Sr+x7soXzOCB3XXR6ye6DOUb0PoeWzx3yGyZKdxl3iOETmjQHYlQGrLPbYn3d7rVFGkStyusnMo0PcVMQut74= X-MS-TrafficTypeDiagnostic: CO2PR07MB2517: X-Microsoft-Exchange-Diagnostics: 1; CO2PR07MB2517; 20:0/c/efEy7VtkQZqOHY2+6pGIc7pjc0Wh9aBcsuXo7wARMt642bg6KEgycR3u3DBlioxSpAsn/kqSLr2hiv0a/8gqrvOW9xUqj0IY+nY6HaZm401gAqMrXKfH5DIgyWqHHFnFUawKi8VH0KnehEhvqPTP7I1qJUrEOVVkk544QzkC98ZRDP9vp1XB9kQrQsJXFMi58mLc6y8c330Eou2w4VxNpDERCVA/PtPBDvDw6s0PmZSoZRtxkndvK/53q197vVp+4vh//Ba0x3TeNGX52yToYm6l2LbfpfR6lnul3cfeqnuCGLYbcrYbaxhNjMHGggu97v56dN2bDfBB55y8F6wkhR85W+Qf6rzlYlBpyd0s6tKWoZJxzheBvLKzCNg0tHbMIRKZ1gGMrbermYEaFX4gZM0mvSJBWF2dpdE5lU/3G/1DVbE1sdOy1tQQnFeIIKN8P5MANTFkIhesmXA7FF+KBFBss5jFh3iT7NHIiaq4CqX0Onxv968K3BcZMrJ7SwyCDg03g0xkRsAytgjxivLH173hANf+9pgsFKfXxfuj9Cl3hKsOSsnvimDsHQ8by/PcJxydsta9c9W22SZUvdYE+Sndp7azVsAmcWYNLyI=; 4:lHqHZylrwCH7n60uWfqFKQHk5512XDf/ua2PIXmGodc38IFme1rGktt1ttdy8Gx6LpNfA/J7qMkWbNlW9ZimG6uxB5gCMTPAN5oiRKHB7Zzdc144yzy51X3dOIocrwIqat6M+YmhmDhyLgqeIxjINS5IcB35ySlvUJ+9KlwA1oH8kh10XXDCE1jQFV+oSbQjDk/U4WpLXnxYqdU042GngAHvXb7ME9FE/wyKa+sn5ehB8yBNfWOne5kwewHnZakzUXOt05WZrHWerP5ZObdfimH4ZBiNuUu5178YvYwqwD8= X-Exchange-Antispam-Report-Test: UriScan:(17755550239193); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(93006095)(100000703101)(100105400095)(6041248)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(20161123562025)(20161123558100)(20161123560025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:CO2PR07MB2517; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:CO2PR07MB2517; X-Forefront-PRVS: 0402872DA1 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(7370300001)(6009001)(189002)(199003)(6506006)(53936002)(2351001)(6512007)(6486002)(6666003)(110136004)(107886003)(106356001)(105586002)(6306002)(42186005)(53416004)(42882006)(6916009)(53376002)(5660300001)(69596002)(7416002)(48376002)(50466002)(101416001)(33646002)(50986999)(36756003)(189998001)(66066001)(478600001)(966005)(3846002)(6116002)(8676002)(81156014)(47776003)(81166006)(1076002)(25786009)(8656003)(7350300001)(2906002)(68736007)(5003940100001)(7736002)(305945005)(4326008)(72206003)(50226002)(2361001)(97736004)(5009440100003)(217873001); DIR:OUT; SFP:1101; SCL:1; SRVR:CO2PR07MB2517; H:jerin.domain.name; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CO2PR07MB2517; 23:NiNPAe5i+GWWV1cDHspC1ELyAHiuvvuJBuQVoGc6G?= =?us-ascii?Q?8iaxRazPrLgjRt4GFZSzMC1DAiz557BefQb2AjHn44o7hkJggGy73Q6dI+M4?= =?us-ascii?Q?ngElTlZT7HWc54LlqZQmhlUz+VlffyPUMe1+C66aYXfP2EyWZXsFHwyR+ozE?= =?us-ascii?Q?MH9sDAzRaEaHsD1812Vsank5S8IWyd3bSgHxmR08FKt7M6FiA688r+LaqsII?= =?us-ascii?Q?tjoM8D9seE9Q+qvZ7OyvBPasaGEh1Zu9PgB7n89i/L6n2dGyUJYrxztggsT7?= =?us-ascii?Q?4ZV+tjts4FSlnBNJiucVt6vQFm2Aeq2wum0aJ+v5VqV/c4zRQYdxfIXbHC+U?= =?us-ascii?Q?e4GaW5sQeE999KBiej3CoyLNWi19VX6Ei6THOwNRbNF+f9I+uvCuWxOkpGVJ?= =?us-ascii?Q?HgWQq17lzY/b0qD0C52+Ey+2o9oCr72v6v51WAVILXu/L8MyAC9XXDxtrRi9?= =?us-ascii?Q?3pqc2/AbIj4Q2wfsujlqj4qHLi8YW3dAtq81Vb3yVaeHJJ4s8pZqaGp4iih1?= =?us-ascii?Q?aFJwawWtUfHRSS5qmxQrXdd0UEm0f6oFyGikmpYjQ8bevmydOjeHei5Xfw/0?= =?us-ascii?Q?JU1zoly3E//uraACcTm5SQfk9oHm7b/f8sxdQZK0tW7Er+Dg3rP2ajLyJiEL?= =?us-ascii?Q?FO0E04ekavrjLpGvbNDKbjD4XCNaFtFZB0n0JIy15z0Yk4AsjeMxfWKbWKlj?= =?us-ascii?Q?Rj4y76AhPjeHzECe+lRheJk9cspUtc2hCB6i4zaqhgt4RcYZHE7WxX3Wi8h6?= =?us-ascii?Q?qs9zZ1RweC/TwhPaniqYLFMK8UZS9bIJLzs5oA0YNg68R+nbo8CNXRp1sGOf?= =?us-ascii?Q?KGaHndaTge8JzLNQL+Hjf/nfkH4+aRs3Tw0su700Jt7r1WZJS7RO6ADLSqjm?= =?us-ascii?Q?QELZP78QyxVIzCWdsXWe0GXqu/AcQNUZOySM1PPEYMCdAzehxrRXFSWTS6WW?= =?us-ascii?Q?QS9ibr4qG0r73JBjwCwlKIPLry+ROVy7gvwIWQ1wDKNb1+i8AM5vKGB9Gk4O?= =?us-ascii?Q?iSsd1sg7una1yBp9yr72SyqzVWufzW75xoucf0YIiUFITufLphteI8sMa81K?= =?us-ascii?Q?JqIOoiym8EODWta/KpJ6iGb52i9HD93sHTau2CPap5zjb1T2YiBgOTvHlAXv?= =?us-ascii?Q?7O7HKzA0N8NpgeRMu1g1VyD+U9sjIF5SmNMJucMcjH4KzuQb8sVmE/kIgzP9?= =?us-ascii?Q?wH93XYRLgPZY1XvcCEmXjZIvXNe90m1ucfMkM1adBsP5wxi7eCBmWnHrsl4Q?= =?us-ascii?Q?mAiOE0aw6EPvv5fejDgvYkYa1RP2P5BaLitzJ2WBiPzz16p1NZpP85WqNcPC?= =?us-ascii?Q?qgfwW6o6ueZswkFLEseiiMgWNHwXPOd29sLSjco+XKw?= X-Microsoft-Exchange-Diagnostics: 1; CO2PR07MB2517; 6:yJ45YP8z05o5+3obmL0aFXQXDE/ygtAGhpekQhwb/v6dNC/FAzEVcCgTda4YF4OXfp+7zZyoWsRpekSw4/qn/NkIkqvIUJSsRajHyLJk6GzV+x3UOgwjpI79FFtY/9BiT+u1vZx6DzrKxtGffODJh2aSESurjZGdPmnOyra5xvRdcF01ASmEitVigWnEYu1ZTQ4XoblmBlFsTJYA7+Kf2HmcYH0i4p938q/UDaUNo05tyeb9w1yitq1QSWXgLocmVWDU6o6Rjwh2Lb/Rx2GewwicfuR40nww9fCHugW+LLyaRs/UVf4RXZnYC6waO819vkjrUp0SJg2Bae0ftaB9GQ==; 5:3sXLswBOIGRlBOSEK4NXu8zlZ2xaVbR42YqSiky1eCVfH7496l13+E9nBkX72D362azUHr4hAROHpgDLykeyUXNjIyItLjCXq/q7oD7Widp+pdxwtbLZNrxWswkVPHDlUcSWrf7xcPaQJJw4LWBEzg==; 24:sknvXVDWL8+kJ34js7uxGqstZ5HJsBOogoBzHdU1HlYcQ1wZIoHbMX6nVZOihuaKTf9dXdB+1MNBzv3Ec1ntdtAWCKCJRRnRAnaEEXfTk4o=; 7:8bWWkd41eacD/Jjhyr9ac5H63X7c0yWITxNb6pSaPVPogNQ7fnnuVNivBrXvJIp/vDIza5suUmccfmgRHXMvoJ06yT7pxZn5NtH4eA2YHKvhXYmlFHp6RG5zHsmfQlGmOadqRJ9CT1vMdKKQT5711Tk65oNYLP6jROjppNfQYlR33MHC4tke8Mq3dueCqm7fOxAiHieC8NX3WDBgd1Gp4P/MOazozNd1eKztuF374t0= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Aug 2017 16:12:15.7446 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO2PR07MB2517 Subject: [dpdk-dev] [RFC PATCH 0/1] eventtimer: introduce event timer wheel X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Aug 2017 16:12:23 -0000 Some of the NPU class of networking hardwares has timer hardware where the user can arm and cancel the event timer. On the expiry of the timeout time, the hardware will post the notification as an event to eventdev HW, Instead of calling a callback like CPU based timer scheme. It enables, highresolution (1us or so) timer management using internal or external clock domains, and offloading the timer housing keeping work from the worker lcores. This RFC attempts to abstract such NPU class of timer Hardware and introduce event timer wheel subsystem inside the eventdev as they are tightly coupled. This RFC introduces the functionality to create an event timer wheel. This allows an application to arm event timers, which shall enqueue an event to a specified event queue on expiry of a given interval. The event timer wheel uses an ops table to which the various event devices (e.g Cavium Octeontx, NXP dpaa2 and SW) register timer subsystem implementation specific ops to use. The RFC extends DPDK event based programming model where event can be of type timer, and expiry event will be notified through CPU over eventdev ports. Some of the use cases of event timer wheel are Beacon Timers, Generic SW Timeout, Wireless MAC Scheduling, 3G Frame Protocols, Packet Scheduling, Protocol Retransmission Timers, Supervision Timers. All these use cases require high resolution and low time drift. The abstract working model of an event timer wheel is as follows: ================================================================= timer_tick_ns + +-------+ | | | | +-------+ bkt 0 +----v---+ | | | | | +-------+ | +---+---+ +---+---+ +---+---+---+---+ | | | | | | | | | | bkt n | | bkt 1 |<-> t0| t1| t2| tn| | | | | | | | | | +---+---+ +---+---+ +---+---+---+---+ | Timer wheel | +---+---+ +---+---+ | | | | | bkt 4 | | bkt 2 |<--- Current bucket | | | | +---+---+ +---+---+ | +-------+ | | | | | +------+ bkt 3 +-------+ | | +-------+ - It has a virtual monotonically increasing 64-bit timer wheel clock based on *enum rte_event_timer_wheel_clk_src* clock source. The clock source could be a CPU clock, or a platform depended external clock. - Application creates a timer wheel instance with given clock source, the total number of event timers, resolution(expressed in ns) to traverse between the buckets. - Each timer wheel may have 0 to n buckets based on the configured max timeout(max_tmo_ns) and resolution(timer_tick_ns). On timer wheel start, the timer starts ticking at *timer_tick_ns* resolution. - Application arms an event timer to be expired at the number of *timer_tick_ns* from now. - Application can cancel the existing armed timer if required. - If not canceled by the application and the timer expires then the library injects the timer expiry event to the designated event queue. - The timer expiry event will be received through *rte_event_dequeue_burst* - Application frees the created timer wheel instance. A more detailed description of the event timer wheel is contained in the header's comments. Implementation thoughts ======================= The event devices have to provide a driver level function that is used to get event timer subsystem capability and the respective event timer wheel ops. if the event device is not capable a software implementation of the event timer wheel ops will be selected. The software implementation of timer wheel will make use of existing rte_timer[1], rte_ring library and EAL service cores[2] to achieve event generation. The worker cores call event timer arm function which enqueues event timer to a rte_ring. The registered service core would then dequeue event timer from rte_ring and use the rte_timer library to register a timer. The service core then invokes rte_timer_manage() function to retrieve expired timers and generates the associated event. The implementation of event timer wheel subsystem for both hardware (Cavium OCTEONTX) and software(if there are no volunteers) will be undertaken by Cavium. [1] http://dpdk.org/doc/guides/prog_guide/timer_lib.html [2] http://dpdk.org/ml/archives/dev/2017-May/065207.html An example code snippet to show the proposed API usage ====================================================== example: TCP Retransmission in abstract form. uint8_t configure_event_dev(...) { /* Create the event device. */ const struct rte_event_dev_config config = { .nb_event_queues = 1, /* Event device related configuration. */ ... }; rte_event_dev_configure(event_dev_id, &config); /* Event queue and port configuration. */ ... /* Start the event device.*/ rte_event_dev_start(event_dev_id); } #define NSECPERSEC 1E9 // No of ns for 1 sec uint8_t configure_event_timer_wheel(...) { /* Create an event timer wheel for reliable connections. */ const struct rte_event_timer_wheel_config wheel_config = { .event_dev_id = event_dev_id, .timer_wheel_id = 0, .clk_src = RTE_EVENT_TIMER_WHEEL_CPU_CLK, .timer_tick_ns = NSECPERSEC / 10, // 100 milliseconds .max_tmo_nsec = 180 * NSECPERSEC // 2 minutes .nb_timers = 40000, // Number of timers that the wheel can hold. .timer_wheel_flags = 0, }; struct rte_event_timer_wheel *wheel = NULL; wheel = rte_event_timer_wheel_create(&wheel_config); if (wheel == NULL) { /* Failed to create event timer wheel. */ ... return false; } /* Start the event timer wheel. */ rte_event_timer_wheel_start(wheel); /* Create a mempool of event timers. */ struct rte_mempool *event_timer_pool = NULL; event_timer_pool = rte_mempool_create("event_timer_mempool", SIZE, sizeof(struct rte_event_timer), ...); if (event_timer_pool == NULL) { /* Failed to create event timer mempool. */ ... return false; } } uint8_t process_tcp_data_packet(...) { /*Classify based on type*/ switch (...) { case ...: /* Setting up a new connection (Protocol dependent.) */ ... /* Setting up a new event timer. */ conn->timer = NULL rte_mempool_get(event_timer_pool, (void **)&conn->timer); if (timer == NULL) { /* Failed to get event timer instance. */ /* Tear down the connection */ return false; } /* Set up the timer event. */ conn->timer->ev.u64 = conn; conn->timer->ev.queue_id = event_queue_id; ... /* All necessary resources successfully allocated */ /* Compute the timer timeout ticks */ conn->timer->timeout_ticks = 30; //3 sec Per RFC1122(TCP returns) /* Arm the timer with our timeout */ ret = rte_event_timer_arm_burst(wheel, &conn->timer, 1); if (ret != 1) { /* Check return value for too early or too late expiration * tick */ ... return false; } return true; case ...: /* Ack for the previous tcp data packet has been received.*/ /* cancel the retransmission timer*/ rte_event_timer_cancel_burst(wheel, &conn->timer, 1); break; } } uint8_t process_timer_event(...) { /* A retransmission timeout for the connection has been received. */ conn = ev.event_ptr; /* Retransmit last packet (e.g. TCP segment). */ ... /* Re-arm timer using original values. */ rte_event_timer_arm_burst(wheel_id, &conn->timer, 1); } void events_processing_loop(...) { while (...) { /* Receive events from the configured event port. */ rte_event_dequeue_burst(event_dev_id, event_port, &ev, 1, 0); ... /* Classify events based on event_type. */ switch(ev.event_type) { case RTE_EVENT_TYPE_ETHDEV: ... process_packets(...); break; case RTE_EVENT_TYPE_TIMER: process_timer_event(ev); ... break; } } } int main() { configure_event_dev(); configure_event_timer_wheel(); on_each_worker_lcores(events_processing_loop()) } Jerin Jacob (1): eventtimer: introduce event timer wheel doc/api/doxy-api-index.md | 3 +- lib/librte_eventdev/Makefile | 1 + lib/librte_eventdev/rte_event_timer_wheel.h | 493 ++++++++++++++++++++++++++++ lib/librte_eventdev/rte_eventdev.h | 4 +- 4 files changed, 498 insertions(+), 3 deletions(-) create mode 100644 lib/librte_eventdev/rte_event_timer_wheel.h -- 2.14.1