From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 19C2242B6F; Mon, 22 May 2023 11:31:10 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E5B75410DD; Mon, 22 May 2023 11:31:09 +0200 (CEST) Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-he1eur01on2087.outbound.protection.outlook.com [40.107.13.87]) by mails.dpdk.org (Postfix) with ESMTP id 32B4A410D1 for ; Mon, 22 May 2023 11:31:08 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=M5CEpDcYvCN/Jln91CJL1vlmcgmtcBtb8bmUJcJ0ZIi7tN70bYV6hWSxfrrr6EIftfW1J7dSn9u+PACX53sTbQtBpnrQDHcrSNHaYs5WXjYSr/17fwSdMOeqzBwo2pxpAeAuKjsujzv9p3eAaptM1TUG0NKBsG433AK2pvJSEcRH0B2+1V/S4jyMfgkmfwV0LqeM31+Dfh3TZdbGzoaPA4t6Knsp7xBVQ8X4qXG+4Sf/WWDE0eBR092q1IX4+ru9gskJF6fdx6zTkmgB7Ua4f4S9VrdvlOpAXc+i8034KPvF/zf/iCk+ocUENFWPMahgabtLy358M7Jk17Q23oQ27Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AxaiGKBDEBphCO/Vb3h/2XyE9eCMebcWfmUfBLPic5o=; b=QEvb1pf26qx00nKeQf6L6p22frsWzUi4yeBaq6iU0uvu5nh+0oqW7t2dGlLQ8on+rtoRv/XPA48iSXXcOqiQaIl7Y1jdgfhoBLRztB6Bzxu9K/f5wABGHpvYF1mksZ2x0rmr50sdpNwlubuniC0ZpxzG/FR0KGc32Pm5iM402cawfHE3V5VdAFuwV+YOBHn+9VlPrkXXXbD6hNXolnbMz5Ouu6KvdjkiIK2dN2RG6Hdxrqb211iQUH2q441ibO6y9pe25WGOVsnLTzAK9XidXzQEgcziPKIu9F+8bOHT57KuSUhIkMFLCKztwdeFDXj4xEnLv1Xgx8dzdQPX8ri9HA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=dpdk.org smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AxaiGKBDEBphCO/Vb3h/2XyE9eCMebcWfmUfBLPic5o=; b=CvJZ5Pop2g6ZZSMPUOApDU6d9mIX7MnmGzIoAr/h3vWVJ1GsVjn3iwVzg82fDAtl2rVuElLwpU/YGxC25JvId+9kylLV6yOxjDciL9MSQ8tRAOkGodkAOSqziQcqI8UTyMzFdtmh37yFL9z5Y6aRdKKCsiuOnhseqKZLb4kYcmo= Received: from AM8P189CA0008.EURP189.PROD.OUTLOOK.COM (2603:10a6:20b:218::13) by DB9PR07MB9050.eurprd07.prod.outlook.com (2603:10a6:10:3da::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6411.28; Mon, 22 May 2023 09:31:06 +0000 Received: from VI1EUR02FT047.eop-EUR02.prod.protection.outlook.com (2603:10a6:20b:218:cafe::d8) by AM8P189CA0008.outlook.office365.com (2603:10a6:20b:218::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6411.28 via Frontend Transport; Mon, 22 May 2023 09:31:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by VI1EUR02FT047.mail.protection.outlook.com (10.13.60.152) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.6433.14 via Frontend Transport; Mon, 22 May 2023 09:31:05 +0000 Received: from ESESBMB505.ericsson.se (153.88.183.172) by ESESBMB504.ericsson.se (153.88.183.171) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23; Mon, 22 May 2023 11:31:05 +0200 Received: from seliicinfr00049.seli.gic.ericsson.se (153.88.183.153) by smtp.internal.ericsson.com (153.88.183.188) with Microsoft SMTP Server id 15.1.2507.23 via Frontend Transport; Mon, 22 May 2023 11:31:05 +0200 Received: from breslau.. (seliicwb00002.seli.gic.ericsson.se [10.156.25.100]) by seliicinfr00049.seli.gic.ericsson.se (Postfix) with ESMTP id 661B73800CE; Mon, 22 May 2023 11:23:00 +0200 (CEST) From: =?UTF-8?q?Mattias=20R=C3=B6nnblom?= To: CC: Jerin Jacob , , , Van@dpdk.org, Haaren@dpdk.org, Harry , =?UTF-8?q?Mattias=20R=C3=B6nnblom?= Subject: [RFC v3 0/3] Add event dispatcher Date: Mon, 22 May 2023 11:16:25 +0200 Message-ID: <20230522091628.96236-1-mattias.ronnblom@ericsson.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20210409113223.65260-1-mattias.ronnblom@ericsson.com> References: <20210409113223.65260-1-mattias.ronnblom@ericsson.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI1EUR02FT047:EE_|DB9PR07MB9050:EE_ X-MS-Office365-Filtering-Correlation-Id: fdd4ef3d-e81c-40cc-d43c-08db5aa7447c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: HszsfAHHnDxVghotXAyEJmzdZXsf0Fd9ipYBuYVTGUkQstHPNCM5tJjILZANuuAOr8/Zx6VMJT7dkr2oupNmQ6eno4vhTNdPo6HaBdRNFlS6bPNmLNpc9jM7MLz3DkKha0E1SDQDs0d+Nk4zPqpwNH+IpDpaXAu+OuSgMb/K2XAjnnGjn2+OFH6CsWxVGtSApKEdYRx2+zvY+nk/eAzKB4OfDZqiTpwvCHG7xdSX9cVXwHMc23PMrlVXJ5GN2kdwnZs/d5mFmL9i0syM4l6qq2lubx2hoLRTab80ja0hC2eu4qs+FvF0odNbnI7EF3SLD1i9p4jGFG6Yyl7cK4mxlssEVzfkqX0KnEkxBzYl4HLliiwwE44xxqZ3RI/ASDP+DQnHyBIkCvdNONtibacgkbmpturj/4+Gs9WDw5jszgy7Xk5uERs6M8pSjibhm6h8/Ru3dkI96Esx6BU3rP/7m5ZxBhTVOY5pOmWlPgRPQTYVDdKYcS7vICxrYe2eOvMB9luRKP5FGYVwOhK6hG3E6SPWM4SJTFTGRz1yzJi4V3g99nnEZ04A+vJPfV/uAkHk0REXTljzJ9mOoTY/7BYZ62rxUTo6Vl3+Zzwu8ZwP03l/0ij0biX3Dbb+x/j03UHKNmx+adk1z1o4bDuw3z5r+V5qyZxn3ufx6sbyromM5dOMQiPYYvG55f23PW10PrMyIoQ/1QIQzDhOGeh4Ir9hWylz9xr1z5DITD1RiiUk7rCp/V81tnNumcCDWRkGNksZkb0btN9pFcZHnPNwQaxbMVtDrWjkp8lM2LPhbo323Zc= X-Forefront-Antispam-Report: CIP:192.176.1.74; CTRY:SE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:oa.msg.ericsson.com; PTR:office365.se.ericsson.net; CAT:NONE; SFS:(13230028)(4636009)(136003)(346002)(376002)(396003)(39860400002)(451199021)(40470700004)(46966006)(36840700001)(82740400003)(82960400001)(356005)(7636003)(40480700001)(82310400005)(36756003)(86362001)(40460700003)(8676002)(8936002)(107886003)(5660300002)(26005)(1076003)(2906002)(47076005)(66574015)(2616005)(336012)(186003)(6266002)(83380400001)(41300700001)(316002)(6666004)(70586007)(70206006)(54906003)(478600001)(36860700001)(6916009)(4326008); DIR:OUT; SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 May 2023 09:31:05.7345 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: fdd4ef3d-e81c-40cc-d43c-08db5aa7447c X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f; Ip=[192.176.1.74]; Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: VI1EUR02FT047.eop-EUR02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR07MB9050 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The purpose of the event dispatcher is to decouple different parts of an application (e.g., processing pipeline stages), sharing the same underlying event device. The event dispatcher replaces the conditional logic (often, a switch statement) that typically follows an event device dequeue operation, where events are dispatched to different parts of the application based on event meta data, such as the queue id or scheduling type. The concept is similar to a UNIX file descriptor event loop library. Instead of tying callback functions to fds as for example libevent does, the event dispatcher relies on application-supplied matching callback functions to decide where to deliver events. An event dispatcher is configured to dequeue events from a specific event device, and ties into the service core framework, to do its (and the application's) work. The event dispatcher provides a convenient way for an eventdev-based application to use service cores for application-level processing, and thus for sharing those cores with other DPDK services. Although the event dispatcher adds some overhead, experience suggests that the net effect on the application (both syntetic benchmarks and more real-world applications) seems to be positive, except for small dequeue sizes. This is likely due to improved temporal locality, and will very greatly. The overhead seems to be ~10 cc/event on a large core, with a handful of handlers. Outstanding questions: o Explore an option where the callbacks (and callback data) are per lcore. More complicated, but more performant for deployments where all dispatcher lcores are not used for the same purpose. o Reshuffle the handler order in runtime, in such a way that the most-often-matched handler is tried first. o Consider adding possibility to express simple match functions (e..queue_id == 7) without a match callback. o Consider runtime reconfiguration. o Consider moving dispatcher id allocation from app to DPDK. Avoids app-level coordination, but makes the API inconsistent with other Eventdev APIs. o Should the events delivered to the application in the process callback be marked 'const' or not ('const' now, but prohibits reuse for TX). o Consider allowing the application setting the process callback to NULL, signalling to the dispatcher that processing will occur already at the time of the match call. May provide some slight performance benefits for applications where the averagenumber of events supplied per process function call is very small. RFC v3: o Change from int16_t to int for ids. No particular gain in using int16_t over int. o Introduce rte_event_dispatcher_start()/stop() to make API consistent with the other Eventdev services. o Unit tests added. o Programming guide added. o Added dispatcher statistics. o Abandonded the promise to run the match functions in order. This is to allow for future performance optimization (i.e., reorder match functions so that often-matched functions are checked first). o Introduced event_port_id and event_dev_id in the deliver callback, to simplify for applications to follow the Eventdev API requirements to enqueue forward-type events on the same event port as the original events were dequeued. o Mark delivered events as constant. o Introduce optional callback called by the dispatcher when a full batch of events has been distributed to the various handlers. This is useful in cases when an event output buffer is shared among several handlers. o Fixed bug where consumer unregistration would invalidate other registration ids. RFC v2: o Introduced a match callback, allowing the application layer to route events to callbacks not only based on queue id (like in v1), but on arbitrary event meta data or payload. o The fallback was removed, since it now easily can be achieved with a match function always returning true. o The dispatcher now rearranges the batch of dequeued events from the event device, in such a way to minimize the number of deliver calls made. This is done primariy to improved application cache locality. Mattias Rönnblom (3): eventdev: introduce event dispatcher test: add event dispatcher test suite doc: add event dispatcher programming guide app/test/meson.build | 1 + app/test/test_event_dispatcher.c | 814 +++++++++++++++++++++ doc/api/doxy-api-index.md | 1 + doc/guides/prog_guide/event_dispatcher.rst | 423 +++++++++++ doc/guides/prog_guide/index.rst | 1 + lib/eventdev/meson.build | 2 + lib/eventdev/rte_event_dispatcher.c | 670 +++++++++++++++++ lib/eventdev/rte_event_dispatcher.h | 440 +++++++++++ lib/eventdev/version.map | 12 + 9 files changed, 2364 insertions(+) create mode 100644 app/test/test_event_dispatcher.c create mode 100644 doc/guides/prog_guide/event_dispatcher.rst create mode 100644 lib/eventdev/rte_event_dispatcher.c create mode 100644 lib/eventdev/rte_event_dispatcher.h -- 2.34.1