From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D27E6432E9; Thu, 9 Nov 2023 19:42:32 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 97E574064A; Thu, 9 Nov 2023 19:42:32 +0100 (CET) Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-vi1eur04on2071.outbound.protection.outlook.com [40.107.8.71]) by mails.dpdk.org (Postfix) with ESMTP id C28B74026B for ; Thu, 9 Nov 2023 19:42:31 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WJIYzj0OdgQFgIVRaJSn4QBmFnTFTyHEPiZYOzju30Nt6IcGpsHGl/hdoAixh3icskRNGPpah8SvpEkVp6mNFrNAElzGumCniIPxOyqjmsMWSh5wZm8TdAr+nXUSrADC4ErhjMiYTLeFhG1L1W6fPUhns/P6o/z42N4Gx5JGfqD9Sp3TMJKPckTbW9Mec+6zqx/lSKs3Lo8SzLFe24LFb8Wpi1uLPlZEEeDiwIe2upHXASl6rfiqoyDNeDinC9gA2HjDWIA8v+NZyDJivn8o8ya9E79Y2utUMFldOugCE9C80JEt5paFRGA5tZAEE6YKyyb7407E2a3jjOB1LJJGjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=P7PnFIbhAiT0pPlJrlYeibnWim2CJaXOAP4pLs3/uSU=; b=HCampMkVSTLTDemULbtMVCJOGWzDGavmu14qRr/PnPcs8919lLE+Rcqa0m4a8QGKq7iEf2SgSIM+8uXrZ2s1t3fxK765zfn513Kmp8KlFX4WFk+I9nozFz8pAKuBvXjkiiKBnlvaw1XfiQZRuQXKYNKGNOCp/IF/AJXcS5AuRO148oPXdrxPReJAoTO9gnSBzIRBpA24qUEdrsTEjjLRzNpsu309tFekCHkdf6TJuSpSJEJShpyS0ygQ/sQwfkFwetzLgvKLZ4XT0nQGh8VMVFbUkD0gYNnBzTglcAEa5VXkUuwMMjoH4J7SWSmXk7zCJsFJuKmwWpE3UL6dd+ZtxQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=dpdk.org smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P7PnFIbhAiT0pPlJrlYeibnWim2CJaXOAP4pLs3/uSU=; b=XRThTAsjC8SIvG/y0BC7aT8k4f9D61AsKNiQ4Nj+HkJ60Zu0qITfoxtrDrea8mkUF517q1pf38hJ91ES5gcTdlPk9kWJhEM3spaHm1AYozlHpX0mzXhm+wZbJXP0DFx935nPl8EfNJhH0XFSFg1DYgDsOLI8KqKn7ix7SCMWkGC8Zzo9CBGOWCGqdozvpDRT+j4kSe/AAMdTEexhYvtUeaC+TGTqQMAOOLW4zyOUd2Iw14Zq5lCCYkqBpPgyktycNNa+Cx5QMiTLFJcnHlMJaESI7min/+v1fwGSKYu483TFXLFHfLwHO44XhwzHjak53+YTfdkfMZAkpf4sP6LRzw== Received: from DUZPR01CA0130.eurprd01.prod.exchangelabs.com (2603:10a6:10:4bc::25) by AS8PR07MB8133.eurprd07.prod.outlook.com (2603:10a6:20b:370::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6977.19; Thu, 9 Nov 2023 18:42:30 +0000 Received: from DB1PEPF00039232.eurprd03.prod.outlook.com (2603:10a6:10:4bc:cafe::4a) by DUZPR01CA0130.outlook.office365.com (2603:10a6:10:4bc::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6977.19 via Frontend Transport; Thu, 9 Nov 2023 18:42:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by DB1PEPF00039232.mail.protection.outlook.com (10.167.8.105) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6977.16 via Frontend Transport; Thu, 9 Nov 2023 18:42:29 +0000 Received: from ESESBMB502.ericsson.se (153.88.183.34) by SESSMR601.ericsson.se (100.87.178.60) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.2.1118.26; Thu, 9 Nov 2023 19:42:29 +0100 Received: from SESAMR603.ericsson.se (100.87.178.31) by ESESBMB502.ericsson.se (153.88.183.57) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.34; Thu, 9 Nov 2023 19:42:28 +0100 Received: from seliicinfr00049.seli.gic.ericsson.se (153.88.142.248) by smtp.internal.ericsson.com (100.87.178.95) with Microsoft SMTP Server id 15.2.1118.26 via Frontend Transport; Thu, 9 Nov 2023 19:42:28 +0100 Received: from breslau.. (seliicwb00002.seli.gic.ericsson.se [10.156.25.100]) by seliicinfr00049.seli.gic.ericsson.se (Postfix) with ESMTP id C0188380061; Thu, 9 Nov 2023 19:42:28 +0100 (CET) From: =?UTF-8?q?Mattias=20R=C3=B6nnblom?= To: CC: , , Jerin Jacob , Peter Nilsson J , =?UTF-8?q?Svante=20J=C3=A4rvstr=C3=A5t?= , Heng Wang , =?UTF-8?q?Mattias=20R=C3=B6nnblom?= Subject: [RFC] event/dsw: support explicit release only mode Date: Thu, 9 Nov 2023 19:33:23 +0100 Message-ID: <20231109183323.2880-1-mattias.ronnblom@ericsson.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DB1PEPF00039232:EE_|AS8PR07MB8133:EE_ X-MS-Office365-Filtering-Correlation-Id: 1e74dd1c-f87b-4994-6b45-08dbe153a0ac X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rGRJb1TooxUiF5Y/y0MNAuMxWLtepnm436MrMEhovRLi2OX2u0oW1UH3Ab3aDQpiLVEOIaeJTtpCCzDYsDuNwshlvod/r+x+0fIVjiadvAEO1J7F8sxrOfAP7QMYX3L29W4/HnVaNCyPaScoodRXzaWcTglZiecjMzwtfHCltn3yg3io1n+M41bHZ2F/biDYTy0b4MfXnhch0/z0iisBazcf9O0viYDZTT7XFRKhaGCJNqVE/tSKd2K3gB4vq0tdm3Q9eqHvN5zP837u++rfXZyOoYzBeJLbo2ZnVrLd6XdftxmECO2qBP0HL5PpOd00N32z+hKjKHpyhHtm8es0sZMrj08TlQJeAg17tergFepBM2ApHUyOTyMIIOKGR4fqHVcM+/7a3v1Y/Q+n9jrIobdRmkH21xg3dTZeL/yYDrEUMpv+/NkqmC9NPqG3xYXOOZVo3xdkQjss6Qk1zlAuxkjHgEBMVIZjxqTtMe2o5NJZDVV4N8lbSMlKyK/if+mb6XjGAMU2XTbOBvXHCJCQhBm1r9ThVjbOWBRP1+usF3FsmD8TWrWjl0kPO0Ln5nlhjm0oHJvJDH2+LL+XgB1Slxtako0A41cby05tuE2gUbk2Tlqn5cuhUouEFewhCALgODq6cu2tHP5NdMQRt82HXHwAssHbh7GmZiNEkaMyxCqtV5twMnpqge++lYgbFKfFGbfE4sJXJkCyh4piqI+kQfDEcTgcds8vQMbqc8AJpuagO6/868oeqASwswdvm/UkADu0xUpgsnZmVb8kkCvHtc6XPah0rvci7WGXaSPyaT8EaU4MXEfy1s3B1FwfcJUP265jvhOKCtU8mba+xlUmpg== X-Forefront-Antispam-Report: CIP:192.176.1.74; CTRY:SE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:oa.msg.ericsson.com; PTR:office365.se.ericsson.net; CAT:NONE; SFS:(13230031)(4636009)(376002)(396003)(346002)(39860400002)(136003)(230922051799003)(186009)(82310400011)(1800799009)(64100799003)(451199024)(46966006)(36840700001)(40470700004)(2616005)(478600001)(36860700001)(107886003)(6666004)(47076005)(66574015)(336012)(83380400001)(2906002)(1076003)(26005)(5660300002)(41300700001)(6266002)(316002)(70586007)(54906003)(6916009)(70206006)(4326008)(8936002)(8676002)(36756003)(7636003)(86362001)(82740400003)(356005)(82960400001)(40480700001)(40460700003); DIR:OUT; SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2023 18:42:29.6465 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1e74dd1c-f87b-4994-6b45-08dbe153a0ac X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f; Ip=[192.176.1.74]; Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: DB1PEPF00039232.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR07MB8133 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add the RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE capability to the DSW event device. This feature may be used by an EAL thread to pull more work from the work scheduler, without giving up the option to forward events originating from a previous dequeue batch. This in turn allows an EAL thread to be productive while waiting for a hardware accelerator to complete some operation. Prior to this change, DSW didn't make any distinction between RTE_EVENT_OP_FORWARD and RTE_EVENT_OP_NEW type events, other than that new events would be backpressured earlier. After this change, DSW tracks the number of released events (i.e., events of type RTE_EVENT_OP_FORWARD and RTE_EVENT_OP_RELASE) that has been enqueued. To reduce overhead, DSW does not track the *identity* of individual events. This in turn implies that a certain stage in the flow migration process, DSW must wait for all pending releases (on the migration source port, only) to be received from the application, to assure that no event pertaining to any of the to-be-migrated flows are being processed. With this change, DSW starts making a distinction between forward and new type events for credit allocation purposes. Only new events needs credits. All events marked as RTE_EVENT_OP_FORWARD must have a corresponding dequeued event from a previous dequeue batch. Flow migration for flows on RTE_SCHED_TYPE_PARALLEL queues remains unaffected by this change. A side-effect of the tweaked DSW migration logic is that the migration latency is reduced, regardless if implicit relase is enabled or not. Signed-off-by: Mattias Rönnblom --- drivers/event/dsw/dsw_evdev.c | 8 +++- drivers/event/dsw/dsw_evdev.h | 3 ++ drivers/event/dsw/dsw_event.c | 84 ++++++++++++++++++++++------------- 3 files changed, 62 insertions(+), 33 deletions(-) diff --git a/drivers/event/dsw/dsw_evdev.c b/drivers/event/dsw/dsw_evdev.c index 1209e73a9d..445f3ac357 100644 --- a/drivers/event/dsw/dsw_evdev.c +++ b/drivers/event/dsw/dsw_evdev.c @@ -23,15 +23,20 @@ dsw_port_setup(struct rte_eventdev *dev, uint8_t port_id, struct rte_event_ring *in_ring; struct rte_ring *ctl_in_ring; char ring_name[RTE_RING_NAMESIZE]; + bool implicit_release; port = &dsw->ports[port_id]; + implicit_release = + !(conf->event_port_cfg & RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL); + *port = (struct dsw_port) { .id = port_id, .dsw = dsw, .dequeue_depth = conf->dequeue_depth, .enqueue_depth = conf->enqueue_depth, - .new_event_threshold = conf->new_event_threshold + .new_event_threshold = conf->new_event_threshold, + .implicit_release = implicit_release }; snprintf(ring_name, sizeof(ring_name), "dsw%d_p%u", dev->data->dev_id, @@ -221,6 +226,7 @@ dsw_info_get(struct rte_eventdev *dev __rte_unused, .max_profiles_per_port = 1, .event_dev_cap = RTE_EVENT_DEV_CAP_BURST_MODE| RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED| + RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE| RTE_EVENT_DEV_CAP_NONSEQ_MODE| RTE_EVENT_DEV_CAP_MULTIPLE_QUEUE_PORT| RTE_EVENT_DEV_CAP_CARRY_FLOW_ID diff --git a/drivers/event/dsw/dsw_evdev.h b/drivers/event/dsw/dsw_evdev.h index 6416a8a898..a245a8940e 100644 --- a/drivers/event/dsw/dsw_evdev.h +++ b/drivers/event/dsw/dsw_evdev.h @@ -128,6 +128,7 @@ struct dsw_queue_flow { enum dsw_migration_state { DSW_MIGRATION_STATE_IDLE, DSW_MIGRATION_STATE_PAUSING, + DSW_MIGRATION_STATE_FINISH_PENDING, DSW_MIGRATION_STATE_UNPAUSING }; @@ -148,6 +149,8 @@ struct dsw_port { int32_t new_event_threshold; + bool implicit_release; + uint16_t pending_releases; uint16_t next_parallel_flow_id; diff --git a/drivers/event/dsw/dsw_event.c b/drivers/event/dsw/dsw_event.c index 93bbeead2e..c70e50dd16 100644 --- a/drivers/event/dsw/dsw_event.c +++ b/drivers/event/dsw/dsw_event.c @@ -1141,6 +1141,15 @@ dsw_port_move_emigrating_flows(struct dsw_evdev *dsw, source_port->migration_state = DSW_MIGRATION_STATE_UNPAUSING; } +static void +dsw_port_try_finish_pending(struct dsw_evdev *dsw, struct dsw_port *source_port) +{ + if (unlikely(source_port->migration_state == + DSW_MIGRATION_STATE_FINISH_PENDING && + source_port->pending_releases == 0)) + dsw_port_move_emigrating_flows(dsw, source_port); +} + static void dsw_port_handle_confirm(struct dsw_evdev *dsw, struct dsw_port *port) { @@ -1149,14 +1158,15 @@ dsw_port_handle_confirm(struct dsw_evdev *dsw, struct dsw_port *port) if (port->cfm_cnt == (dsw->num_ports-1)) { switch (port->migration_state) { case DSW_MIGRATION_STATE_PAUSING: - dsw_port_move_emigrating_flows(dsw, port); + port->migration_state = + DSW_MIGRATION_STATE_FINISH_PENDING; break; case DSW_MIGRATION_STATE_UNPAUSING: dsw_port_end_emigration(dsw, port, RTE_SCHED_TYPE_ATOMIC); break; default: - RTE_ASSERT(0); + RTE_VERIFY(0); break; } } @@ -1195,19 +1205,18 @@ dsw_port_note_op(struct dsw_port *port, uint16_t num_events) static void dsw_port_bg_process(struct dsw_evdev *dsw, struct dsw_port *port) { - /* For simplicity (in the migration logic), avoid all - * background processing in case event processing is in - * progress. - */ - if (port->pending_releases > 0) - return; - /* Polling the control ring is relatively inexpensive, and * polling it often helps bringing down migration latency, so * do this for every iteration. */ dsw_port_ctl_process(dsw, port); + /* Always check if a migration is waiting for pending releases + * to arrive, to keep the time at which dequeuing new events + * from the port is disabled. + */ + dsw_port_try_finish_pending(dsw, port); + /* To avoid considering migration and flushing output buffers * on every dequeue/enqueue call, the scheduler only performs * such 'background' tasks every nth @@ -1252,8 +1261,8 @@ static __rte_always_inline uint16_t dsw_event_enqueue_burst_generic(struct dsw_port *source_port, const struct rte_event events[], uint16_t events_len, bool op_types_known, - uint16_t num_new, uint16_t num_release, - uint16_t num_non_release) + uint16_t num_new, uint16_t num_forward, + uint16_t num_release) { struct dsw_evdev *dsw = source_port->dsw; bool enough_credits; @@ -1287,14 +1296,14 @@ dsw_event_enqueue_burst_generic(struct dsw_port *source_port, if (!op_types_known) for (i = 0; i < events_len; i++) { switch (events[i].op) { - case RTE_EVENT_OP_RELEASE: - num_release++; - break; case RTE_EVENT_OP_NEW: num_new++; - /* Falls through. */ - default: - num_non_release++; + break; + case RTE_EVENT_OP_FORWARD: + num_forward++; + break; + case RTE_EVENT_OP_RELEASE: + num_release++; break; } } @@ -1309,15 +1318,15 @@ dsw_event_enqueue_burst_generic(struct dsw_port *source_port, source_port->new_event_threshold)) return 0; - enough_credits = dsw_port_acquire_credits(dsw, source_port, - num_non_release); + enough_credits = dsw_port_acquire_credits(dsw, source_port, num_new); if (unlikely(!enough_credits)) return 0; - source_port->pending_releases -= num_release; + dsw_port_return_credits(dsw, source_port, num_release); + + source_port->pending_releases -= (num_forward + num_release); - dsw_port_enqueue_stats(source_port, num_new, - num_non_release-num_new, num_release); + dsw_port_enqueue_stats(source_port, num_new, num_forward, num_release); for (i = 0; i < events_len; i++) { const struct rte_event *event = &events[i]; @@ -1329,9 +1338,9 @@ dsw_event_enqueue_burst_generic(struct dsw_port *source_port, } DSW_LOG_DP_PORT(DEBUG, source_port->id, "%d non-release events " - "accepted.\n", num_non_release); + "accepted.\n", num_new + num_forward); - return (num_non_release + num_release); + return (num_new + num_forward + num_release); } uint16_t @@ -1358,7 +1367,7 @@ dsw_event_enqueue_new_burst(void *port, const struct rte_event events[], return dsw_event_enqueue_burst_generic(source_port, events, events_len, true, events_len, - 0, events_len); + 0, 0); } uint16_t @@ -1371,8 +1380,8 @@ dsw_event_enqueue_forward_burst(void *port, const struct rte_event events[], events_len = source_port->enqueue_depth; return dsw_event_enqueue_burst_generic(source_port, events, - events_len, true, 0, 0, - events_len); + events_len, true, 0, + events_len, 0); } uint16_t @@ -1484,21 +1493,34 @@ dsw_event_dequeue_burst(void *port, struct rte_event *events, uint16_t num, struct dsw_evdev *dsw = source_port->dsw; uint16_t dequeued; - source_port->pending_releases = 0; + if (source_port->implicit_release) { + dsw_port_return_credits(dsw, port, + source_port->pending_releases); + + source_port->pending_releases = 0; + } dsw_port_bg_process(dsw, source_port); if (unlikely(num > source_port->dequeue_depth)) num = source_port->dequeue_depth; - dequeued = dsw_port_dequeue_burst(source_port, events, num); + if (unlikely(source_port->migration_state == + DSW_MIGRATION_STATE_FINISH_PENDING)) + /* Do not take on new work - only finish outstanding + * (unreleased) events, to allow the migration + * procedure to complete. + */ + dequeued = 0; + else + dequeued = dsw_port_dequeue_burst(source_port, events, num); if (unlikely(source_port->migration_state == DSW_MIGRATION_STATE_PAUSING)) dsw_port_stash_migrating_events(source_port, events, &dequeued); - source_port->pending_releases = dequeued; + source_port->pending_releases += dequeued; dsw_port_load_record(source_port, dequeued); @@ -1508,8 +1530,6 @@ dsw_event_dequeue_burst(void *port, struct rte_event *events, uint16_t num, DSW_LOG_DP_PORT(DEBUG, source_port->id, "Dequeued %d events.\n", dequeued); - dsw_port_return_credits(dsw, source_port, dequeued); - /* One potential optimization one might think of is to * add a migration state (prior to 'pausing'), and * only record seen events when the port is in this -- 2.34.1