From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3D804A052E; Mon, 9 Mar 2020 07:51:59 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0C8511BFFC; Mon, 9 Mar 2020 07:51:51 +0100 (CET) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60048.outbound.protection.outlook.com [40.107.6.48]) by dpdk.org (Postfix) with ESMTP id 59CD81BFF4 for ; Mon, 9 Mar 2020 07:51:46 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=e34LsBVMXX2hTGsWm/+c3my1IL+oKD3eV1o+Dp+KLJ77bYB/SetD4Q7WOVR3hZJbVh3j83v9DWxaH0MFNlfUvGjx7csAC5zEkaoWwT/7SwEwqSgZvOmTWmKOE+42zBwYatvqxUltLPgV2HkBwKFLury/tPC+19dRRhYu4/a9s4X0iCqnbi2uB2K/E+4tNrOnUDCWwr4Cm7ae6OBMeuGeKWftLzXkL38r2jiINls7o38JE6CqzQ161XZwPPLog407XDMpy197llt2dq2wCLtNgmNlXwdo5Xm3u3KUykMnlcsRDsHUoFhqn/yNDszNKSyrpCOp6qfg43QkA2Mn/0atiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tmMDB4HlN9TnEZ2sxA6Oc9RrwKRf5RWn7jk+v49xADo=; b=kzVpwrAVMnC3U1HgyJNjy3P9+CloRpKWmI2dTC88F+YNTfC18rIyrGRM4wvs0K3vNUgeC360UWVYHb14uHfvu0oX+ejna03MmzHz5VscJnCVk4qWlEWqPRlC9RGyyuoSspvz3qLOFF+DCOqqcJhPcxZ+ISTbmENuPku7UkO+jf88jbiA+q1oY299bnO1DF1bCwOtnrljAFz41X6ztBRKo82UmOjuTHshQI3CSAuBbNRcw0DitzKOGK7Tv3zqhona3iiVHXQi2okDtQwSm+MdgaivWXRhNuRbyTJ9qIz/eDWPB6lPsSMAwEHq4Z3X4uf++lHi4Nq0x6YDXYTExoy1wg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=arm.com smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=none pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tmMDB4HlN9TnEZ2sxA6Oc9RrwKRf5RWn7jk+v49xADo=; b=aeHXnDKPB+hVnfUb4Cq8xK+TOq1jlM5HM+msTwfzXE5MVERugrRODXrDQRHl0l1jfxldgKUSkMg3tE3og65Up6FdKdJH2F9Zlva9J4KoLIdU5sMuPVBrSZCxJaWi/DGlTvS0bGfkn1a5ehK6YIMPVQQ59KY/nhRIMYC19Sn7+q4= Received: from AM6PR05CA0033.eurprd05.prod.outlook.com (2603:10a6:20b:2e::46) by DB6PR0701MB2311.eurprd07.prod.outlook.com (2603:10a6:4:5e::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2814.10; Mon, 9 Mar 2020 06:51:45 +0000 Received: from AM5EUR02FT046.eop-EUR02.prod.protection.outlook.com (2603:10a6:20b:2e:cafe::ea) by AM6PR05CA0033.outlook.office365.com (2603:10a6:20b:2e::46) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2793.15 via Frontend Transport; Mon, 9 Mar 2020 06:51:45 +0000 Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; arm.com; dkim=none (message not signed) header.d=none;arm.com; dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; Received: from oa.msg.ericsson.com (192.176.1.74) by AM5EUR02FT046.mail.protection.outlook.com (10.152.8.221) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.2793.11 via Frontend Transport; Mon, 9 Mar 2020 06:51:45 +0000 Received: from ESESBMB503.ericsson.se (153.88.183.170) by ESESBMR506.ericsson.se (153.88.183.202) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1713.5; Mon, 9 Mar 2020 07:51:44 +0100 Received: from ESESSMB503.ericsson.se (153.88.183.164) by ESESBMB503.ericsson.se (153.88.183.170) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1713.5; Mon, 9 Mar 2020 07:51:43 +0100 Received: from selio1a020.lmera.ericsson.se (153.88.183.153) by smtp.internal.ericsson.com (153.88.183.191) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Mon, 9 Mar 2020 07:51:43 +0100 Received: from breslau.lmera.ericsson.se (breslau.lmera.ericsson.se [150.132.109.241]) by selio1a020.lmera.ericsson.se (8.15.1+Sun/8.15.1) with ESMTP id 0296pgIP024126; Mon, 9 Mar 2020 07:51:43 +0100 (CET) From: =?UTF-8?q?Mattias=20R=C3=B6nnblom?= To: CC: , , , =?UTF-8?q?Mattias=20R=C3=B6nnblom?= Date: Mon, 9 Mar 2020 07:51:03 +0100 Message-ID: <20200309065106.23800-6-mattias.ronnblom@ericsson.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200309065106.23800-1-mattias.ronnblom@ericsson.com> References: <20200309065106.23800-1-mattias.ronnblom@ericsson.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:192.176.1.74; IPV:; CTRY:SE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(396003)(39860400002)(136003)(346002)(376002)(199004)(189003)(6666004)(356004)(4326008)(70586007)(2906002)(8676002)(336012)(5660300002)(478600001)(7636002)(70206006)(316002)(246002)(26005)(8936002)(186003)(86362001)(54906003)(107886003)(66574012)(6916009)(1076003)(2616005)(956004)(36756003); DIR:OUT; SFP:1101; SCL:1; SRVR:DB6PR0701MB2311; H:oa.msg.ericsson.com; FPR:; SPF:Pass; LANG:en; PTR:office365.se.ericsson.net; A:1; MX:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a02fc77d-9b69-4ae5-6dcb-08d7c3f6551f X-MS-TrafficTypeDiagnostic: DB6PR0701MB2311: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-Forefront-PRVS: 0337AFFE9A X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Ei2CAiXGtht/1pxaV0Hz4ZY3wjmpGgUh5H2BEdeZ2AcOeZXC0TCjNUUoX7EtJ0tERe3C0ff2DIcIorajbsvOqYgp1pCm944l2YD1Vqj2CPSJd0ojvH5Oip251z0/OK+JyjVyJONYunAVjm5Rx8V7acUgEFNOupw2rE2hpiuSH6z207up1O60DO/eK+8dM+Pjx32wUg1KrpuARq11Kf9D6SU8qXqV4W/yuUvoXL2NBzHkEbtdclJRHTLwpJNdXTyFUHzACbtewGyWXJzj+ZOvB+bkggN1EX5Wg9LQmo0FAaxVEBTvS0Ds1hdYnz8mBf+T0AEmdtPLKYejl6GEWN0F6YAD7AGT6ryCmk8onRkzmyDO2YPMf+UA65mE4lQI8OvDQKfpyruQiKu9SFbkuCWXSo7mLXqXfUQ9E14co220WZKQ53g0+/t4mB0/G2MuhUmo X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Mar 2020 06:51:45.3088 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a02fc77d-9b69-4ae5-6dcb-08d7c3f6551f X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f; Ip=[192.176.1.74]; Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR0701MB2311 Subject: [dpdk-dev] [PATCH 5/8] event/dsw: avoid migration waves in large systems X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" DSW limits the rate of migrations on a per-port basis. Hence, as the number of cores grows, so does the total migration capacity. In high core-count systems, this allows for a situation where flows are migrated to a lightly loaded port which recently already received a number of new flows (from other ports). The processing load generated by these new flows may not yet be reflected in the lightly loaded port's load estimate. The result is that the previously lightly loaded port is now overloaded. This patch adds a rough estimate of the size of the inbound migrations to a particular port, which can be factored into the migration logic, avoiding the above problem. Signed-off-by: Mattias Rönnblom --- drivers/event/dsw/dsw_evdev.c | 1 + drivers/event/dsw/dsw_evdev.h | 2 ++ drivers/event/dsw/dsw_event.c | 18 ++++++++++++++++-- 3 files changed, 19 insertions(+), 2 deletions(-) diff --git a/drivers/event/dsw/dsw_evdev.c b/drivers/event/dsw/dsw_evdev.c index 7798a38ad..e796975df 100644 --- a/drivers/event/dsw/dsw_evdev.c +++ b/drivers/event/dsw/dsw_evdev.c @@ -62,6 +62,7 @@ dsw_port_setup(struct rte_eventdev *dev, uint8_t port_id, port->ctl_in_ring = ctl_in_ring; rte_atomic16_init(&port->load); + rte_atomic32_init(&port->immigration_load); port->load_update_interval = (DSW_LOAD_UPDATE_INTERVAL * rte_get_timer_hz()) / US_PER_S; diff --git a/drivers/event/dsw/dsw_evdev.h b/drivers/event/dsw/dsw_evdev.h index ced40ef8d..6cb77cfc4 100644 --- a/drivers/event/dsw/dsw_evdev.h +++ b/drivers/event/dsw/dsw_evdev.h @@ -220,6 +220,8 @@ struct dsw_port { /* Estimate of current port load. */ rte_atomic16_t load __rte_cache_aligned; + /* Estimate of flows currently migrating to this port. */ + rte_atomic32_t immigration_load __rte_cache_aligned; } __rte_cache_aligned; struct dsw_queue { diff --git a/drivers/event/dsw/dsw_event.c b/drivers/event/dsw/dsw_event.c index 21c102275..f87656703 100644 --- a/drivers/event/dsw/dsw_event.c +++ b/drivers/event/dsw/dsw_event.c @@ -160,6 +160,11 @@ dsw_port_load_update(struct dsw_port *port, uint64_t now) (DSW_OLD_LOAD_WEIGHT+1); rte_atomic16_set(&port->load, new_load); + + /* The load of the recently immigrated flows should hopefully + * be reflected the load estimate by now. + */ + rte_atomic32_set(&port->immigration_load, 0); } static void @@ -362,7 +367,13 @@ dsw_retrieve_port_loads(struct dsw_evdev *dsw, int16_t *port_loads, uint16_t i; for (i = 0; i < dsw->num_ports; i++) { - int16_t load = rte_atomic16_read(&dsw->ports[i].load); + int16_t measured_load = rte_atomic16_read(&dsw->ports[i].load); + int32_t immigration_load = + rte_atomic32_read(&dsw->ports[i].immigration_load); + int32_t load = measured_load + immigration_load; + + load = RTE_MIN(load, DSW_MAX_LOAD); + if (load < load_limit) below_limit = true; port_loads[i] = load; @@ -491,6 +502,9 @@ dsw_select_emigration_target(struct dsw_evdev *dsw, target_qfs[*targets_len] = *candidate_qf; (*targets_len)++; + rte_atomic32_add(&dsw->ports[candidate_port_id].immigration_load, + candidate_flow_load); + return true; } @@ -503,7 +517,7 @@ dsw_select_emigration_targets(struct dsw_evdev *dsw, struct dsw_queue_flow *target_qfs = source_port->emigration_target_qfs; uint8_t *target_port_ids = source_port->emigration_target_port_ids; uint8_t *targets_len = &source_port->emigration_targets_len; - uint8_t i; + uint16_t i; for (i = 0; i < DSW_MAX_FLOWS_PER_MIGRATION; i++) { bool found; -- 2.17.1