From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id EEDA1A057B; Tue, 24 Mar 2020 12:40:01 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D5AA81C0C9; Tue, 24 Mar 2020 12:40:00 +0100 (CET) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2044.outbound.protection.outlook.com [40.107.21.44]) by dpdk.org (Postfix) with ESMTP id 78DDC1C0C4 for ; Tue, 24 Mar 2020 12:39:59 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LspWEc34yV6rSEsCrU0mI0Ew07Z9tfY6yDaF0EblH7i91+YDs9Y2W3GhmPq7h/yZ7wQ8GjTZTg+oanfIcSdZEAf+oxcjJA7DjwpA68ci+bPGYQYcv3QtubwXuks9bGnCMTMahQbu2vmAInfCWQfaFc47dFHoYSbP8bVOiuSiykIpZeej72k4Lb0RcgrH4W9nPuiVs+o58rQmBwnjJ50CB6l1XC9XqWPuV6kKJtwaTPYNA4idpsXzgeAdNfONJWps2pttgwNgSqptavoZxs1cltmO6WIztpnR7m6xLDYaTn4oXwzD1n9GkdsagEMNEOGg9ddQSPGAVoFQ+D9qoDLJ7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rLrOn7634AppZCaLPCogm7KT5ZRJ2rwUfNpiy2QyvAk=; b=kwmk7JoExV+HwfI/T66qdUt38m454j+coW8O8jBgLw7ja6ENWsZN61CSrlgKyjjd6olxaCphsqin+LTV2McWsx/O7FJehgwwQ1Agw2CcADhvm5fkTJ6QPX92GEjN5+Es6RhkbgXfedLacF23UIQXWXJ5Ff69HGgKN/drpgE5XrY+cWTYA50iC3U7CERuFvg+QjNYIKfoBjt+bFdjDGiqDnQp2vugH7c6zZ+PnlslEJX+gO0ZUSyPwsPVaTyOohmcnyVX4qpgKD+73yvkUbWjLPzWAU3cSLEr1f4Jylvdvg+PzBMGrTZqck18Syt+u8b5pxpHOYpI14EtG68e8+ve7Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rLrOn7634AppZCaLPCogm7KT5ZRJ2rwUfNpiy2QyvAk=; b=o0y51S3kQm7B0BNp9FTTwCBgRQOx4DOrig7sS2on8jqRNEuhATSDb+BH3XImz9HvbG3UgU5TkFQluMj4OCdtFRhgit0hmJA6FWQe8yPd+0Ib4xrL9Df6x0sdYsd1TMH3nJgG0ePAG3t4kHApJ/yU6zgh9GjGC+jPupeX7/H9g+8= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=shirik@mellanox.com; Received: from AM0PR0502MB3874.eurprd05.prod.outlook.com (52.133.45.29) by AM0PR0502MB3905.eurprd05.prod.outlook.com (52.133.45.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2835.18; Tue, 24 Mar 2020 11:39:56 +0000 Received: from AM0PR0502MB3874.eurprd05.prod.outlook.com ([fe80::e1d9:634c:6a67:8d59]) by AM0PR0502MB3874.eurprd05.prod.outlook.com ([fe80::e1d9:634c:6a67:8d59%7]) with mapi id 15.20.2835.023; Tue, 24 Mar 2020 11:39:56 +0000 From: Shiri Kuzin To: dev@dpdk.org Cc: matan@mellanox.com, rasland@mellanox.com, viacheslavo@mellanox.com Date: Tue, 24 Mar 2020 13:39:39 +0200 Message-Id: <1585049979-12990-1-git-send-email-shirik@mellanox.com> X-Mailer: git-send-email 1.8.3.1 Content-Type: text/plain X-ClientProxiedBy: AM4PR0701CA0006.eurprd07.prod.outlook.com (2603:10a6:200:42::16) To AM0PR0502MB3874.eurprd05.prod.outlook.com (2603:10a6:208:1a::29) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mellanox.com (94.188.199.18) by AM4PR0701CA0006.eurprd07.prod.outlook.com (2603:10a6:200:42::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2856.12 via Frontend Transport; Tue, 24 Mar 2020 11:39:56 +0000 X-Mailer: git-send-email 1.8.3.1 X-Originating-IP: [94.188.199.18] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 2668f5ea-53f4-42ea-47ac-08d7cfe813b4 X-MS-TrafficTypeDiagnostic: AM0PR0502MB3905:|AM0PR0502MB3905: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtFwd X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:185; X-Forefront-PRVS: 03524FBD26 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4636009)(366004)(39850400004)(346002)(136003)(396003)(376002)(55016002)(2906002)(86362001)(316002)(36756003)(81156014)(6666004)(66476007)(8676002)(8886007)(66556008)(66946007)(8936002)(5660300002)(4326008)(107886003)(81166006)(16526019)(6916009)(7696005)(52116002)(478600001)(956004)(186003)(2616005)(26005); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB3905; H:AM0PR0502MB3874.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: e7cBARgWvTCzBld957A/SemGaoEbw1K+ZCDLc3sf+gT6S72doH+7r0Fe8geEfmg56tt4LdHK/BUCXavukysEzcZAswxx3O8wL7Xaa6oAA/VpYr8JF5pnSZ5feJ4d3h03YU030ps1rnQ9Bzrea4GwazpiUqWRfbLm4RBOGa7jpjpD/BYWAoTqj9ApxTtmX0olMTMvU5kgMSXQBpAXN3nCg0G2QgBKpK0RaJ7n9mPKyoBCXH7IdL2q++JxSErJUaj75P/8G6D9SrE7IiibU0snZc1aSj9gOg8ZsyO+Zc9JV8CdRzMh754vvOlyHS/p+GqWp1GapOZv4qo9VEP/j40JSH8gqinhNl6KuVdtZ9ac85XYLifNl3CEXGdua59uNtuYKZYrHm5Sv25+Mce8PgflpyiClzEOCwm0Z61Fk4PgkRKc2jT2BR9WD6qbfpOdWjtZ X-MS-Exchange-AntiSpam-MessageData: 8Q/rvyqfXrh4A72NKd+9G84tPFttc3GKQrGH9l0uXZXblzY0eaXh6dmYWDAGgMZ/cjNTHCJMicIAp6h7KxQlsVHl1QFK4SgEtupyyR2xMpuaKQ6mfdnB5soDiiLzcj75eTEuMjdI/XCFBb5Q6vKOuA== X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2668f5ea-53f4-42ea-47ac-08d7cfe813b4 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Mar 2020 11:39:56.8387 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: TUMvXZVrCD2sA5CmJoeDBFEaY+EAFwNgLfUtr1qi439YD3DSxKynr3EcEplJKJYOgiGoOcmS9U6gZ/arZr1NRg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB3905 Subject: [dpdk-dev] [PATCH] net/mlx5: creating relaxed ordering memory regions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" In the current state, when preforming read/write transactions we must wait for a completion in order to run the next transaction, and all transactions are performed by order. Relaxed Ordering is a PCI optimization which by enabling it we allow the system to perform read/writes in a different order without having to wait for completion and improve the performance in that matter. This commit introduces the creation of relaxed ordering memory regions in mlx5. As relaxed ordering is an optimization, drivers that do not support it can simply ignore it and therefore it is enabled by default. Signed-off-by: Shiri Kuzin Acked-by: Matan Azrad --- doc/guides/rel_notes/release_20_05.rst | 2 +- drivers/common/mlx5/mlx5_devx_cmds.c | 4 ++++ drivers/common/mlx5/mlx5_devx_cmds.h | 1 + drivers/common/mlx5/mlx5_glue.h | 4 ++++ drivers/common/mlx5/mlx5_prm.h | 4 +++- drivers/net/mlx5/mlx5_flow_dv.c | 1 + drivers/net/mlx5/mlx5_mr.c | 6 ++++-- drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 1 + drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 1 + 9 files changed, 20 insertions(+), 4 deletions(-) diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst index 000bbf5..c960fd2 100644 --- a/doc/guides/rel_notes/release_20_05.rst +++ b/doc/guides/rel_notes/release_20_05.rst @@ -61,7 +61,7 @@ New Features Updated Mellanox mlx5 driver with new features and improvements, including: * Added support for matching on IPv4 Time To Live and IPv6 Hop Limit. - + * Added support for creating Relaxed Ordering Memory Regions. Removed Items ------------- diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c index d960bc9..1157a44 100644 --- a/drivers/common/mlx5/mlx5_devx_cmds.c +++ b/drivers/common/mlx5/mlx5_devx_cmds.c @@ -196,6 +196,10 @@ struct mlx5_devx_obj * MLX5_SET(mkc, mkc, pd, attr->pd); MLX5_SET(mkc, mkc, mkey_7_0, attr->umem_id & 0xFF); MLX5_SET(mkc, mkc, translations_octword_size, translation_size); + if (attr->relaxed_ordering == 1) { + MLX5_SET(mkc, mkc, relaxed_ordering_write, 0x1); + MLX5_SET(mkc, mkc, relaxed_ordering_read, 0x1); + } MLX5_SET64(mkc, mkc, start_addr, attr->addr); MLX5_SET64(mkc, mkc, len, attr->size); mkey->obj = mlx5_glue->devx_obj_create(ctx, in, in_size_dw * 4, out, diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h index 6912dc6..20bb294 100644 --- a/drivers/common/mlx5/mlx5_devx_cmds.h +++ b/drivers/common/mlx5/mlx5_devx_cmds.h @@ -33,6 +33,7 @@ struct mlx5_devx_mkey_attr { uint32_t pd; uint32_t log_entity_size; uint32_t pg_access:1; + uint32_t relaxed_ordering:1; struct mlx5_klm *klm_array; int klm_num; }; diff --git a/drivers/common/mlx5/mlx5_glue.h b/drivers/common/mlx5/mlx5_glue.h index 6238b43..cd1136f 100644 --- a/drivers/common/mlx5/mlx5_glue.h +++ b/drivers/common/mlx5/mlx5_glue.h @@ -98,6 +98,10 @@ uint64_t comp_mask; }; #endif +#ifndef IBV_ACCESS_RELAXED_ORDERING +#define IBV_ACCESS_RELAXED_ORDERING 0 +#endif + /* LIB_GLUE_VERSION must be updated every time this structure is modified. */ struct mlx5_glue { const char *version; diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index eee3a4b..00fd7c1 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -882,7 +882,9 @@ struct mlx5_ifc_mkc_bits { u8 translations_octword_size[0x20]; - u8 reserved_at_1c0[0x1b]; + u8 reserved_at_1c0[0x19]; + u8 relaxed_ordering_read[0x1]; + u8 reserved_at_1da[0x1]; u8 log_page_size[0x5]; u8 reserved_at_1e0[0x20]; diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c index 2090631..809833b 100644 --- a/drivers/net/mlx5/mlx5_flow_dv.c +++ b/drivers/net/mlx5/mlx5_flow_dv.c @@ -3962,6 +3962,7 @@ struct field_modify_info modify_tcp[] = { mkey_attr.pg_access = 0; mkey_attr.klm_array = NULL; mkey_attr.klm_num = 0; + mkey_attr.relaxed_ordering = 1; mem_mng->dm = mlx5_devx_cmd_mkey_create(sh->ctx, &mkey_attr); if (!mem_mng->dm) { mlx5_glue->devx_umem_dereg(mem_mng->umem); diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c index cb97c87..fd1245d 100644 --- a/drivers/net/mlx5/mlx5_mr.c +++ b/drivers/net/mlx5/mlx5_mr.c @@ -768,7 +768,8 @@ struct mr_update_mp_data { * through mlx5_alloc_verbs_buf(). */ mr->ibv_mr = mlx5_glue->reg_mr(sh->pd, (void *)data.start, len, - IBV_ACCESS_LOCAL_WRITE); + IBV_ACCESS_LOCAL_WRITE | + IBV_ACCESS_RELAXED_ORDERING); if (mr->ibv_mr == NULL) { DEBUG("port %u fail to create a verbs MR for address (%p)", dev->data->port_id, (void *)addr); @@ -1217,7 +1218,8 @@ struct mr_update_mp_data { if (mr == NULL) return NULL; mr->ibv_mr = mlx5_glue->reg_mr(priv->sh->pd, (void *)addr, len, - IBV_ACCESS_LOCAL_WRITE); + IBV_ACCESS_LOCAL_WRITE | + IBV_ACCESS_RELAXED_ORDERING); if (mr->ibv_mr == NULL) { DRV_LOG(WARNING, "port %u fail to create a verbs MR for address (%p)", diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c index 3358704..4457760 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c @@ -39,6 +39,7 @@ .pg_access = 1, .klm_array = NULL, .klm_num = 0, + .relaxed_ordering = 0, }; struct mlx5_devx_virtq_attr attr = { .type = MLX5_VIRTQ_MODIFY_TYPE_DIRTY_BITMAP_PARAMS, diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c index 398ca35..da31b47 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c @@ -263,6 +263,7 @@ mkey_attr.pg_access = 1; mkey_attr.klm_array = NULL; mkey_attr.klm_num = 0; + mkey_attr.relaxed_ordering = 0; entry->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr); if (!entry->mkey) { DRV_LOG(ERR, "Failed to create direct Mkey."); -- 1.8.3.1