From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id AF789A034F;
	Tue,  9 Nov 2021 13:37:02 +0100 (CET)
Received: from [217.70.189.124] (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 08BFC41147;
	Tue,  9 Nov 2021 13:36:45 +0100 (CET)
Received: from NAM11-CO1-obe.outbound.protection.outlook.com
 (mail-co1nam11on2053.outbound.protection.outlook.com [40.107.220.53])
 by mails.dpdk.org (Postfix) with ESMTP id 9526D4111B;
 Tue,  9 Nov 2021 13:36:41 +0100 (CET)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=gas5zA484wQdNa6iMYI2Yla6aQC5XWI0QytuGHitwP/ubriLZ1+xC1iIHeTe+L2sn39TE+njOFEsaYOOw6qjYEtoQXWfYAYQJYbkAwPhVnG18o2lPUzdsYsRor/pIhqzMNOhGIziWNZ3oevYmhrVtiPNC6wkOavFelLwaM3Qf2IrtUfL/aj0GMHtGfhamELjFAvnqeofp2WpBIfpcbinjG4KOCI2YUPnHRNftEx9tFmKL716nF3KwUAM8tie/kijIsWXUHpiajr4gYnRVsTO/e3PFUmqo8+HWorv+3lwp7IFOmphfmz/6j8ypYeLC9KmPE3VsXl9ctYST02ww0yZog==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=D6ewkXlxV/EWrI3vsOGCCOAGZzYRr+e4jclystuH3II=;
 b=h2QX6r/Rco2eoOWXeap94nnSBs78RASZRwqfTY5IxWJCuOEwU0uZJ1J1V6SjgSbdSj6sbsNLxHoSyKkhCYYzi5I5c6I0pzyVJYrm4/W/8TE4a/sG5gd/76D9gYnyj9HvWpjpA+DYx8reAhMCCGZCKg3UibWGBGekDCZsdayOBG0dMlGiolHo1sDY9Yf1J0Kx9qvW+TIzGuRTRGSGoeKa3KvFNLnRdd9z1lKNN0cCWQxkKdv0GsfxGxuNrIJddf+g+jdZ/x7SDThX6xfogCc/Fh4M4YwIlwjk4RTW7aNUeJDIKM6z/etpUpDJeuJfRbo9yte3lyRJjF2iyD+wwNDEeA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is
 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com;
 dmarc=pass (p=quarantine sp=quarantine pct=100) action=none
 header.from=nvidia.com; dkim=none (message not signed); arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=D6ewkXlxV/EWrI3vsOGCCOAGZzYRr+e4jclystuH3II=;
 b=UG38bzDI3CnbXyYQChycZxbsQxKNWNn2yw9GZgLG1blWTlLRIzxqjqck/CVXWpI2ym+U6pk/QhfIZNOVvAFQym1dORYWnSg2p4tv09Lms7C8zcaZ0yrp2ruRRqOUcdhBpqvidFrwZt01mrPX2FNIXxh2W5I1VO45sfuyP4EZVc5tfGLcS7gLCKy62cVrY/HjcGMlAKo4Q2q9m2iwyjYiaizzjZ4RvsoLcF4g69Ph9sQd1TUDGiJ3C51PBUW9KAVwMFNYk10sfODuFJkFCLM/Dc8GaJhgDTOb1i4rcG+EstSxl+r6NEr3yWO94kpNrfJ5XXDL5hy6y8Mxqkk6DbDD1w==
Received: from MWHPR15CA0050.namprd15.prod.outlook.com (2603:10b6:301:4c::12)
 by MW2PR12MB2395.namprd12.prod.outlook.com (2603:10b6:907:3::16) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.11; Tue, 9 Nov
 2021 12:36:39 +0000
Received: from CO1NAM11FT065.eop-nam11.prod.protection.outlook.com
 (2603:10b6:301:4c:cafe::f6) by MWHPR15CA0050.outlook.office365.com
 (2603:10b6:301:4c::12) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4690.15 via Frontend
 Transport; Tue, 9 Nov 2021 12:36:39 +0000
X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34)
 smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed)
 header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.34 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.34; helo=mail.nvidia.com;
Received: from mail.nvidia.com (216.228.112.34) by
 CO1NAM11FT065.mail.protection.outlook.com (10.13.174.62) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id
 15.20.4669.10 via Frontend Transport; Tue, 9 Nov 2021 12:36:38 +0000
Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com
 (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 9 Nov
 2021 12:36:36 +0000
From: Matan Azrad <matan@nvidia.com>
To: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
CC: <dev@dpdk.org>, Thomas Monjalon <thomas@monjalon.net>, Michael Baum
 <michaelba@nvidia.com>, <stable@dpdk.org>
Date: Tue, 9 Nov 2021 14:36:12 +0200
Message-ID: <20211109123612.2301442-6-matan@nvidia.com>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20211109123612.2301442-1-matan@nvidia.com>
References: <20211108172113.2241853-1-matan@nvidia.com>
 <20211109123612.2301442-1-matan@nvidia.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
X-Originating-IP: [172.20.187.6]
X-ClientProxiedBy: HQMAIL101.nvidia.com (172.20.187.10) To
 HQMAIL107.nvidia.com (172.20.187.13)
X-EOPAttributedMessage: 0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 13126b01-5492-4acd-7876-08d9a37d936f
X-MS-TrafficTypeDiagnostic: MW2PR12MB2395:
X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr
X-Microsoft-Antispam-PRVS: <MW2PR12MB2395AAE370F36106D466A631DF929@MW2PR12MB2395.namprd12.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:246;
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: pL0uaFvZROcxqY7bEsixWiBLMBWPzUMvuNN4mV5kJS2R+JmfgZJUL5xgj4mwGBtuVLdXgZxoZrTjMQ/rFzUjxjFwQ2+glOEkBWd8tktqKzfyVuhFJuJAm05C4lOPS8Kolc39BE9HvqcwuERgctva+3ifyo26m6nbRXs4d1vYkzNCtWSAQY8b8a3JV+S3CY2zXmivftQN99X/uDvbrxGKaAr5yf6WP04Ni6w12Y/qpl6vShrIXKj7VjLbdJ8r6AmqCjRCAwB7emT7AH5MtilnIPuavvTdD1u15qCxTw4iGLuKlF5cx+C4PtKPLQz8r3wUjLf5WzSga1YCFKe8oUaYpCriLno0/LBVD1CgnwDH6j5WOH85pHs8q2UZgKbWmQAxfgWrKqglQTIzZKdeuyAloYwfu+NROOuN8yqeCRfy5L2M339M3pVfZytF8K0MVDeQfnj0CVhCnd23EWkBHJ6QW2a4xfA1U++5NjIoBwJTc3VdNgfI9QGN5ECNwnczuIUQVW4kW3tKhPza1+WEFCQa/q9cYiNBASHg4Wz9sV5aJriwatw0zfWW0h6smVycUTNUWpk6m62jxyZmbWl26TOPtq5LofqVF5FfZxJeidZKEdOmBAvlvHeJAW9vTpOYISHwBzydNrqp38pKL6AgTxujBv84W5CPZiFJuTxyKOgeQMyifXDafuVjdKpEY916M9nXlsIZTYir+FpiXo4OOA8T3w==
X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE;
 SFS:(4636009)(46966006)(36840700001)(316002)(36860700001)(36906005)(83380400001)(6666004)(37006003)(54906003)(82310400003)(7636003)(2616005)(70586007)(2906002)(8676002)(8936002)(70206006)(6636002)(508600001)(426003)(26005)(55016002)(36756003)(336012)(86362001)(5660300002)(7696005)(16526019)(186003)(4326008)(6862004)(6286002)(356005)(47076005)(1076003);
 DIR:OUT; SFP:1101; 
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2021 12:36:38.8421 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 13126b01-5492-4acd-7876-08d9a37d936f
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34];
 Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT065.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW2PR12MB2395
Subject: [dpdk-dev] [PATCH v3 5/5] net/mlx5: workaround MR creation for flow
 counter
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>

From: Michael Baum <michaelba@nvidia.com>

Due to kernel driver / FW issues in direct MKEY creation using the DevX
API, this patch replaces the counter MR creation to use wrapped mkey
API.

Fixes: 5382d28c2110 ("net/mlx5: accelerate DV flow counter transactions")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/net/mlx5/mlx5.c      |  8 +-------
 drivers/net/mlx5/mlx5.h      |  5 +----
 drivers/net/mlx5/mlx5_flow.c | 25 ++++++-------------------
 3 files changed, 8 insertions(+), 30 deletions(-)

diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index f5990dd757..2a3efb3588 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -522,7 +522,6 @@ mlx5_flow_aging_init(struct mlx5_dev_ctx_shared *sh)
 static void
 mlx5_flow_counters_mng_init(struct mlx5_dev_ctx_shared *sh)
 {
-	struct mlx5_hca_attr *attr = &sh->cdev->config.hca_attr;
 	int i;
 
 	memset(&sh->cmng, 0, sizeof(sh->cmng));
@@ -535,10 +534,6 @@ mlx5_flow_counters_mng_init(struct mlx5_dev_ctx_shared *sh)
 		TAILQ_INIT(&sh->cmng.counters[i]);
 		rte_spinlock_init(&sh->cmng.csl[i]);
 	}
-	if (sh->devx && !haswell_broadwell_cpu) {
-		sh->cmng.relaxed_ordering_write = attr->relaxed_ordering_write;
-		sh->cmng.relaxed_ordering_read = attr->relaxed_ordering_read;
-	}
 }
 
 /**
@@ -553,8 +548,7 @@ mlx5_flow_destroy_counter_stat_mem_mng(struct mlx5_counter_stats_mem_mng *mng)
 	uint8_t *mem = (uint8_t *)(uintptr_t)mng->raws[0].data;
 
 	LIST_REMOVE(mng, next);
-	claim_zero(mlx5_devx_cmd_destroy(mng->dm));
-	claim_zero(mlx5_os_umem_dereg(mng->umem));
+	mlx5_os_wrapped_mkey_destroy(&mng->wm);
 	mlx5_free(mem);
 }
 
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index c2a13b6de4..bdadd6e024 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -462,8 +462,7 @@ struct mlx5_flow_counter_pool {
 struct mlx5_counter_stats_mem_mng {
 	LIST_ENTRY(mlx5_counter_stats_mem_mng) next;
 	struct mlx5_counter_stats_raw *raws;
-	struct mlx5_devx_obj *dm;
-	void *umem;
+	struct mlx5_pmd_wrapped_mr wm;
 };
 
 /* Raw memory structure for the counter statistics values of a pool. */
@@ -494,8 +493,6 @@ struct mlx5_flow_counter_mng {
 	uint8_t pending_queries;
 	uint16_t pool_index;
 	uint8_t query_thread_on;
-	bool relaxed_ordering_read;
-	bool relaxed_ordering_write;
 	bool counter_fallback; /* Use counter fallback management. */
 	LIST_HEAD(mem_mngs, mlx5_counter_stats_mem_mng) mem_mngs;
 	LIST_HEAD(stat_raws, mlx5_counter_stats_raw) free_stat_raws;
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 2f30a35525..40625688b0 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -7775,7 +7775,6 @@ mlx5_counter_query(struct rte_eth_dev *dev, uint32_t cnt,
 static int
 mlx5_flow_create_counter_stat_mem_mng(struct mlx5_dev_ctx_shared *sh)
 {
-	struct mlx5_devx_mkey_attr mkey_attr;
 	struct mlx5_counter_stats_mem_mng *mem_mng;
 	volatile struct flow_counter_stats *raw_data;
 	int raws_n = MLX5_CNT_CONTAINER_RESIZE + MLX5_MAX_PENDING_QUERIES;
@@ -7785,6 +7784,7 @@ mlx5_flow_create_counter_stat_mem_mng(struct mlx5_dev_ctx_shared *sh)
 			sizeof(struct mlx5_counter_stats_mem_mng);
 	size_t pgsize = rte_mem_page_size();
 	uint8_t *mem;
+	int ret;
 	int i;
 
 	if (pgsize == (size_t)-1) {
@@ -7799,23 +7799,10 @@ mlx5_flow_create_counter_stat_mem_mng(struct mlx5_dev_ctx_shared *sh)
 	}
 	mem_mng = (struct mlx5_counter_stats_mem_mng *)(mem + size) - 1;
 	size = sizeof(*raw_data) * MLX5_COUNTERS_PER_POOL * raws_n;
-	mem_mng->umem = mlx5_os_umem_reg(sh->cdev->ctx, mem, size,
-						 IBV_ACCESS_LOCAL_WRITE);
-	if (!mem_mng->umem) {
-		rte_errno = errno;
-		mlx5_free(mem);
-		return -rte_errno;
-	}
-	memset(&mkey_attr, 0, sizeof(mkey_attr));
-	mkey_attr.addr = (uintptr_t)mem;
-	mkey_attr.size = size;
-	mkey_attr.umem_id = mlx5_os_get_umem_id(mem_mng->umem);
-	mkey_attr.pd = sh->cdev->pdn;
-	mkey_attr.relaxed_ordering_write = sh->cmng.relaxed_ordering_write;
-	mkey_attr.relaxed_ordering_read = sh->cmng.relaxed_ordering_read;
-	mem_mng->dm = mlx5_devx_cmd_mkey_create(sh->cdev->ctx, &mkey_attr);
-	if (!mem_mng->dm) {
-		mlx5_os_umem_dereg(mem_mng->umem);
+	ret = mlx5_os_wrapped_mkey_create(sh->cdev->ctx, sh->cdev->pd,
+					  sh->cdev->pdn, mem, size,
+					  &mem_mng->wm);
+	if (ret) {
 		rte_errno = errno;
 		mlx5_free(mem);
 		return -rte_errno;
@@ -7934,7 +7921,7 @@ mlx5_flow_query_alarm(void *arg)
 	ret = mlx5_devx_cmd_flow_counter_query(pool->min_dcs, 0,
 					       MLX5_COUNTERS_PER_POOL,
 					       NULL, NULL,
-					       pool->raw_hw->mem_mng->dm->id,
+					       pool->raw_hw->mem_mng->wm.lkey,
 					       (void *)(uintptr_t)
 					       pool->raw_hw->data,
 					       sh->devx_comp,
-- 
2.25.1