From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id DD3A442BA3;
	Fri, 26 May 2023 05:16:27 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id D09F442D42;
	Fri, 26 May 2023 05:15:54 +0200 (CEST)
Received: from NAM10-MW2-obe.outbound.protection.outlook.com
 (mail-mw2nam10on2083.outbound.protection.outlook.com [40.107.94.83])
 by mails.dpdk.org (Postfix) with ESMTP id 894A642D42
 for <dev@dpdk.org>; Fri, 26 May 2023 05:15:53 +0200 (CEST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=YwkDYjwndZw5NgglmNW5nLFXq4U1YJopXz0p/gIyfNCn0l+MUbCrQnGkmmEUSIn1U9Jo3kQetUML2tma/cpEMUdrMzkp8rG9UqdfjjatC0tZ8N1FXr2WAWA7LIMBOZp/MvmJk6hScMM2xDWKRfYCQzCp02bNyh/ulaXQ08kq8NNq5065rgX6YbUdRAYHLEZhNh8G0uMCRtdqCVAhvKL9ur6JlJfgKZ0yIhPogXzAcAXR3k9WxrG/j8SsoBKlk7EvkLzMUp3UTJLdAW2I8eWXYk4h7UKjuYXcbUBzDfylOJe4tPM7TicWVYOpmjJlHwVsZTHvmjAmpLb4HfgfMW5SZA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=ijmIr9v00YGPL9ba/rDGsddWtRTGyYVm0ro6P890GV4=;
 b=mJXzm1vtvUd96eFrH/abWXH86nNtvYTuH98rkRZiCmIvpAELn3EUC4s9SpbSS+A+AT4f1MRdi2C4+PILfW907N7GZkQBvrbftc0OvbjedLiZT5LokIPkY4M3f4XBdcqdsI6UR9jArsIIXQmq9lC+/6h4XS2DuScwGyBqMVyaEfbtLQkt2uxEy4WAF5JUGNvEQR1hn38YNku6WbtXegbl8NcGdHbwuYbI87bUwQSjZ1nTFYOFmDkMYUald25HkyRDdhgEl50//P117wET/Qj6VecZM+ory1IWuFcHOjjgPNmp62f2Iwguij1yz4zZxaPOtAFqbgIdjKmQ6HyaIeV4yg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is
 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com;
 dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com;
 dkim=none (message not signed); arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=ijmIr9v00YGPL9ba/rDGsddWtRTGyYVm0ro6P890GV4=;
 b=K6aKx1xNMAZ/RvgtuDEOvjJOzd9MLws+uVqs7cgzo+5A+IR+G7/Sbuk/QeblHxvqerPTIqqETAnDmzmnUt/rKKgabj2YxGzFoKA34iN3UPTA4SB7VubeV4E+0O65uDLvoxgzYy7McR+3dAHzlkbVKK2ZpZyEZUzjXZb7GopivdZQiJwNBX0dA9HAntKVSGN4zOQaolMMR8FHK9Dgtm17FhSUdb4qqPWgy0W0QthpH2QQGMAMFvllQC6lsm4JQ6I/Ui0eZr4WzA12809xdB+w/eVbiz8kcLeyf+Zbqp3Kve0d3EYVxWeRJkU1PLtLv+GNLUzY9mLhoFrrFvw0W/QuGw==
Received: from BN0PR10CA0027.namprd10.prod.outlook.com (2603:10b6:408:143::28)
 by IA1PR12MB6436.namprd12.prod.outlook.com (2603:10b6:208:3ac::12)
 with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6411.28; Fri, 26 May
 2023 03:15:50 +0000
Received: from BN8NAM11FT091.eop-nam11.prod.protection.outlook.com
 (2603:10b6:408:143:cafe::1e) by BN0PR10CA0027.outlook.office365.com
 (2603:10b6:408:143::28) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6433.17 via Frontend
 Transport; Fri, 26 May 2023 03:15:50 +0000
X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161)
 smtp.mailfrom=nvidia.com;
 dkim=none (message not signed)
 header.d=none;dmarc=pass action=none header.from=nvidia.com;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.117.161 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C
Received: from mail.nvidia.com (216.228.117.161) by
 BN8NAM11FT091.mail.protection.outlook.com (10.13.176.134) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.6433.18 via Frontend Transport; Fri, 26 May 2023 03:15:50 +0000
Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com
 (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Thu, 25 May 2023
 20:15:35 -0700
Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com
 (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Thu, 25 May
 2023 20:15:33 -0700
From: Suanming Mou <suanmingm@nvidia.com>
To: Matan Azrad <matan@nvidia.com>, Viacheslav Ovsiienko
 <viacheslavo@nvidia.com>
CC: <dev@dpdk.org>, <rasland@nvidia.com>
Subject: [PATCH v2 6/9] common/mlx5: add WQE-based QP synchronous basics
Date: Fri, 26 May 2023 06:14:18 +0300
Message-ID: <20230526031422.913377-7-suanmingm@nvidia.com>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20230526031422.913377-1-suanmingm@nvidia.com>
References: <20230418092325.2578712-1-suanmingm@nvidia.com>
 <20230526031422.913377-1-suanmingm@nvidia.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
X-Originating-IP: [10.126.231.35]
X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To
 rnnvmail201.nvidia.com (10.129.68.8)
X-EOPAttributedMessage: 0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: BN8NAM11FT091:EE_|IA1PR12MB6436:EE_
X-MS-Office365-Filtering-Correlation-Id: 375704ca-9088-48c8-decd-08db5d978227
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: CkSUqJbf7f4h/JcEezngO1G2kYAENENa0GLTYRZBEkcFEqYoTVb3p09069TnbheLOcN7F7umjcpTMlpByT6dxX06H+JOO0l0LQvd9gOzAd1zkl0Gk8lsQpD0Xcg4V+ss6hcLNTeK+KJHrv9LlECKdzRXFpv4MC/zuB7a3TzpUUO/4qkQBkbLXFzOBGMEgpUansPspuiFuvbzdCX7VK0Drv11tzcnA4kD0mcgIl7lAEsVvg/Ke9GPnM/xXnJxzzCyck/xS8u9MQEj2Wx19jBHfgF7n8B2NWd52Y5MkJajuZ/bvMqB6aXN2ajjS+rlxIov5/1C/ubfevZ/+7ymjE4A+fpftNwdbiIASLauEOB/i13HyDtJcd8mVbDwLzpzr3/lrOpJALk/j/cgi1lIMGSy7cIHkOK7GszjRyT6iBpHCrEaoStX095VidXa7K9OwzTtlD/ew1N/Npq1tFvsMDpkrk9lLzFgmbUvn6nqIUK269x0QVNrA4pnPgnkSLFPq01jyhzS4A6JY4DqQS6D1MODrKu/ZdR3xZkSqBhJmz2uh5HApeNz35I1qLZ8lewFFWoyPGBUP5E/1wCbZfZJcB/iUl7Yhn34fKT0vuuV4+v9et7jdprn2dJUCpI40epuT5azqlK1N9vOzYDGAQbXeycP1eeiWEHJ705izrrt4ONydk7jCNCcvLy+WkEs//2cNYfNltBiOnh+hMKqyHSgHM7Ip3Wv4ENJjIEamWoRzeUBSME=
X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE;
 SFS:(13230028)(4636009)(136003)(376002)(39860400002)(396003)(346002)(451199021)(46966006)(40470700004)(36840700001)(82740400003)(6666004)(7696005)(26005)(36860700001)(1076003)(6286002)(16526019)(107886003)(186003)(86362001)(110136005)(478600001)(54906003)(7636003)(356005)(336012)(47076005)(426003)(2616005)(4326008)(70586007)(70206006)(40460700003)(316002)(6636002)(5660300002)(8936002)(8676002)(2906002)(55016003)(41300700001)(40480700001)(36756003)(82310400005);
 DIR:OUT; SFP:1101; 
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 May 2023 03:15:50.6278 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 375704ca-9088-48c8-decd-08db5d978227
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161];
 Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT091.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6436
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org

Nvidia HW provides a synchronous mechanism between QPs. When
creating the QPs, user can set one as primary and another as
follower. The follower QP's WQE execution can be controlled
by primary QP via SEND_EN WQE.

This commit introduces the SEND_EN WQE to improve the WQE
execution sync-up between primary and follower QPs.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c |  6 ++++++
 drivers/common/mlx5/mlx5_devx_cmds.h |  3 +++
 drivers/common/mlx5/mlx5_prm.h       | 11 +++++++++++
 3 files changed, 20 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 4332081165..ef87862a6d 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -2475,6 +2475,12 @@ mlx5_devx_cmd_create_qp(void *ctx,
 				 attr->dbr_umem_valid);
 			MLX5_SET(qpc, qpc, dbr_umem_id, attr->dbr_umem_id);
 		}
+		if (attr->cd_master)
+			MLX5_SET(qpc, qpc, cd_master, attr->cd_master);
+		if (attr->cd_slave_send)
+			MLX5_SET(qpc, qpc, cd_slave_send, attr->cd_slave_send);
+		if (attr->cd_slave_recv)
+			MLX5_SET(qpc, qpc, cd_slave_receive, attr->cd_slave_recv);
 		MLX5_SET64(qpc, qpc, dbr_addr, attr->dbr_address);
 		MLX5_SET64(create_qp_in, in, wq_umem_offset,
 			   attr->wq_umem_offset);
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index cb3f3a211b..e071cd841f 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -559,6 +559,9 @@ struct mlx5_devx_qp_attr {
 	uint64_t wq_umem_offset;
 	uint32_t user_index:24;
 	uint32_t mmo:1;
+	uint32_t cd_master:1;
+	uint32_t cd_slave_send:1;
+	uint32_t cd_slave_recv:1;
 };
 
 struct mlx5_devx_virtio_q_couners_attr {
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 3b26499a47..96d5eb8de3 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -589,6 +589,17 @@ struct mlx5_rdma_write_wqe {
 	struct mlx5_wqe_dseg dseg[];
 } __rte_packed;
 
+struct mlx5_wqe_send_en_seg {
+	uint32_t reserve[2];
+	uint32_t sqnpc;
+	uint32_t qpn;
+} __rte_packed;
+
+struct mlx5_wqe_send_en_wqe {
+	struct mlx5_wqe_cseg ctr;
+	struct mlx5_wqe_send_en_seg sseg;
+} __rte_packed;
+
 #ifdef PEDANTIC
 #pragma GCC diagnostic error "-Wpedantic"
 #endif
-- 
2.25.1