From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 1F50EA0547;
	Thu, 29 Apr 2021 17:48:52 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 2DBC741361;
	Thu, 29 Apr 2021 17:48:42 +0200 (CEST)
Received: from NAM11-BN8-obe.outbound.protection.outlook.com
 (mail-bn8nam11on2047.outbound.protection.outlook.com [40.107.236.47])
 by mails.dpdk.org (Postfix) with ESMTP id DD37141216
 for <dev@dpdk.org>; Thu, 29 Apr 2021 17:48:40 +0200 (CEST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=NyKo89zc49LXGEOtketIl4r7jqRNK8qvZdAMmq56bGRmZ2ywF8twEdrz4/fPqdwCXgL8nH+eW4Ijzu+NSimy4VRO+PtWgmDYPx71NrZQZVIYlukoFbBI79SchvAsA9wegPFdBW2wNHEwjM3kHIVn4JWIHz3CeZuOAHWWjGd7jMTybK7xHQeajg/maoa2VOIySAgBBiElYrOFS63yrrHb2MkUD9Gd95o1PpwbbnqUkcIUxnlU6+tQCZv+P1XBHhU7EU0FU16nsohmt40TlZwZM7nTgyVAx0tFDW76slFCvlaBWtGmGVgbDwOjZt+KeZEuPk9AZ2FS2uMIu3B704dK/Q==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=EAqR7vtlHlBB8bie+e9pkBEJ2gUdk1gFB6zmWtmxtG0=;
 b=fuU2q9Dp6VCHGF6XrhLPl7WhiCWDoPV3EWQXOLHZDjODr+AW2ypF3EftBbDtEJodMWgtsNRLah7a1xrWBL4KGqLUFKqsy7lFP94fEObzXfHy48rDsFQ6+4ZylH4MwsTpy6HCWQ77PZRjUfBZJbYxCE1ESPkyCf/tlarTFcm70XBbKdqr6RKn8XYeU2EhQj2NPIJSEqrPppdh2tPGSxfJ6CE7oAM8e+3VSBGsSs2INqy4ZiNuBJu8BGxrYWQrlCsi6VkhAaktUftqJUOKdr2kJiQ6r1FVt5XvuK+YxnFNmmOVgKhdV9nhp+l1oWq0AHamp5oN90voFZ/CeoR9m5798w==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is
 216.228.112.34) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com;
 dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com;
 dkim=none (message not signed); arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=EAqR7vtlHlBB8bie+e9pkBEJ2gUdk1gFB6zmWtmxtG0=;
 b=cSllVxykCuM9KDD0VWeWDlAMQ88yEhMCw1cgqEjFVVN3a4e/MIg6MHfg5VN6Zt7egxrSdzBu2WHtihxg3b913AJlaauzZxUmxmrp5U5Lx9tv7GQgYPup18QWvYtoR3s7Tl2CDlYwjnf5+soWS/pL7Q6MxQRIpUM6XdVFHI0IHXrSgaXcNfYqmjLHkkBsn7G2b1p96Q7HkacCDrV2e+XnIeoOln8daBOIYbLgZlAPLHKrazKxaROWLQXSQBZuVIXUnFTDRAOn0pVtv3O20ful7ZmT2Jrdk14o0cfhMSpnW609eFIk5zFD/80rJP8sci5CHWXYK+6FEBkbxt8fRlIEEg==
Received: from MW4PR04CA0313.namprd04.prod.outlook.com (2603:10b6:303:82::18)
 by DM5PR1201MB0058.namprd12.prod.outlook.com (2603:10b6:4:50::11)
 with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4065.24; Thu, 29 Apr
 2021 15:48:39 +0000
Received: from CO1NAM11FT066.eop-nam11.prod.protection.outlook.com
 (2603:10b6:303:82:cafe::da) by MW4PR04CA0313.outlook.office365.com
 (2603:10b6:303:82::18) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4087.29 via Frontend
 Transport; Thu, 29 Apr 2021 15:48:39 +0000
X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34)
 smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed)
 header.d=none;marvell.com; dmarc=pass action=none header.from=nvidia.com;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.34 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.34; helo=mail.nvidia.com;
Received: from mail.nvidia.com (216.228.112.34) by
 CO1NAM11FT066.mail.protection.outlook.com (10.13.175.18) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id
 15.20.4087.27 via Frontend Transport; Thu, 29 Apr 2021 15:48:39 +0000
Received: from nvidia.com (172.20.145.6) by HQMAIL107.nvidia.com
 (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 29 Apr
 2021 15:48:37 +0000
From: Matan Azrad <matan@nvidia.com>
To: <dev@dpdk.org>
CC: <matan@nvidia.com>, <gakhil@marvell.com>, <suanmingm@nvidia.com>
Date: Thu, 29 Apr 2021 18:47:08 +0300
Message-ID: <20210429154712.2820159-12-matan@nvidia.com>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20210429154712.2820159-1-matan@nvidia.com>
References: <20210408204849.9543-1-shirik@nvidia.com>
 <20210429154712.2820159-1-matan@nvidia.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
X-Originating-IP: [172.20.145.6]
X-ClientProxiedBy: HQMAIL107.nvidia.com (172.20.187.13) To
 HQMAIL107.nvidia.com (172.20.187.13)
X-EOPAttributedMessage: 0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 2152c74e-c1a6-439b-af2d-08d90b2641e0
X-MS-TrafficTypeDiagnostic: DM5PR1201MB0058:
X-Microsoft-Antispam-PRVS: <DM5PR1201MB00588E36013E91F014023571DF5F9@DM5PR1201MB0058.namprd12.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:193;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: WpBZ5KhzzuZhpBOK5jgjo1bryMfRdpRG5g3lw+BwPPPV0dOIdPxdWPDmvfxf4L2XxVYFXed36eF2S0eVAE1jiYUoYlAjvDT5y53dIzSFJKxYJ1bgD9tE81LdZh9ujJ0+FrUD0grntvgJhvu4UL0xEO+SZzCnDZjzXMBwqPQQgtlHvjIJ84wsU2NmB6T1ev+XqIVtjWsX5GCrLb0T7IbfsHAO+cGZQdiKotXY+hCU7GJ/Ar63bJBsRvqkVDoE/hSE3FujQAJ8QblstZ6X+Pb2q/OOp1rdcHCNEaNXS5djmdmxovE2U1wzyZed43NltIoZmqV42de2Nw5XecgaSAgJ9QJW9x4zc940A+t5/q7IsiY0qdUBIaJgWZmYc6qJFUc2p7A0tok564ECzDwu+y/B1xtCUsAZ/KLE68iL4TKfKQLJtsG/r5R656PuPPPgEsrWuWBg8eZbZNz0Ss+9dcLiRdXbactqSTeed09k8R5j25wc1AH0+gRbqC3Q0KnvuMhi6/TUYrQ6KypLoJ7wqHqV+ciMb57I7BgkG1K7DBa5Oe+0Z4IrRt6RAHjptSlvDMSjpIatBNDEC7CtIBnGAxz+XxyL+EIdT5AzUvEf90seNviZcqQIfUr7n6DJz89FJkLRk87O2XDZ9Z2N5EhTjCa6CbVeD8Md+/Cw/DxJ+RucKao=
X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE;
 SFS:(4636009)(346002)(136003)(39860400002)(376002)(396003)(46966006)(36840700001)(1076003)(47076005)(4326008)(426003)(478600001)(26005)(82310400003)(336012)(70586007)(36906005)(36756003)(6286002)(316002)(6666004)(54906003)(356005)(70206006)(86362001)(7696005)(186003)(2616005)(5660300002)(6916009)(8676002)(7636003)(82740400003)(107886003)(2906002)(36860700001)(16526019)(8936002)(83380400001)(55016002);
 DIR:OUT; SFP:1101; 
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Apr 2021 15:48:39.0513 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 2152c74e-c1a6-439b-af2d-08d90b2641e0
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34];
 Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT066.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1201MB0058
Subject: [dpdk-dev] [PATCH v2 11/15] crypto/mlx5: add maximum segments
 device argument
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>

From: Suanming Mou <suanmingm@nvidia.com>

The mlx5 HW crypto operations are done by attaching crypto property
to a memory region. Once done, every access to the memory via the
crypto-enabled memory region will result with in-line encryption or
decryption of the data.

As a result, the design choice is to provide two types of WQEs. One
is UMR WQE which sets the crypto property and the other is rdma write
WQE which sends DMA command to copy data from local MR to remote MR.

The size of the WQEs will be defined by a new devarg called
max_segs_num.

This devarg also defines the maximum segments in mbuf chain that will be
supported for crypto operations.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 35 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  7 +++++++
 2 files changed, 38 insertions(+), 4 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 9f270e4382..5e8e7b63d0 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -21,6 +21,7 @@
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
+#define MLX5_CRYPTO_MAX_SEGS 56
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -464,14 +465,24 @@ mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
 		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
 		return -errno;
 	}
-	if (strcmp(key, "import_kek_id") == 0)
+	if (strcmp(key, "max_segs_num") == 0) {
+		if (!tmp || tmp > MLX5_CRYPTO_MAX_SEGS) {
+			DRV_LOG(WARNING, "Invalid max_segs_num: %d, should"
+				" be less than %d.",
+				(uint32_t)tmp, MLX5_CRYPTO_MAX_SEGS);
+			rte_errno = EINVAL;
+			return -rte_errno;
+		}
+		devarg_prms->max_segs_num = (uint32_t)tmp;
+	} else if (strcmp(key, "import_kek_id") == 0) {
 		attr->session_import_kek_ptr = (uint32_t)tmp;
-	else if (strcmp(key, "credential_id") == 0)
+	} else if (strcmp(key, "credential_id") == 0) {
 		attr->credential_pointer = (uint32_t)tmp;
-	else if (strcmp(key, "keytag") == 0)
+	} else if (strcmp(key, "keytag") == 0) {
 		devarg_prms->keytag = tmp;
-	else
+	} else {
 		DRV_LOG(WARNING, "Invalid key %s.", key);
+	}
 	return 0;
 }
 
@@ -486,6 +497,7 @@ mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
 	attr->credential_pointer = 0;
 	attr->session_import_kek_ptr = 0;
 	devarg_prms->keytag = 0;
+	devarg_prms->max_segs_num = 8;
 	if (devargs == NULL) {
 		DRV_LOG(ERR,
 	"No login devargs in order to enable crypto operations in the device.");
@@ -626,6 +638,21 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
 	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
+	priv->max_segs_num = devarg_prms.max_segs_num;
+	priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
+			     sizeof(struct mlx5_umr_wqe) +
+			     RTE_ALIGN(priv->max_segs_num, 4) *
+			     sizeof(struct mlx5_wqe_dseg);
+	priv->rdmw_wqe_size = sizeof(struct mlx5_rdma_write_wqe) +
+			      sizeof(struct mlx5_wqe_dseg) *
+			      (priv->max_segs_num <= 2 ? 2 : 2 +
+			       RTE_ALIGN(priv->max_segs_num - 2, 4));
+	priv->wqe_set_size = priv->umr_wqe_size + priv->rdmw_wqe_size;
+	priv->wqe_stride = (priv->umr_wqe_size + priv->rdmw_wqe_size) /
+							       MLX5_SEND_WQE_BB;
+	priv->max_rdmaw_klm_n = (priv->rdmw_wqe_size -
+				 sizeof(struct mlx5_rdma_write_wqe)) /
+				 sizeof(struct mlx5_wqe_dseg);
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 34c65f9a24..81452bd700 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -25,12 +25,18 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
 	uint32_t pdn; /* Protection Domain number. */
+	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
 	uint64_t keytag;
+	uint16_t wqe_set_size;
+	uint16_t umr_wqe_size;
+	uint16_t rdmw_wqe_size;
+	uint16_t wqe_stride;
+	uint16_t max_rdmaw_klm_n;
 };
 
 struct mlx5_crypto_qp {
@@ -54,6 +60,7 @@ struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
 	uint64_t keytag;
+	uint32_t max_segs_num;
 };
 
 int
-- 
2.25.1