From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 82DD842CFF; Tue, 20 Jun 2023 03:25:04 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B4A2842D20; Tue, 20 Jun 2023 03:24:46 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2060.outbound.protection.outlook.com [40.107.223.60]) by mails.dpdk.org (Postfix) with ESMTP id 7655342BFE for ; Tue, 20 Jun 2023 03:24:45 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=j5KZVCPm/eBVBzpvyw4zVfOTg7ZEU/bVgOJWXFw283vcNylb9Omwxl3SR00aB+XgE3Nb/Fxhfiuyt65ZJ/NOB93ijVB2JS/gA/j/aqO6zG/GDjoO0a+nq2yFH0XXiBnJQKVxNmsdOEWXzJ8Nt0hBD2Hf1J6Al1MvZfmsrjshY660ooAgneMiFZ9T0FMl66vE/26PJGm6JqNKOuHkrgiS+Yl+8nddNGrKoZL+1DcSvpkHMFLFDP4A13KA+41i2Cd6NVuFcGwNFpPLFbh6rVcc1BTadlWoabYNn8DfXmVXhg+x/YUbeORi+cbtJ4puvRrGotMNxZSQ2DvxDBwp5fcITw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jyDWUknA+dROiWhtN82hIisQExcikrj/ElHvkVAyyts=; b=Mb3EKATH45a5BYwvQrtp6cgLdGTBp3MkaBr1spZ172mEMOYc3iCnt6DvVHa8LARQ6B6rk2Upk7H/8PtgOnaYxqeeLIMGfhee8dtPq2RdbVvyUDtcBhPK03/Xl2ZTqTecYqEULRSRa3toDdHDJn8DiaOJMUvhdjhFrMxzNP/kOkczVb+5hM69Hl5RFxoBYZQUNgcVhhwqqpr2ZPUkU1i8J4FmzL2nsl40kRQ8WxCiKDWTfTgj5qxpUge10HCT8SwxpCfV5N9/2BgdW+S+7H8znIkSKrXchdSi3edmgSGD7VWcDq+I53mQsazu1a/KpDehAIxqEMCNEOTFqIg6pozKZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jyDWUknA+dROiWhtN82hIisQExcikrj/ElHvkVAyyts=; b=tCIpTPqUc4J+7qMq7cIQqgSo4aaXUDvTImc4AbL5YIv/CZjkbx1UBFIYh865mixQlvYfAAeIlCawZeF8wet+uMAm+0lNsONAh77roP5fOSy+ObW6vlvmmiO0IXRMmFX1W4MSLtcYQnFvyT3xwklx+ude0WCoA/G3AR/4F0YoK8HAXJTw/nC4FQpyS69IWw4O5PBAuJOFCQmyEKu8Dpu7WrmYabW8l3PJy6COtzIC1f20eHDimJY7bYQm5dJoZv+wJ/kPK15gegvP4+lyEFEZWjf6ShXwrrEc+wI0DgDtUzIN3L1yU//ah9frwh0LIjjrESJNF/0xj3sLn7vnBaAbCA== Received: from SA0PR11CA0007.namprd11.prod.outlook.com (2603:10b6:806:d3::12) by CH3PR12MB7548.namprd12.prod.outlook.com (2603:10b6:610:144::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.37; Tue, 20 Jun 2023 01:24:40 +0000 Received: from SN1PEPF000252A3.namprd05.prod.outlook.com (2603:10b6:806:d3:cafe::77) by SA0PR11CA0007.outlook.office365.com (2603:10b6:806:d3::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.37 via Frontend Transport; Tue, 20 Jun 2023 01:24:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SN1PEPF000252A3.mail.protection.outlook.com (10.167.242.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.27 via Frontend Transport; Tue, 20 Jun 2023 01:24:40 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Mon, 19 Jun 2023 18:24:29 -0700 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Mon, 19 Jun 2023 18:24:24 -0700 From: Suanming Mou To: Matan Azrad , Viacheslav Ovsiienko , Ori Kam CC: , , Subject: [PATCH v3 7/9] crypto/mlx5: add queue pair setup for GCM Date: Tue, 20 Jun 2023 04:23:20 +0300 Message-ID: <20230620012322.788099-8-suanmingm@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230620012322.788099-1-suanmingm@nvidia.com> References: <20230418092325.2578712-1-suanmingm@nvidia.com> <20230620012322.788099-1-suanmingm@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF000252A3:EE_|CH3PR12MB7548:EE_ X-MS-Office365-Filtering-Correlation-Id: 1896b50e-f04f-4ad1-3c92-08db712d1ed7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: s1IJl/GaxfBbYIKXxvC8oRiB8VFEk127dRG2Zl9cU9s4ZzAsnH9Ub0J3gEOnJuPNy+HuwzyPUL2Lkc4HLDAON671xJKg5qopcfAjnzi3HlZ+xxsLpNoRytwp0mIq8GNz6fjCoIFJsL7O3cHpm8oKofSQr2l1+Y9cCT740dzfXiSUQwIMX8s9yMlMBIJM6k6crTDtoBJAY8Ir01z9mK9RPsL0S/yHWSoqxs7gWHuqPrFJ6HtbYMQFvOQmO2hgJvh157/BlGRMheJZ6jgIf7UQ5gci2PsPsKci5wWuVonn0T1OT7vXCWfwMDTwZ+fhc1s7c91TKYJ9eSP5XVmrFQVt0MKP1imHr0MN7jy4/z3OGeulVazhr1f9YKHprciq16eSq2m1L3NL2NTe1YAKy8os7DqnNLu8FJ5+TRAohqM53yQASEo1ta1SE3ssSW5MhCzWS98705nFJfXm2n4aW8w+o6rQtKxRLJkfmmcsDnQEXDKELtZcblxOPWV7jzWq8R4/N9RixSdi2jUDy7PKnSEKUXHQH7SoEox9Lr4mGBX/1l+v53qYVxIdI7bk/aRM4aRK/1pr9cwRXh9f3w967VQgHwKM3RBJ+Wg9GJgKOrhlGS3ic6ghSSoyDN5Q7O/Ef3irKsrUuBxHOhOQg4IuWSLiSfKjaP+8kIcpqVO0O7ZgXmzF0Tsvp3EQmtnefy5hjUhYW7MlX8OErYCjkKjEty7dgRlmBQibmHO1QBGTDEEKCcjdQUHWW7HVhJgO1zzR/PEq X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(136003)(39860400002)(376002)(396003)(346002)(451199021)(36840700001)(46966006)(40470700004)(8936002)(41300700001)(40460700003)(8676002)(6666004)(82310400005)(478600001)(82740400003)(7636003)(356005)(2616005)(83380400001)(426003)(336012)(16526019)(26005)(186003)(6286002)(36860700001)(7696005)(47076005)(40480700001)(55016003)(54906003)(110136005)(86362001)(36756003)(316002)(6636002)(4326008)(70586007)(1076003)(70206006)(30864003)(2906002)(5660300002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2023 01:24:40.6540 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1896b50e-f04f-4ad1-3c92-08db712d1ed7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF000252A3.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7548 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Crypto queue pair is for handling the encryption/decryption operations. As AES-GCM AEAD API provides AAD, mbuf, digest separately, low-level FW only accepts the data in a single contiguous memory region, two internal QPs are created for AES-GCM queue pair. One for organizing the memory to be contiguous if they are not. The other is for crypto. If the buffers are checked as implicitly contiguous, the buffer will be sent to the crypto QP directly for encryption/decryption. If not, the buffers will be handled by the first UMR QP. The UMR QP will convert the buffers to be contiguous one. Then the well organized "new" buffer can be handled by crypto QP. The crypto QP is initialized as follower, and UMR as leader. Once crypto operation input buffer requires memory address space converting by UMR QP, the crypto QP processing will be triggered by UMR QP. Otherwise, the ring crypto QP doorbell directly. The existing max_segs_num devarg is used for define how many segments the chained mbuf contains same as AES-XTS before. Signed-off-by: Suanming Mou --- drivers/common/mlx5/mlx5_common_mr.h | 1 + drivers/common/mlx5/mlx5_prm.h | 22 +++ drivers/common/mlx5/version.map | 2 + drivers/crypto/mlx5/mlx5_crypto.h | 15 ++ drivers/crypto/mlx5/mlx5_crypto_gcm.c | 230 ++++++++++++++++++++++++++ 5 files changed, 270 insertions(+) diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h index 66623868a2..8789d403b1 100644 --- a/drivers/common/mlx5/mlx5_common_mr.h +++ b/drivers/common/mlx5/mlx5_common_mr.h @@ -254,6 +254,7 @@ __rte_internal void mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr); +__rte_internal void mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb); diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index 2d0a34ffbc..60dff9dcda 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -470,6 +470,15 @@ struct mlx5_wqe_rseg { #define MLX5_UMRC_KO_OFFSET 16u #define MLX5_UMRC_TO_BS_OFFSET 0u +/* + * As PRM describes, the address of the UMR pointer must be + * aligned to 2KB. + */ +#define MLX5_UMR_KLM_PTR_ALIGN (1 << 11) + +#define MLX5_UMR_KLM_NUM_ALIGN \ + (MLX5_UMR_KLM_PTR_ALIGN / sizeof(struct mlx5_klm)) + struct mlx5_wqe_umr_cseg { uint32_t if_cf_toe_cq_res; uint32_t ko_to_bs; @@ -674,6 +683,19 @@ union mlx5_gga_compress_opaque { uint32_t data[64]; }; +union mlx5_gga_crypto_opaque { + struct { + uint32_t syndrome; + uint32_t reserved0[2]; + struct { + uint32_t iv[3]; + uint32_t tag_size; + uint32_t aad_size; + } cp __rte_packed; + } __rte_packed; + uint8_t data[64]; +}; + struct mlx5_ifc_regexp_mmo_control_bits { uint8_t reserved_at_31[0x2]; uint8_t le[0x1]; diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index f860b069de..0758ba76de 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -159,5 +159,7 @@ INTERNAL { mlx5_os_interrupt_handler_create; # WINDOWS_NO_EXPORT mlx5_os_interrupt_handler_destroy; # WINDOWS_NO_EXPORT + + mlx5_os_set_reg_mr_cb; local: *; }; diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h index 6cb4d4ddec..88a09a6b1c 100644 --- a/drivers/crypto/mlx5/mlx5_crypto.h +++ b/drivers/crypto/mlx5/mlx5_crypto.h @@ -28,8 +28,11 @@ struct mlx5_crypto_priv { TAILQ_ENTRY(mlx5_crypto_priv) next; struct mlx5_common_device *cdev; /* Backend mlx5 device. */ struct rte_cryptodev *crypto_dev; + mlx5_reg_mr_t reg_mr_cb; /* Callback to reg_mr func */ + mlx5_dereg_mr_t dereg_mr_cb; /* Callback to dereg_mr func */ struct mlx5_uar uar; /* User Access Region. */ uint32_t max_segs_num; /* Maximum supported data segs. */ + uint32_t max_klm_num; /* Maximum supported klm. */ struct mlx5_hlist *dek_hlist; /* Dek hash list. */ const struct rte_cryptodev_capabilities *caps; struct rte_cryptodev_config dev_config; @@ -46,15 +49,27 @@ struct mlx5_crypto_qp { struct mlx5_crypto_priv *priv; struct mlx5_devx_cq cq_obj; struct mlx5_devx_qp qp_obj; + struct mlx5_devx_qp umr_qp_obj; struct rte_cryptodev_stats stats; struct rte_crypto_op **ops; struct mlx5_devx_obj **mkey; /* WQE's indirect mekys. */ + struct mlx5_klm *klm_array; + union mlx5_gga_crypto_opaque *opaque_addr; struct mlx5_mr_ctrl mr_ctrl; + struct mlx5_pmd_mr mr; + /* Crypto QP. */ uint8_t *wqe; uint16_t entries_n; + uint16_t cq_entries_n; uint16_t pi; uint16_t ci; uint16_t db_pi; + /* UMR QP. */ + uint8_t *umr_wqe; + uint16_t umr_wqbbs; + uint16_t umr_pi; + uint16_t umr_ci; + uint32_t umr_errors; }; struct mlx5_crypto_dek { diff --git a/drivers/crypto/mlx5/mlx5_crypto_gcm.c b/drivers/crypto/mlx5/mlx5_crypto_gcm.c index 5f55314382..c3859547ee 100644 --- a/drivers/crypto/mlx5/mlx5_crypto_gcm.c +++ b/drivers/crypto/mlx5/mlx5_crypto_gcm.c @@ -18,6 +18,20 @@ #include "mlx5_crypto_utils.h" #include "mlx5_crypto.h" +/* + * AES-GCM uses indirect KLM mode. The UMR WQE comprises of WQE control + + * UMR control + mkey context + indirect KLM. The WQE size is aligned to + * be 3 WQEBBS. + */ +#define MLX5_UMR_GCM_WQE_SIZE \ + (RTE_ALIGN(sizeof(struct mlx5_umr_wqe) + sizeof(struct mlx5_wqe_dseg), \ + MLX5_SEND_WQE_BB)) + +#define MLX5_UMR_GCM_WQE_SET_SIZE \ + (MLX5_UMR_GCM_WQE_SIZE + \ + RTE_ALIGN(sizeof(struct mlx5_wqe_send_en_wqe), \ + MLX5_SEND_WQE_BB)) + static struct rte_cryptodev_capabilities mlx5_crypto_gcm_caps[] = { { .op = RTE_CRYPTO_OP_TYPE_UNDEFINED, @@ -86,6 +100,8 @@ mlx5_crypto_sym_gcm_session_configure(struct rte_cryptodev *dev, sess_private_data->mmo_ctrl = rte_cpu_to_be_32 (op_type << MLX5_CRYPTO_MMO_OP_OFFSET | MLX5_ENCRYPTION_TYPE_AES_GCM << MLX5_CRYPTO_MMO_TYPE_OFFSET); + sess_private_data->wqe_aad_len = rte_cpu_to_be_32((uint32_t)aead->aad_length); + sess_private_data->wqe_tag_len = rte_cpu_to_be_32((uint32_t)aead->digest_length); sess_private_data->aad_len = aead->aad_length; sess_private_data->tag_len = aead->digest_length; sess_private_data->iv_offset = aead->iv.offset; @@ -102,6 +118,216 @@ mlx5_crypto_sym_gcm_session_configure(struct rte_cryptodev *dev, return 0; } +static void * +mlx5_crypto_gcm_mkey_klm_update(struct mlx5_crypto_priv *priv, + struct mlx5_crypto_qp *qp __rte_unused, + uint32_t idx) +{ + return &qp->klm_array[idx * priv->max_klm_num]; +} + +static int +mlx5_crypto_gcm_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) +{ + struct mlx5_crypto_priv *priv = dev->data->dev_private; + struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id]; + + if (qp->umr_qp_obj.qp != NULL) + mlx5_devx_qp_destroy(&qp->umr_qp_obj); + if (qp->qp_obj.qp != NULL) + mlx5_devx_qp_destroy(&qp->qp_obj); + if (qp->cq_obj.cq != NULL) + mlx5_devx_cq_destroy(&qp->cq_obj); + if (qp->mr.obj != NULL) { + void *opaq = qp->mr.addr; + + priv->dereg_mr_cb(&qp->mr); + rte_free(opaq); + } + mlx5_crypto_indirect_mkeys_release(qp, qp->entries_n); + mlx5_mr_btree_free(&qp->mr_ctrl.cache_bh); + rte_free(qp); + dev->data->queue_pairs[qp_id] = NULL; + return 0; +} + +static void +mlx5_crypto_gcm_init_qp(struct mlx5_crypto_qp *qp) +{ + volatile struct mlx5_gga_wqe *restrict wqe = + (volatile struct mlx5_gga_wqe *)qp->qp_obj.wqes; + volatile union mlx5_gga_crypto_opaque *opaq = qp->opaque_addr; + const uint32_t sq_ds = rte_cpu_to_be_32((qp->qp_obj.qp->id << 8) | 4u); + const uint32_t flags = RTE_BE32(MLX5_COMP_ALWAYS << + MLX5_COMP_MODE_OFFSET); + const uint32_t opaq_lkey = rte_cpu_to_be_32(qp->mr.lkey); + int i; + + /* All the next fields state should stay constant. */ + for (i = 0; i < qp->entries_n; ++i, ++wqe) { + wqe->sq_ds = sq_ds; + wqe->flags = flags; + wqe->opaque_lkey = opaq_lkey; + wqe->opaque_vaddr = rte_cpu_to_be_64((uint64_t)(uintptr_t)&opaq[i]); + } +} + +static inline int +mlx5_crypto_gcm_umr_qp_setup(struct rte_cryptodev *dev, struct mlx5_crypto_qp *qp, + int socket_id) +{ + struct mlx5_crypto_priv *priv = dev->data->dev_private; + struct mlx5_devx_qp_attr attr = {0}; + uint32_t ret; + uint32_t log_wqbb_n; + + /* Set UMR + SEND_EN WQE as maximum same with crypto. */ + log_wqbb_n = rte_log2_u32(qp->entries_n * + (MLX5_UMR_GCM_WQE_SET_SIZE / MLX5_SEND_WQE_BB)); + attr.pd = priv->cdev->pdn; + attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar.obj); + attr.cqn = qp->cq_obj.cq->id; + attr.num_of_receive_wqes = 0; + attr.num_of_send_wqbbs = RTE_BIT32(log_wqbb_n); + attr.ts_format = + mlx5_ts_format_conv(priv->cdev->config.hca_attr.qp_ts_format); + attr.cd_master = 1; + ret = mlx5_devx_qp_create(priv->cdev->ctx, &qp->umr_qp_obj, + attr.num_of_send_wqbbs * MLX5_SEND_WQE_BB, + &attr, socket_id); + if (ret) { + DRV_LOG(ERR, "Failed to create UMR QP."); + return -1; + } + if (mlx5_devx_qp2rts(&qp->umr_qp_obj, qp->umr_qp_obj.qp->id)) { + DRV_LOG(ERR, "Failed to change UMR QP state to RTS."); + return -1; + } + /* Save the UMR WQEBBS for checking the WQE boundary. */ + qp->umr_wqbbs = attr.num_of_send_wqbbs; + return 0; +} + +static int +mlx5_crypto_gcm_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, + const struct rte_cryptodev_qp_conf *qp_conf, + int socket_id) +{ + struct mlx5_crypto_priv *priv = dev->data->dev_private; + struct mlx5_hca_attr *attr = &priv->cdev->config.hca_attr; + struct mlx5_crypto_qp *qp; + struct mlx5_devx_cq_attr cq_attr = { + .uar_page_id = mlx5_os_get_devx_uar_page_id(priv->uar.obj), + }; + struct mlx5_devx_qp_attr qp_attr = { + .pd = priv->cdev->pdn, + .uar_index = mlx5_os_get_devx_uar_page_id(priv->uar.obj), + .user_index = qp_id, + }; + struct mlx5_devx_mkey_attr mkey_attr = { + .pd = priv->cdev->pdn, + .umr_en = 1, + .klm_num = priv->max_klm_num, + }; + uint32_t log_ops_n = rte_log2_u32(qp_conf->nb_descriptors); + uint32_t entries = RTE_BIT32(log_ops_n); + uint32_t alloc_size = sizeof(*qp); + size_t mr_size, opaq_size; + void *mr_buf; + int ret; + + alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE); + alloc_size += (sizeof(struct rte_crypto_op *) + + sizeof(struct mlx5_devx_obj *)) * entries; + qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE, + socket_id); + if (qp == NULL) { + DRV_LOG(ERR, "Failed to allocate qp memory."); + rte_errno = ENOMEM; + return -rte_errno; + } + qp->priv = priv; + qp->entries_n = entries; + if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->cdev->mr_scache.dev_gen, + priv->dev_config.socket_id)) { + DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.", + (uint32_t)qp_id); + rte_errno = ENOMEM; + goto err; + } + /* + * The following KLM pointer must be aligned with + * MLX5_UMR_KLM_PTR_ALIGN. Aligned opaq_size here + * to make the KLM pointer with offset be aligned. + */ + opaq_size = RTE_ALIGN(sizeof(union mlx5_gga_crypto_opaque) * entries, + MLX5_UMR_KLM_PTR_ALIGN); + mr_size = (priv->max_klm_num * sizeof(struct mlx5_klm) * entries) + opaq_size; + mr_buf = rte_calloc(__func__, (size_t)1, mr_size, MLX5_UMR_KLM_PTR_ALIGN); + if (mr_buf == NULL) { + DRV_LOG(ERR, "Failed to allocate mr memory."); + rte_errno = ENOMEM; + goto err; + } + if (priv->reg_mr_cb(priv->cdev->pd, mr_buf, mr_size, &qp->mr) != 0) { + rte_free(mr_buf); + DRV_LOG(ERR, "Failed to register opaque MR."); + rte_errno = ENOMEM; + goto err; + } + qp->opaque_addr = qp->mr.addr; + qp->klm_array = RTE_PTR_ADD(qp->opaque_addr, opaq_size); + /* + * Triple the CQ size as UMR QP which contains UMR and SEND_EN WQE + * will share this CQ . + */ + qp->cq_entries_n = rte_align32pow2(entries * 3); + ret = mlx5_devx_cq_create(priv->cdev->ctx, &qp->cq_obj, + rte_log2_u32(qp->cq_entries_n), + &cq_attr, socket_id); + if (ret != 0) { + DRV_LOG(ERR, "Failed to create CQ."); + goto err; + } + qp_attr.cqn = qp->cq_obj.cq->id; + qp_attr.ts_format = mlx5_ts_format_conv(attr->qp_ts_format); + qp_attr.num_of_receive_wqes = 0; + qp_attr.num_of_send_wqbbs = entries; + qp_attr.mmo = attr->crypto_mmo.crypto_mmo_qp; + /* Set MMO QP as follower as the input data may depend on UMR. */ + qp_attr.cd_slave_send = 1; + ret = mlx5_devx_qp_create(priv->cdev->ctx, &qp->qp_obj, + qp_attr.num_of_send_wqbbs * MLX5_WQE_SIZE, + &qp_attr, socket_id); + if (ret != 0) { + DRV_LOG(ERR, "Failed to create QP."); + goto err; + } + mlx5_crypto_gcm_init_qp(qp); + ret = mlx5_devx_qp2rts(&qp->qp_obj, 0); + if (ret) + goto err; + qp->ops = (struct rte_crypto_op **)(qp + 1); + qp->mkey = (struct mlx5_devx_obj **)(qp->ops + entries); + if (mlx5_crypto_gcm_umr_qp_setup(dev, qp, socket_id)) { + DRV_LOG(ERR, "Failed to setup UMR QP."); + goto err; + } + DRV_LOG(INFO, "QP %u: SQN=0x%X CQN=0x%X entries num = %u", + (uint32_t)qp_id, qp->qp_obj.qp->id, qp->cq_obj.cq->id, entries); + if (mlx5_crypto_indirect_mkeys_prepare(priv, qp, &mkey_attr, + mlx5_crypto_gcm_mkey_klm_update)) { + DRV_LOG(ERR, "Cannot allocate indirect memory regions."); + rte_errno = ENOMEM; + goto err; + } + dev->data->queue_pairs[qp_id] = qp; + return 0; +err: + mlx5_crypto_gcm_qp_release(dev, qp_id); + return -1; +} + int mlx5_crypto_gcm_init(struct mlx5_crypto_priv *priv) { @@ -110,6 +336,10 @@ mlx5_crypto_gcm_init(struct mlx5_crypto_priv *priv) /* Override AES-GCM specified ops. */ dev_ops->sym_session_configure = mlx5_crypto_sym_gcm_session_configure; + mlx5_os_set_reg_mr_cb(&priv->reg_mr_cb, &priv->dereg_mr_cb); + dev_ops->queue_pair_setup = mlx5_crypto_gcm_qp_setup; + dev_ops->queue_pair_release = mlx5_crypto_gcm_qp_release; + priv->max_klm_num = RTE_ALIGN((priv->max_segs_num + 1) * 2 + 1, MLX5_UMR_KLM_NUM_ALIGN); priv->caps = mlx5_crypto_gcm_caps; return 0; } -- 2.25.1