From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AB64F454DE; Mon, 24 Jun 2024 11:17:48 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0898740B8D; Mon, 24 Jun 2024 11:17:39 +0200 (CEST) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2046.outbound.protection.outlook.com [40.107.243.46]) by mails.dpdk.org (Postfix) with ESMTP id BE73B406BA for ; Mon, 24 Jun 2024 11:17:36 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=klnAR0dN6rSv4VgqdLzGe5jINPYz9KjwphFLq+Tgc6Qe9cXa+h9XlsiHQvRE43in6SrHPs/HU9LFpwFNyZKA195PCaW0g1dqPbETkAE0COWz9jHk4jq0D5ZX4P4Uu0sdZH90ClBnR3Lpvd1S85wJCz13J9bAEn/aHyZxZU1kIGduFUbvaVouGBvwOIfrgMDiMBYNNGmAqkN67Gjd6Iv2z0gIKwgtCGeipMEBMWjkfDKNhIVwF81pG09qQCUJRyfmv5WB5u4vph+cd+15xcgBCFmp32sRJqC+Fre2Sn7j24IDd5ww66aBMe45QktCsszHFMVitNunIi0nrqHZ1f1FLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KALD01J/Ve7WearllKqLslnJJQHVjIzvv4yd3S4yeMo=; b=RvXpd7dFP9/9rNM1tNa9nTduYkgs76VCTCYdHo75r4zKuzIQ4hH5rdS52z0cyWlrzKaObrVNHoHXD++ptfCD8N5dyJRH6VQVfXY3DKXtXfqNhl2AX+5U/Kq9uxtfyjWM+zdDq6Rr0XBVBuc7wzzk/UI9vPRMq9o7SwqL7jhn+WsUVPJ3egB7BbtzRuQOgRMZmF5Zw/qxZK6Y2K5Z9g7fQW5mehVXf/z/Ag+t3MwO0Ezd9TXWaqM59a758mFkLENAZSPu/4wjSq5KFxfS2UNYh+8vhMkvJ6NQF0oNRcfzJ+fLrPlgXNqCUj0x1dm6N9vEMq3Z1pGaAEfVDn4SdEO9LA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KALD01J/Ve7WearllKqLslnJJQHVjIzvv4yd3S4yeMo=; b=Rj+8MCihm3LOcRICxd6k1z3wwRM2hMusXVpX6OVvOviHqET5IQOicAsjpuI023bIp8m4pFjdndoIvfVcIpTmg48Uj3rEquZlV/fYv7SM1+n/YKNJbV2H2cRuDl6fOR+aZx7Fyrs3DIz1vLuhtya+EoWQpQGWGkJVunC2UmOChGl0+Bfu35Wq5+zH65/POh2IgKVs+5HpGglk0XS6USkHUN/tGFEK89k3F9jvaunleVaOIYY2eiDqe3wHZOylxnxvna3ZCTc6V9o9LLm729IfS3qstBnoGuM58yjCbwlPvpq7ySXA/8H505VNb9irJ5yiR16dFURJQj+qmHUbO1VTGw== Received: from CH2PR05CA0004.namprd05.prod.outlook.com (2603:10b6:610::17) by DS7PR12MB6309.namprd12.prod.outlook.com (2603:10b6:8:96::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7698.28; Mon, 24 Jun 2024 09:17:34 +0000 Received: from CH2PEPF00000142.namprd02.prod.outlook.com (2603:10b6:610:0:cafe::10) by CH2PR05CA0004.outlook.office365.com (2603:10b6:610::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7719.19 via Frontend Transport; Mon, 24 Jun 2024 09:17:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CH2PEPF00000142.mail.protection.outlook.com (10.167.244.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.15 via Frontend Transport; Mon, 24 Jun 2024 09:17:33 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 24 Jun 2024 02:17:17 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 24 Jun 2024 02:17:16 -0700 From: Suanming Mou To: Matan Azrad CC: , Subject: [PATCH v3 2/2] crypto/mlx5: add out of place mode for IPsec operation Date: Mon, 24 Jun 2024 17:16:51 +0800 Message-ID: <20240624091651.2295533-3-suanmingm@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240624091651.2295533-1-suanmingm@nvidia.com> References: <20240530072413.1602343-1-suanmingm@nvidia.com> <20240624091651.2295533-1-suanmingm@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000142:EE_|DS7PR12MB6309:EE_ X-MS-Office365-Filtering-Correlation-Id: ad1bda7f-b08d-44b8-60fe-08dc942e7b49 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230037|36860700010|376011|1800799021|82310400023; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?AVWlXdPEnfZrm0D8nYzEacMTtvx5mVuX+KzVoU5MBEUGyvzlvcDIcIJNy9lQ?= =?us-ascii?Q?A/4LsLrKdCzE42o4sR4vhTKm2X7reTfh3j4rJj4bcWGtLeHpNqF+/Onh0GbH?= =?us-ascii?Q?yvUHkWpYwlvD88d+QfGJ3PTt7CuWPQeL8a3Y+aF5GxYMygbJioHGTEgV3bPz?= =?us-ascii?Q?22QQZhOSQ9kguXCKot4/8YpiC6WC6kcMT8LTXOLbuyB+xxkGSGHN//tG1ybg?= =?us-ascii?Q?ize9RGneK211t4CJRdp4WOO/XanANutHdx/hMKzgyIKTVCNucHAWNlSuDK+g?= =?us-ascii?Q?z5uliZy06s/K/yxY6jwW0EVTqMd0DeOcDrdRZKUh8h4Vtu9NXyooQALp60wX?= =?us-ascii?Q?bf78x8ewJX02x+90kZbee3gu6TwdNq5vd0oPriaS2ABYs8U0+j6C25vWFyPs?= =?us-ascii?Q?5U/Z4beLAB6wClhFm0GvM/qypLlZ//WijLKTrCoDOjLUYyGmFjaOkRjfvXQE?= =?us-ascii?Q?OLZx8k2jLnemhKsLAXzTMUsyS+drSt5Y1OoU76FAzG3pei6QZTdyhdUg1qpR?= =?us-ascii?Q?VQHJdiYv7mKKNHOweXdqIxFlV8/2cFgYJ02OKce4ClQgToDXl5IwinORiotP?= =?us-ascii?Q?JAMRDk+cfFxc1JNJMq8ngVV0IgqFbT8/okqV8T5at9v5mJhZ+UVVBhFb8Hh4?= =?us-ascii?Q?4MLo+S2yK+QzyKtWhqK8GTZstnYu7FtQkQm1StxeukeMoqlDfB7gQ3In4SY3?= =?us-ascii?Q?t4jESO6tTeMFifPy97PunHhoUvHaL6zeHbtPsJXJbq4lUzZxYHORxXf/qdSV?= =?us-ascii?Q?V6iKvx1PbD97wLCsdgCI6A9eTHELG6P2e6BSlZOA8npfVJ/koZqR4mr/vace?= =?us-ascii?Q?ow46ZiL1iXYkHpWPYgFSq22Og/L5cZ9bUYRxyVHq00TMyDbI2qPEzzLIRK+d?= =?us-ascii?Q?c36OP5yEcBq+jhP8luyJlaXGC1eYkGpw2lJmMP6gK/SBHEHIptIT+xB6TKyt?= =?us-ascii?Q?pna9/pCxikQnj0ulrmOs1+LOhRLbHDXq/JK9LH5TcsuyJOAXVOpPBw5eyq5D?= =?us-ascii?Q?Hwy080ACZLn3xWP3aFubfWg3JoRosdh7OSphNj6qSBqon0/ZBFOS+fhVj28n?= =?us-ascii?Q?UgyU+R7R64X3XAWYmrCQGjm2nLHyODvhofw0d3jKJFtp9aKcjfRlbKao6vpb?= =?us-ascii?Q?Y5x88YkGJ6qI4l0z1PbqWHqJ1ZKXGuW27hoSpyctkSkY+dHI7l2HNttn0brX?= =?us-ascii?Q?l0SIAnfQvlS9y/uPPHTVA5h2He1IzjtPqy9JD/qHFdgfeGB9EzFg1Dr1Fa+y?= =?us-ascii?Q?+toFdhlcXIedIJDeytuT4NszcoEbj/aWx0Vh+9koZQeLb1s3D+GgGC6TwZ1d?= =?us-ascii?Q?rAKfikg2trYfPaUKkSWRBy7xEmygkVrSlzjg+9K8to6d5XSJdle6yPXlTeEF?= =?us-ascii?Q?DoIrCtrifilCdDvvxflLwRziGLDPnqe1bPpVtn4GtVX81ZolHw=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230037)(36860700010)(376011)(1800799021)(82310400023); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2024 09:17:33.5862 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ad1bda7f-b08d-44b8-60fe-08dc942e7b49 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000142.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6309 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The IPsec operation shrinks AAD directly before payload in enqueue burst and restores the memory in dequeue burst. This commit adds the support of OOP mode follows the similar strategy. Signed-off-by: Suanming Mou Acked-by: Matan Azrad --- doc/guides/cryptodevs/mlx5.rst | 3 ++ drivers/crypto/mlx5/mlx5_crypto.c | 2 +- drivers/crypto/mlx5/mlx5_crypto_gcm.c | 43 +++++++++++++++++++++------ 3 files changed, 38 insertions(+), 10 deletions(-) diff --git a/doc/guides/cryptodevs/mlx5.rst b/doc/guides/cryptodevs/mlx5.rst index fd0aa1ed8b..0568852571 100644 --- a/doc/guides/cryptodevs/mlx5.rst +++ b/doc/guides/cryptodevs/mlx5.rst @@ -201,6 +201,9 @@ for an additional list of options shared with other mlx5 drivers. AAD (ESP SPI and SN) to the payload during enqueue OP. It then restores the original memory layout in the decrypt OP. ESP.IV size supported range is [0,16] bytes. + For OOP case, PMD will replace the bytes preceding the OP destination + address to match the information found between the AAD pointer and the + OP source address. User should prepare this headroom in this case. Set to ``full_capable`` by default. diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c index d49a375dcb..bf9cbd4a6a 100644 --- a/drivers/crypto/mlx5/mlx5_crypto.c +++ b/drivers/crypto/mlx5/mlx5_crypto.c @@ -25,6 +25,7 @@ #define MLX5_CRYPTO_FEATURE_FLAGS(wrapped_mode) \ (RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \ + RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | \ (wrapped_mode ? RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY : 0) | \ RTE_CRYPTODEV_FF_CIPHER_MULTIPLE_DATA_UNITS) @@ -61,7 +62,6 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev, RTE_CRYPTODEV_FF_IN_PLACE_SGL | RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT | RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | - RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT; dev_info->capabilities = priv->caps; diff --git a/drivers/crypto/mlx5/mlx5_crypto_gcm.c b/drivers/crypto/mlx5/mlx5_crypto_gcm.c index 189e798d1d..f598273873 100644 --- a/drivers/crypto/mlx5/mlx5_crypto_gcm.c +++ b/drivers/crypto/mlx5/mlx5_crypto_gcm.c @@ -1000,6 +1000,7 @@ mlx5_crypto_gcm_ipsec_enqueue_burst(void *queue_pair, struct mlx5_crypto_gcm_data gcm_data; struct rte_crypto_op *op; struct rte_mbuf *m_src; + struct rte_mbuf *m_dst; uint16_t mask = qp->entries_n - 1; uint16_t remain = qp->entries_n - (qp->pi - qp->qp_ci); uint32_t idx; @@ -1029,19 +1030,32 @@ mlx5_crypto_gcm_ipsec_enqueue_burst(void *queue_pair, MLX5_ASSERT(pkt_iv_len <= MLX5_CRYPTO_GCM_IPSEC_IV_SIZE); gcm_data.src_bytes = op->sym->aead.data.length + sess->aad_len; gcm_data.src_mkey = mlx5_mr_mb2mr(&qp->mr_ctrl, op->sym->m_src); - /* OOP mode is not supported. */ - MLX5_ASSERT(!op->sym->m_dst || op->sym->m_dst == m_src); - gcm_data.dst_addr = gcm_data.src_addr; - gcm_data.dst_mkey = gcm_data.src_mkey; + m_dst = op->sym->m_dst; + if (m_dst && m_dst != m_src) { + MLX5_ASSERT(m_dst->nb_segs == 1 && + (rte_pktmbuf_headroom(m_dst) + op->sym->aead.data.offset) + >= sess->aad_len + pkt_iv_len); + gcm_data.dst_addr = RTE_PTR_SUB + (rte_pktmbuf_mtod_offset(m_dst, + void *, op->sym->aead.data.offset), sess->aad_len); + gcm_data.dst_mkey = mlx5_mr_mb2mr(&qp->mr_ctrl, m_dst); + } else { + gcm_data.dst_addr = gcm_data.src_addr; + gcm_data.dst_mkey = gcm_data.src_mkey; + } gcm_data.dst_bytes = gcm_data.src_bytes; /* Digest should follow payload. */ - MLX5_ASSERT(RTE_PTR_ADD - (gcm_data.src_addr, sess->aad_len + op->sym->aead.data.length) == - op->sym->aead.digest.data); - if (sess->op_type == MLX5_CRYPTO_OP_TYPE_ENCRYPTION) + if (sess->op_type == MLX5_CRYPTO_OP_TYPE_ENCRYPTION) { + MLX5_ASSERT(RTE_PTR_ADD(gcm_data.dst_addr, + sess->aad_len + op->sym->aead.data.length) == + op->sym->aead.digest.data); gcm_data.dst_bytes += sess->tag_len; - else + } else { + MLX5_ASSERT(RTE_PTR_ADD(gcm_data.src_addr, + sess->aad_len + op->sym->aead.data.length) == + op->sym->aead.digest.data); gcm_data.src_bytes += sess->tag_len; + } mlx5_crypto_gcm_wqe_set(qp, op, idx, &gcm_data); /* * All the data such as IV have been copied above, @@ -1080,6 +1094,7 @@ mlx5_crypto_gcm_restore_ipsec_mem(struct mlx5_crypto_qp *qp, struct mlx5_crypto_session *sess; struct rte_crypto_op *op; struct rte_mbuf *m_src; + struct rte_mbuf *m_dst; uint8_t *payload; while (orci != rci) { @@ -1095,6 +1110,16 @@ mlx5_crypto_gcm_restore_ipsec_mem(struct mlx5_crypto_qp *qp, RTE_PTR_SUB(payload, sess->aad_len), sess->aad_len); rte_memcpy(RTE_PTR_SUB(payload, MLX5_CRYPTO_GCM_IPSEC_IV_SIZE), &qp->ipsec_mem[idx], MLX5_CRYPTO_GCM_IPSEC_IV_SIZE); + m_dst = op->sym->m_dst; + if (m_dst && m_dst != m_src) { + uint32_t bytes_to_copy; + + bytes_to_copy = RTE_PTR_DIFF(payload, op->sym->aead.aad.data); + rte_memcpy(RTE_PTR_SUB(rte_pktmbuf_mtod_offset(m_dst, void *, + op->sym->aead.data.offset), bytes_to_copy), + op->sym->aead.aad.data, + bytes_to_copy); + } orci++; } } -- 2.34.1