From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BAD1B457A1 for ; Mon, 12 Aug 2024 14:53:44 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B4F5140668; Mon, 12 Aug 2024 14:53:44 +0200 (CEST) Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2042.outbound.protection.outlook.com [40.107.100.42]) by mails.dpdk.org (Postfix) with ESMTP id EA5254029C for ; Mon, 12 Aug 2024 14:53:42 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ratTQUra6foACIdeH2hNKKik9rIUc8CGogecOzm0mSLlD39ifq3UDQnkxIcX3z4i4w+JPX0b4M5O+YONKRi7R0FAekSoCal8LM7RJhE1Bd8E5SuUJrxhUu0p6P/oLsUu2tzoR8HR+CJXdKJ3ku/nN3DGhk61wHWFuPQcToZ6erzwZqFZZWRW8IB3lYM/DuCuMn6ioXje2Wx/nKJhHWyM9+5QDdNHCFYVpHgFNJICaVqhuY57d6f7udH8xcOmMgCcuIMIsrGq9/lQ6B3qlJ5d1/B2MTO32XIkuRJNTMgJOJyQ93EUiEW27/lLfY63SbDmLVNDVf4z+Jl0GRgidpDtWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2G18xf7koYtaZTnnDWL238vxiJoRjI+Sqh1WdRxt2b4=; b=R/BOu6lJhKNgWEo5SmE7tpPepAOsvrAW3+Nh5CVyegtErGU8zAaEX2OLa0wh33y5FwjISqqg/L8zbqA8/rQsapiNStRmvswStuG3sbrqrv5I0rfmqMtFnKOMCwLq1qaOSZFH8cs731uf1q3ko8LWvMnffR2N7PPj/iQANBYRH9PT+hHNtpLd9tYZhYJsloegdBGtoAcM24PZk8BBT19IpLMHd72fUPGBh1yV9ecuZuv4xBghF85fGwCP+cVYQolXcp7d2lyRwsYHNBi49dNMWzs+olELqTPwML1YcECAcGnOBNLTO983UsJiJOoSOlkKR+PS1TmEwnZ00VEbBeoYLA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=foss.arm.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2G18xf7koYtaZTnnDWL238vxiJoRjI+Sqh1WdRxt2b4=; b=pn8OqsKqTBc2+RdF7dlu2WCZVYpO10GAg9ciuXNOZbRKJbrf87OmCjdj+fv7x/Fhy0Ex3YoTLZ1UX7KAm4h2IF+AS2ttwJx2gw6T5l11mne7Bzu/GthkplguN8u7RcmtCMKQIBG4LDNbLHxAKRaefgqqQLIK79SUWJNwMzQfoL3EfpcAVZKQ1OZ9XzNXNRLgapMJw1bJK3Zqp6h+PiXuSrzuDXqDdNd5fLORNXBreeZE5nPtFuBMWkIY8WalISdxcOA9zxNf+KG1ouR1f1iNz7JFJpJY7/eecVZwvRsfptkRbIPB7krOgBJyVNEoRnPLTH219ZcQWFcfKn8KlLxyuQ== Received: from DS7PR03CA0346.namprd03.prod.outlook.com (2603:10b6:8:55::9) by DM6PR12MB4330.namprd12.prod.outlook.com (2603:10b6:5:21d::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7849.20; Mon, 12 Aug 2024 12:53:39 +0000 Received: from DS1PEPF0001708F.namprd03.prod.outlook.com (2603:10b6:8:55:cafe::66) by DS7PR03CA0346.outlook.office365.com (2603:10b6:8:55::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7828.33 via Frontend Transport; Mon, 12 Aug 2024 12:53:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DS1PEPF0001708F.mail.protection.outlook.com (10.167.17.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7828.19 via Frontend Transport; Mon, 12 Aug 2024 12:53:39 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 12 Aug 2024 05:53:25 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 12 Aug 2024 05:53:23 -0700 From: Xueming Li To: Jack Bond-Preston CC: , Kai Ji , Wathsala Vithanage , dpdk stable Subject: patch 'crypto/openssl: make per-QP cipher context clones' has been queued to stable release 23.11.2 Date: Mon, 12 Aug 2024 20:48:24 +0800 Message-ID: <20240812125035.389667-28-xuemingl@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240812125035.389667-1-xuemingl@nvidia.com> References: <20240712110153.309690-23-xuemingl@nvidia.com> <20240812125035.389667-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS1PEPF0001708F:EE_|DM6PR12MB4330:EE_ X-MS-Office365-Filtering-Correlation-Id: a7b99d00-8087-4f00-14c1-08dcbacdc994 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|82310400026|36860700013|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?wDqWldTXllfHxSHHVwYqhHg53F+69Tp8eW1t2jLrE1yYCbIHenxy5ubp3Jeq?= =?us-ascii?Q?gMX7kY4y8WMI/bQ9XFFSAwfs0/p96wUKlJO78FZ6tJQtxUi6vsnFF8viCQd+?= =?us-ascii?Q?9Ww1W1urIircPK59ByM+y0v+ayXn3RrBPKAJf9CpynGpnCxrhHVDyIigdIcx?= =?us-ascii?Q?FkMLEg4N/B2h02OsjCva+c4sDMXiE3zvEUHSS0WmDv2Bzgcp7naS9U6R+Pi9?= =?us-ascii?Q?z3q+KPPPm85SZO06eYMiyDVgCN39yPnGiOg6uxY8NQAtgJ+AG3Ezaet1ZndV?= =?us-ascii?Q?fBdqlZ74sUnpv/IYmbRWqtXTVRaPQyFaBghLVaEAjueWLBneJV+95Z0Xvu1W?= =?us-ascii?Q?WaIAvX7qMEnwhRm+DFNIMI33gsSRFR9LGbW1Ih0HY04kOindlAZVuzc39u2X?= =?us-ascii?Q?ZInmFatxD/DcFGIUBHHl155Msks5oODEWiY9AxOEhord7R5UqA3pnt/FNk/9?= =?us-ascii?Q?VFMJLY4w8wiGv2v3DBmnhz25viTJoTxiN6rmnmq0VWpSiMp8AQfeBKGV/rP/?= =?us-ascii?Q?uZ3avIDZl0JgW/MaGn5MqWjU84XhC2NyTpaQv7aUR4O8OKN+qnVYNwm6YGlO?= =?us-ascii?Q?+vDqwPanLxvbpeKLvepzojWkQ1HZjCPTuzRo7KuPFhh+JIrZ4XBcshWMh0Cw?= =?us-ascii?Q?8cbw2eZH27d0u89f4HXX67yraI9l/yi6nMYGtRtpPkMZ3Ip90GQnA7NVxEue?= =?us-ascii?Q?+BCSdWY13O1RZJ1tLxBe2Xb8syh2My8SpbY4k0Sg/KTMdy2dHUHgV1isHKcf?= =?us-ascii?Q?QIGzOQYOpESvCrtiyVhN/SLygQNruHqBlQCLOO1KhwytSorAt7m84ItSNP1l?= =?us-ascii?Q?KjKxm0rPwFuMnglMG4VozdGFGnImVDLFEIPrh1xpkEn2/4AT4et6I7klCjJj?= =?us-ascii?Q?zs+Sm/T7CRDlDR1T8Xwo8R0F2pkP5asvbIi38DQAaA+U+IwlCWOwKubg6KoE?= =?us-ascii?Q?CLroeiW0StbbIJId0GKKkdsDbEu6PPgt7iLAykVMe3LDrzYJN2ymq1Ewco7E?= =?us-ascii?Q?9HtYB4VH98Zj0VnE/PpsNnNZRyMWZf3vWBDU5dl+CH0z8YwO3zJHW2Wc39OA?= =?us-ascii?Q?lwpmAEt9Wbzavg646UTyyP+wMkvWxVOq/0AEoTQDfloAzTiyTV8/LaxENktn?= =?us-ascii?Q?7uPtwh8zzlDjvD3DoWl8YXmfVBZ0QQB4B7pH/FUqKm7ykgRklffLPBKod5Ms?= =?us-ascii?Q?VSmKtVNT0qQ4pjvCSzOnnt6BH6qibgqF7AS00sRCx+bnkg9Z32hMNpd3gN8M?= =?us-ascii?Q?WYDpfKU/I8ViTQq2MY6iAuhOnU7M6Id7ceeTT457n8+s4qOKfYZc+VfjUWc5?= =?us-ascii?Q?RL2oQnHqQinpWgHsh9gbrGoqTPkZwNqsQwbYG8Go+iML5MTpyocAX/5I95yl?= =?us-ascii?Q?IYCIVFON5DJmL/IkSH3BuH7ooyo29+SC9nPfeO308tGz2fRu6VN7uxhbBH+2?= =?us-ascii?Q?yccIW1PPaIIemk+KhJVCbosI9QLg1gMZ?= X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(1800799024)(82310400026)(36860700013)(376014); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Aug 2024 12:53:39.1192 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a7b99d00-8087-4f00-14c1-08dcbacdc994 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS1PEPF0001708F.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4330 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 23.11.2 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 08/14/24. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://git.dpdk.org/dpdk-stable/log/?h=23.11-staging This queued commit can be viewed at: https://git.dpdk.org/dpdk-stable/commit/?h=23.11-staging&id=4f8c97e94187b19adde37b6b01d42441e10f91da Thanks. Xueming Li --- >From 4f8c97e94187b19adde37b6b01d42441e10f91da Mon Sep 17 00:00:00 2001 From: Jack Bond-Preston Date: Wed, 3 Jul 2024 13:45:49 +0000 Subject: [PATCH] crypto/openssl: make per-QP cipher context clones Cc: Xueming Li [ upstream commit b1d71126023521fe740ec473abfe5b295035b859 ] Currently EVP_CIPHER_CTXs are allocated, copied to (from openssl_session), and then freed for every cipher operation (ie. per packet). This is very inefficient, and avoidable. Make each openssl_session hold an array of pointers to per-queue-pair cipher context copies. These are populated on first use by allocating a new context and copying from the main context. These copies can then be used in a thread-safe manner by different worker lcores simultaneously. Consequently the cipher context allocation and copy only has to happen once - the first time a given qp uses an openssl_session. This brings about a large performance boost. Throughput performance uplift measurements for AES-CBC-128 encrypt on Ampere Altra Max platform: 1 worker lcore | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 1.51 | 2.94 | 94.4% | | 256 | 4.90 | 8.05 | 64.3% | | 1024 | 11.07 | 14.21 | 28.3% | | 2048 | 14.03 | 16.28 | 16.0% | | 4096 | 16.20 | 17.59 | 8.6% | 8 worker lcores | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 3.05 | 23.74 | 678.8% | | 256 | 10.46 | 64.86 | 520.3% | | 1024 | 40.97 | 113.80 | 177.7% | | 2048 | 73.25 | 130.21 | 77.8% | | 4096 | 103.89 | 140.62 | 35.4% | Signed-off-by: Jack Bond-Preston Acked-by: Kai Ji Reviewed-by: Wathsala Vithanage --- drivers/crypto/openssl/openssl_pmd_private.h | 11 +- drivers/crypto/openssl/rte_openssl_pmd.c | 105 ++++++++++++------- drivers/crypto/openssl/rte_openssl_pmd_ops.c | 34 +++++- 3 files changed, 108 insertions(+), 42 deletions(-) diff --git a/drivers/crypto/openssl/openssl_pmd_private.h b/drivers/crypto/openssl/openssl_pmd_private.h index 334912d335..370de1d53b 100644 --- a/drivers/crypto/openssl/openssl_pmd_private.h +++ b/drivers/crypto/openssl/openssl_pmd_private.h @@ -166,6 +166,14 @@ struct openssl_session { /**< digest length */ } auth; + uint16_t ctx_copies_len; + /* < number of entries in ctx_copies */ + EVP_CIPHER_CTX *qp_ctx[]; + /**< Flexible array member of per-queue-pair pointers to copies of EVP + * context structure. Cipher contexts are not safe to use from multiple + * cores simultaneously, so maintaining these copies allows avoiding + * per-buffer copying into a temporary context. + */ } __rte_cache_aligned; /** OPENSSL crypto private asymmetric session structure */ @@ -217,7 +225,8 @@ struct openssl_asym_session { /** Set and validate OPENSSL crypto session parameters */ extern int openssl_set_session_parameters(struct openssl_session *sess, - const struct rte_crypto_sym_xform *xform); + const struct rte_crypto_sym_xform *xform, + uint16_t nb_queue_pairs); /** Reset OPENSSL crypto session parameters */ extern void diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c index bd09d58d88..df44cc097e 100644 --- a/drivers/crypto/openssl/rte_openssl_pmd.c +++ b/drivers/crypto/openssl/rte_openssl_pmd.c @@ -467,13 +467,10 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess, return 0; } +#if (OPENSSL_VERSION_NUMBER >= 0x30000000L && OPENSSL_VERSION_NUMBER < 0x30200000L) static int openssl_aesni_ctx_clone(EVP_CIPHER_CTX **dest, struct openssl_session *sess) { -#if (OPENSSL_VERSION_NUMBER >= 0x30200000L) - *dest = EVP_CIPHER_CTX_dup(sess->ctx); - return 0; -#elif (OPENSSL_VERSION_NUMBER >= 0x30000000L) /* OpenSSL versions 3.0.0 <= V < 3.2.0 have no dupctx() implementation * for AES-GCM and AES-CCM. In this case, we have to create new empty * contexts and initialise, as we did the original context. @@ -489,13 +486,8 @@ static int openssl_aesni_ctx_clone(EVP_CIPHER_CTX **dest, return openssl_set_sess_aead_dec_param(sess, sess->aead_algo, sess->auth.digest_length, sess->cipher.key.data, dest); -#else - *dest = EVP_CIPHER_CTX_new(); - if (EVP_CIPHER_CTX_copy(*dest, sess->cipher.ctx) != 1) - return -EINVAL; - return 0; -#endif } +#endif /** Set session cipher parameters */ static int @@ -824,7 +816,8 @@ openssl_set_session_aead_parameters(struct openssl_session *sess, /** Parse crypto xform chain and set private session parameters */ int openssl_set_session_parameters(struct openssl_session *sess, - const struct rte_crypto_sym_xform *xform) + const struct rte_crypto_sym_xform *xform, + uint16_t nb_queue_pairs) { const struct rte_crypto_sym_xform *cipher_xform = NULL; const struct rte_crypto_sym_xform *auth_xform = NULL; @@ -886,6 +879,12 @@ openssl_set_session_parameters(struct openssl_session *sess, } } + /* + * With only one queue pair, the array of copies is not needed. + * Otherwise, one entry per queue pair is required. + */ + sess->ctx_copies_len = nb_queue_pairs > 1 ? nb_queue_pairs : 0; + return 0; } @@ -893,6 +892,13 @@ openssl_set_session_parameters(struct openssl_session *sess, void openssl_reset_session(struct openssl_session *sess) { + for (uint16_t i = 0; i < sess->ctx_copies_len; i++) { + if (sess->qp_ctx[i] != NULL) { + EVP_CIPHER_CTX_free(sess->qp_ctx[i]); + sess->qp_ctx[i] = NULL; + } + } + EVP_CIPHER_CTX_free(sess->cipher.ctx); if (sess->chain_order == OPENSSL_CHAIN_CIPHER_BPI) @@ -959,7 +965,7 @@ get_session(struct openssl_qp *qp, struct rte_crypto_op *op) sess = (struct openssl_session *)_sess->driver_priv_data; if (unlikely(openssl_set_session_parameters(sess, - op->sym->xform) != 0)) { + op->sym->xform, 1) != 0)) { rte_mempool_put(qp->sess_mp, _sess); sess = NULL; } @@ -1607,11 +1613,45 @@ process_auth_err: # endif /*----------------------------------------------------------------------------*/ +static inline EVP_CIPHER_CTX * +get_local_cipher_ctx(struct openssl_session *sess, struct openssl_qp *qp) +{ + /* If the array is not being used, just return the main context. */ + if (sess->ctx_copies_len == 0) + return sess->cipher.ctx; + + EVP_CIPHER_CTX **lctx = &sess->qp_ctx[qp->id]; + + if (unlikely(*lctx == NULL)) { +#if OPENSSL_VERSION_NUMBER >= 0x30200000L + /* EVP_CIPHER_CTX_dup() added in OSSL 3.2 */ + *lctx = EVP_CIPHER_CTX_dup(sess->cipher.ctx); + return *lctx; +#elif OPENSSL_VERSION_NUMBER >= 0x30000000L + if (sess->chain_order == OPENSSL_CHAIN_COMBINED) { + /* AESNI special-cased to use openssl_aesni_ctx_clone() + * to allow for working around lack of + * EVP_CIPHER_CTX_copy support for 3.0.0 <= OSSL Version + * < 3.2.0. + */ + if (openssl_aesni_ctx_clone(lctx, sess) != 0) + *lctx = NULL; + return *lctx; + } +#endif + + *lctx = EVP_CIPHER_CTX_new(); + EVP_CIPHER_CTX_copy(*lctx, sess->cipher.ctx); + } + + return *lctx; +} + /** Process auth/cipher combined operation */ static void -process_openssl_combined_op - (struct rte_crypto_op *op, struct openssl_session *sess, - struct rte_mbuf *mbuf_src, struct rte_mbuf *mbuf_dst) +process_openssl_combined_op(struct openssl_qp *qp, struct rte_crypto_op *op, + struct openssl_session *sess, struct rte_mbuf *mbuf_src, + struct rte_mbuf *mbuf_dst) { /* cipher */ uint8_t *dst = NULL, *iv, *tag, *aad; @@ -1628,11 +1668,7 @@ process_openssl_combined_op return; } - EVP_CIPHER_CTX *ctx; - if (openssl_aesni_ctx_clone(&ctx, sess) != 0) { - op->status = RTE_CRYPTO_OP_STATUS_ERROR; - return; - } + EVP_CIPHER_CTX *ctx = get_local_cipher_ctx(sess, qp); iv = rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv.offset); @@ -1688,8 +1724,6 @@ process_openssl_combined_op dst, tag, taglen, ctx); } - EVP_CIPHER_CTX_free(ctx); - if (status != 0) { if (status == (-EFAULT) && sess->auth.operation == @@ -1702,14 +1736,13 @@ process_openssl_combined_op /** Process cipher operation */ static void -process_openssl_cipher_op - (struct rte_crypto_op *op, struct openssl_session *sess, - struct rte_mbuf *mbuf_src, struct rte_mbuf *mbuf_dst) +process_openssl_cipher_op(struct openssl_qp *qp, struct rte_crypto_op *op, + struct openssl_session *sess, struct rte_mbuf *mbuf_src, + struct rte_mbuf *mbuf_dst) { uint8_t *dst, *iv; int srclen, status; uint8_t inplace = (mbuf_src == mbuf_dst) ? 1 : 0; - EVP_CIPHER_CTX *ctx_copy; /* * Segmented OOP destination buffer is not supported for encryption/ @@ -1728,24 +1761,22 @@ process_openssl_cipher_op iv = rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv.offset); - ctx_copy = EVP_CIPHER_CTX_new(); - EVP_CIPHER_CTX_copy(ctx_copy, sess->cipher.ctx); + + EVP_CIPHER_CTX *ctx = get_local_cipher_ctx(sess, qp); if (sess->cipher.mode == OPENSSL_CIPHER_LIB) if (sess->cipher.direction == RTE_CRYPTO_CIPHER_OP_ENCRYPT) status = process_openssl_cipher_encrypt(mbuf_src, dst, op->sym->cipher.data.offset, iv, - srclen, ctx_copy, inplace); + srclen, ctx, inplace); else status = process_openssl_cipher_decrypt(mbuf_src, dst, op->sym->cipher.data.offset, iv, - srclen, ctx_copy, inplace); + srclen, ctx, inplace); else status = process_openssl_cipher_des3ctr(mbuf_src, dst, - op->sym->cipher.data.offset, iv, srclen, - ctx_copy); + op->sym->cipher.data.offset, iv, srclen, ctx); - EVP_CIPHER_CTX_free(ctx_copy); if (status != 0) op->status = RTE_CRYPTO_OP_STATUS_ERROR; } @@ -3150,13 +3181,13 @@ process_op(struct openssl_qp *qp, struct rte_crypto_op *op, switch (sess->chain_order) { case OPENSSL_CHAIN_ONLY_CIPHER: - process_openssl_cipher_op(op, sess, msrc, mdst); + process_openssl_cipher_op(qp, op, sess, msrc, mdst); break; case OPENSSL_CHAIN_ONLY_AUTH: process_openssl_auth_op(qp, op, sess, msrc, mdst); break; case OPENSSL_CHAIN_CIPHER_AUTH: - process_openssl_cipher_op(op, sess, msrc, mdst); + process_openssl_cipher_op(qp, op, sess, msrc, mdst); /* OOP */ if (msrc != mdst) copy_plaintext(msrc, mdst, op); @@ -3164,10 +3195,10 @@ process_op(struct openssl_qp *qp, struct rte_crypto_op *op, break; case OPENSSL_CHAIN_AUTH_CIPHER: process_openssl_auth_op(qp, op, sess, msrc, mdst); - process_openssl_cipher_op(op, sess, msrc, mdst); + process_openssl_cipher_op(qp, op, sess, msrc, mdst); break; case OPENSSL_CHAIN_COMBINED: - process_openssl_combined_op(op, sess, msrc, mdst); + process_openssl_combined_op(qp, op, sess, msrc, mdst); break; case OPENSSL_CHAIN_CIPHER_BPI: process_openssl_docsis_bpi_op(op, sess, msrc, mdst); diff --git a/drivers/crypto/openssl/rte_openssl_pmd_ops.c b/drivers/crypto/openssl/rte_openssl_pmd_ops.c index b16baaa08f..4209c6ab6f 100644 --- a/drivers/crypto/openssl/rte_openssl_pmd_ops.c +++ b/drivers/crypto/openssl/rte_openssl_pmd_ops.c @@ -794,9 +794,34 @@ qp_setup_cleanup: /** Returns the size of the symmetric session structure */ static unsigned -openssl_pmd_sym_session_get_size(struct rte_cryptodev *dev __rte_unused) +openssl_pmd_sym_session_get_size(struct rte_cryptodev *dev) { - return sizeof(struct openssl_session); + /* + * For 0 qps, return the max size of the session - this is necessary if + * the user calls into this function to create the session mempool, + * without first configuring the number of qps for the cryptodev. + */ + if (dev->data->nb_queue_pairs == 0) { + unsigned int max_nb_qps = ((struct openssl_private *) + dev->data->dev_private)->max_nb_qpairs; + return sizeof(struct openssl_session) + + (sizeof(void *) * max_nb_qps); + } + + /* + * With only one queue pair, the thread safety of multiple context + * copies is not necessary, so don't allocate extra memory for the + * array. + */ + if (dev->data->nb_queue_pairs == 1) + return sizeof(struct openssl_session); + + /* + * Otherwise, the size of the flexible array member should be enough to + * fit pointers to per-qp contexts. + */ + return sizeof(struct openssl_session) + + (sizeof(void *) * dev->data->nb_queue_pairs); } /** Returns the size of the asymmetric session structure */ @@ -808,7 +833,7 @@ openssl_pmd_asym_session_get_size(struct rte_cryptodev *dev __rte_unused) /** Configure the session from a crypto xform chain */ static int -openssl_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused, +openssl_pmd_sym_session_configure(struct rte_cryptodev *dev, struct rte_crypto_sym_xform *xform, struct rte_cryptodev_sym_session *sess) { @@ -820,7 +845,8 @@ openssl_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused, return -EINVAL; } - ret = openssl_set_session_parameters(sess_private_data, xform); + ret = openssl_set_session_parameters(sess_private_data, xform, + dev->data->nb_queue_pairs); if (ret != 0) { OPENSSL_LOG(ERR, "failed configure session parameters"); -- 2.34.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2024-08-12 20:44:03.442227826 +0800 +++ 0027-crypto-openssl-make-per-QP-cipher-context-clones.patch 2024-08-12 20:44:01.965069269 +0800 @@ -1 +1 @@ -From b1d71126023521fe740ec473abfe5b295035b859 Mon Sep 17 00:00:00 2001 +From 4f8c97e94187b19adde37b6b01d42441e10f91da Mon Sep 17 00:00:00 2001 @@ -4,0 +5,3 @@ +Cc: Xueming Li + +[ upstream commit b1d71126023521fe740ec473abfe5b295035b859 ] @@ -38,2 +40,0 @@ -Cc: stable@dpdk.org - @@ -50 +51 @@ -index 0f038b218c..bad7dcf2f5 100644 +index 334912d335..370de1d53b 100644 @@ -53 +54 @@ -@@ -166,6 +166,14 @@ struct __rte_cache_aligned openssl_session { +@@ -166,6 +166,14 @@ struct openssl_session { @@ -65 +66 @@ - }; + } __rte_cache_aligned; @@ -68 +69 @@ -@@ -217,7 +225,8 @@ struct __rte_cache_aligned openssl_asym_session { +@@ -217,7 +225,8 @@ struct openssl_asym_session {