From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A8ACD45538; Mon, 1 Jul 2024 14:55:52 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 479CA427DE; Mon, 1 Jul 2024 14:55:50 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.19]) by mails.dpdk.org (Postfix) with ESMTP id 15083427C0 for ; Mon, 1 Jul 2024 14:55:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1719838548; x=1751374548; h=from:to:cc:subject:date:message-id:references: in-reply-to:mime-version; bh=Ha8i+9LY4Y7HhrMQb/Fb85vq9Af+DAzxFS9NxdUNMMc=; b=nveKs2MsT49o5q1nsKmnjcuux1auzzRZ0RGyaacRCQFCtVgm7VVKyu96 s3lhYsc/gWk+lb/KEj2ACuKVhdfurOhWDdYYeQy5Pl7Y/nwZATw2ITLX2 UtragHvNgPg+UxImIKFT2uTjNf4zgonVLGpwk1knkytHXerp9zBBzV5Q7 rjeM0WfPdDCVoYSrTBF6IYZ8YA+YRRXSFWVDa3iG5jGziRbjwc9CAvn98 0fOqOrcuhivrmZrXYNs7Whrl3P+6xm3j+qUT8qUdnkTGf2c1s4LNkHg8K AdQBZy8SOEgiIF65AT4pbk2l7JW9G/QHniTJOaff8qUwNK1G4EU36negS g==; X-CSE-ConnectionGUID: fpFkdHmOSFSivQzHmw2QPw== X-CSE-MsgGUID: 6p+CY0QvQHqnsjQA1zjFeQ== X-IronPort-AV: E=McAfee;i="6700,10204,11120"; a="16691714" X-IronPort-AV: E=Sophos;i="6.09,176,1716274800"; d="scan'208,217";a="16691714" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa113.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jul 2024 05:55:47 -0700 X-CSE-ConnectionGUID: mlIo2FCHQN+iYIDg4I2L3Q== X-CSE-MsgGUID: Xgx1Q0i8TAKpMnLh7xVgpw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,176,1716274800"; d="scan'208,217";a="76234301" Received: from orsmsx601.amr.corp.intel.com ([10.22.229.14]) by orviesa002.jf.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 01 Jul 2024 05:55:47 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Mon, 1 Jul 2024 05:55:46 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Mon, 1 Jul 2024 05:55:46 -0700 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Mon, 1 Jul 2024 05:55:46 -0700 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (104.47.55.48) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Mon, 1 Jul 2024 05:55:45 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hyzku2H/VhxVPJgtxtD4aXOlKCbSDhOD5dIHtZJ/7lUGJtIq8DbLxcMrrVtjLvpuO8QpOcncuLbHH4DIMj0Nj42tsvkYrwpiZh45ByICkuO1wU5NgsBwd+qgNBeEzlDJRbZ23vavkc9zyiRh2t9cmFA95shliiD6QMeVUTT69dlQepqnt+tc7wls/KWtGhD7a/zZL4ruFfi5mTAAlrsDGKQ35Erp3Nb1+EHHcVkY0ljAyQRwlvEBSGIoKKr3ZXe2cbW1jwV+qkJR1cmfmnZMWgWJthiFiYZrxXUj1NA6JhRhE5J+z7uDnfaG9L5v/A35dnYGJMsgZClxKhejG0b60g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2inIosH0277goKXXc8ngfvybdyNLXD1IBWI+16DygUQ=; b=nQaQSTy47jqtxaZ7nvraqQ1CdEnuEz0jEw9uXAkUO3QLZYTWmUZvytvyI+f8IaAPkohZuZnendtwkw/4ucEHCYpTYpsbuIRDNGl1vMwpBgwO2PeDCQywhD/m14Haj/W4M4Ouk5mDUO2NI93AObquguhVRqeMu+kyXzkixKhyl26Q8oJX5OfmQJLexh2pqjU39zTy9i4bUixKzLDH3RP8RKWFbne6/QmGBHFuLeMaAdGEK2W28ztNODdEGA6sqniXjJE9oYqT57bqxExHB6bYcbNWpByd4VcK183wqhUf0CLzEIZFDHEtDep9Mzqn7O4W5+Bo/UJDxdsKXKB1rmNizA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from DS0PR11MB7458.namprd11.prod.outlook.com (2603:10b6:8:145::13) by DM4PR11MB6408.namprd11.prod.outlook.com (2603:10b6:8:b7::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7719.32; Mon, 1 Jul 2024 12:55:44 +0000 Received: from DS0PR11MB7458.namprd11.prod.outlook.com ([fe80::1a9e:53a6:9603:8f79]) by DS0PR11MB7458.namprd11.prod.outlook.com ([fe80::1a9e:53a6:9603:8f79%5]) with mapi id 15.20.7719.028; Mon, 1 Jul 2024 12:55:44 +0000 From: "Ji, Kai" To: Jack Bond-Preston CC: "dev@dpdk.org" , Wathsala Vithanage Subject: Re: [PATCH v4 4/5] crypto/openssl: per-qp auth context clones Thread-Topic: [PATCH v4 4/5] crypto/openssl: per-qp auth context clones Thread-Index: AQHauNkOYGPLdpoAWkCHCOJ6+g+BrrHh+jYc Date: Mon, 1 Jul 2024 12:55:44 +0000 Message-ID: References: <20240603160119.1279476-1-jack.bond-preston@foss.arm.com> <20240607124756.3968704-1-jack.bond-preston@foss.arm.com> <20240607124756.3968704-5-jack.bond-preston@foss.arm.com> In-Reply-To: <20240607124756.3968704-5-jack.bond-preston@foss.arm.com> Accept-Language: en-GB, en-US, en-IE Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DS0PR11MB7458:EE_|DM4PR11MB6408:EE_ x-ms-office365-filtering-correlation-id: 3f13fa35-dcc2-408c-9d2c-08dc99cd1e9b x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|1800799024|376014|366016|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?xeVZ9l7XhJCd5pm9BEuqbDFXA8N+W2aso+EgBQTJAKAeL9mU/YYRxQfo06w6?= =?us-ascii?Q?VSWWR4x6g0vq+zlzwVCCE3qaWUHT1p1Wr9Oo/CLjr2GYQz+4h3WZ5B5Az3fU?= =?us-ascii?Q?+O2ufUnROxqQ6pzON4WDkD+m1S0ZCCwX3YHsxqAPsOotJXcHR7lRDlB/vwIl?= =?us-ascii?Q?XRTd1kz2nxYDKSAS+ie4aYFcIgt0yk7cElV3XSt+n02YIBw1KNz+NIAKv1m5?= =?us-ascii?Q?P4ZlB1Rj+JOigozTH9A2HEEaI5mpaxWw9k3XbyRx+7XL5ECjUOgHpClc2dkF?= =?us-ascii?Q?ntat3O4YuvAV8VSAroEFcOzFXdcdJAJETaqrm3a3bai1G2TABMNG0FARe+2C?= =?us-ascii?Q?8tTCiqdctpL11l5LDqcWdyvmSF0ntOwdPMY/kvIBp8i1ARVm3IH0KGHzeewZ?= =?us-ascii?Q?C45zoLDSya5kDmukOFzvWj8F5KGwlR+mD3EGUF+qhUDKWFTEoohaJrTYl4CV?= =?us-ascii?Q?O8nAN2IVLWkuBrW0rDFUM9uPeaOQiauVYAaXaC+4RT8RwCjWUZWwnpmhPs70?= =?us-ascii?Q?JJCQlL6KMosAmlO3zrZB3yuu9qZ2rj+zn9F6X6e+Avztw7HgjY50cIueHAle?= =?us-ascii?Q?YzB227CoE37/u8W2QqvVpNF5j0HTGPm+54BYbLjOcZt6Y0r3BuuEmQA+gKhR?= =?us-ascii?Q?mybz9iBv46fR7LsG2ZIXW8a9b2r4s59I6ZffGxWIeb5WVYf3h0IAfHQJgBdU?= =?us-ascii?Q?GCo07wZTSq2w07qiRduJ5G45HG0C6h2v60oH8fM0YuwtYv8GPTagXh93ECuL?= =?us-ascii?Q?MAd8hMnYf3e85ZcTkRX9gRxiJgH8Z6isGfTMtJo0w5P0DbnBKK4+cYW6Jhtg?= =?us-ascii?Q?ajATknHfA7IltGzKmYscK+so5GTFJoDperGAkhRO0SYDUuwAElfQkYadknRk?= =?us-ascii?Q?7qzMHYofxawzywePOOvnGD1ykmDYuX9pf4gRNviYRjsfN9wDqPyJGJBWMht6?= =?us-ascii?Q?gkTIoNSA8eXzRAWhTx10PpOqgaEdl0imzAqnerYCWwqVX3d6K3ILgIJKFfow?= =?us-ascii?Q?yjmEfYvk1Jzbp8XSxSq+1oY3OxZWoWbpMakLfoBgIngdb/lSaDEaXItQWSFp?= =?us-ascii?Q?liQ4zS8X/QAfW2ZjgtgwFJhJ/Jz8iCrn0di6+kW++KjBgi06jMZ4A0EaRnZu?= =?us-ascii?Q?3ZUe5o7xbxSFVbvWXE5b3TyDFmNeB1v5hpoFK9xzojuwXaT858FM0lZ0U83w?= =?us-ascii?Q?AfQG8D2atfWK1fzrO7mHAJ21pTXXmSNTv91wJRcAcj6JrWlmtVE0CBBk4WEP?= =?us-ascii?Q?OhqA9x57M++Abf1gvtOXpUtljlmOzs/2Q5IiKyqMncBVsZ6dCmIOTnQpvkzB?= =?us-ascii?Q?5LNAuHL2yQpLOlQIRo2Sluo8eLoRXIs3OrSC80XYXsw3AWoVEsvgQyoHeYX9?= =?us-ascii?Q?l+VBVTfnyfYFZHI2dETzX8Zksvo4iyKomZq9FeznidZ5Eboetw=3D=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7458.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(376014)(366016)(38070700018); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?xxCThDt1NUJ848ahVikEF8ukViyTNaSmSNTPVcwmZVr/PMmIaw+DM3WUfkmh?= =?us-ascii?Q?zqvNuGuiI8uMXLkwfhDpkR36VlibNIlnlljtEkirDAxo/DZ8QSkl9EBTY3z+?= =?us-ascii?Q?8dK5f76T+S89VcQjSgLDcdjjkQSOQGiR3FA+wPlWQ0LcrpPX05PFaNEOUzRb?= =?us-ascii?Q?2syKCwNGVrLqXMH1eUPjHL3osy3v2nIfUoa2UOgy2CPEVVP0aLtUfV4yySxT?= =?us-ascii?Q?KdIwJdt4LhXdKpbs5jcRBXHJyz02PVDpEEEu3RH6JjpsBFtivovDP6xCqXWz?= =?us-ascii?Q?1h4lH6Wp8xU0pJBJEg6mGSftTwXQ9Lm/kfygaP5kt1k+pZrt+nOtr9knziuO?= =?us-ascii?Q?1Q2qJwGBaRWZl8ZuSv67IdKPcELb993/Xd/QZ+iognCUSmbbciuHlBr5jXZK?= =?us-ascii?Q?nndld24lyqxGA+zC2v2U6BWUR0wzwAvWOWZ7Q/9Lr8x6oITOviFF1qOQMR3Y?= =?us-ascii?Q?x5PaFYamJ+8EV+frEtQ2HPK2aGxvB5OTKpy4G6inwifFvapF7xWuCsv34ezM?= =?us-ascii?Q?1QKobbKfVAG9oR0o1bIkA4etjEB/KrJyiT9XwC7vGyTzIjgDHbQSJSolJoOP?= =?us-ascii?Q?XA7SMq+Su8Uwu//6DRR5tva2PB7ijJb3czBm/2iJXbbANjMK9sK7crza9DV9?= =?us-ascii?Q?Ib2C30ZHNFCvnORhwiv3d17XWjFJX0H1AVYoVVPa7vQchWt8QNLm95QRT9Rn?= =?us-ascii?Q?WXfw/rFrLsQEZdIWtCalbsu6op2gNue939oKGvkop2Bdw9yS7W3L7UB0MZJk?= =?us-ascii?Q?yUZSrSvT+JE150IwpHwVTCseFgTQJk7jI7c8KGu5rxJYN7OYcW8dbKS+1Bb3?= =?us-ascii?Q?7x0fZks4+f43EUBWCs/c53a4E2NKAewRbkpfUrG7QXtIclh1cNTX9zb7DmbI?= =?us-ascii?Q?KflGD1+krYXX1vBY2UVcrYZ9GzHQURb/YgnLcNVinf4Tn5NUkGP+Wp9Nxg5s?= =?us-ascii?Q?xwFUNkk049ok3jv7MU6VNR0cEoNoqtjYmSCKXWUq2Nc2aX2JY4od1g7WMKna?= =?us-ascii?Q?A8uBzubb29NFrbo4X7sIYOTNDpACTep7VE1XrzmY3wsMBwEmXiroPuHrd6Ow?= =?us-ascii?Q?Yh2jGULRDkEMOsDECfCVp+8v/j0vMLTRhUqjH2FgVxc5iIrKgXmnvT470vaH?= =?us-ascii?Q?CU85q+sdeps0FJP0ECAlGAo/szlQVv2Q0OFyr8PzS68OvvFJsT+y/vsnyc7G?= =?us-ascii?Q?qsr5H6OYbbVVnjMjC3cuHOz4mI7+KOl/RBimDJRcZbpQwuINXtvFuMrkMmgx?= =?us-ascii?Q?7OkZdmcFsAzBOAkJweLpTzN2XT8ISAs45T/P6+L43I6s62/dwD82e1FsOWWJ?= =?us-ascii?Q?L7uIXzht9C9Ebz1oDQwugrzs2kJKzqqgqJafSoVAxrIvj1Q0eZSisBoyQAAR?= =?us-ascii?Q?MbjhsyiEcCh22rsw9HrA+NETn6yy/SnPeDk+/r8VK+T3axFGMjt03YSL75g1?= =?us-ascii?Q?9pf+jkobPv88/hnSlq6lGlq2QTgUofG67NDosbnqvWf/GgE4B3hF9QjyJWBy?= =?us-ascii?Q?2XiCVOeNGmVCTjPi1U8fXJGleT7CT+8CZRNlLn3CfEkbcHIWWx85RtmzbGYT?= =?us-ascii?Q?BP/2ZtXr0QwT54Oudl0=3D?= Content-Type: multipart/alternative; boundary="_000_DS0PR11MB7458E2235E4EA742C4B4DB6181D32DS0PR11MB7458namp_" MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7458.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3f13fa35-dcc2-408c-9d2c-08dc99cd1e9b X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Jul 2024 12:55:44.0414 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: TRw6BSk9Zjn1yJ5qc/o7Mh+GEAJLtE3kH3k6kKnZ9p/MK5tshUsd+SKH37FOwPsQ9pdx45hs36swRyxv06DU7w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB6408 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --_000_DS0PR11MB7458E2235E4EA742C4B4DB6181D32DS0PR11MB7458namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Acked-by: Kai Ji ________________________________ From: Jack Bond-Preston Sent: 07 June 2024 13:47 To: Ji, Kai Cc: dev@dpdk.org ; Wathsala Vithanage Subject: [PATCH v4 4/5] crypto/openssl: per-qp auth context clones Currently EVP auth ctxs (e.g. EVP_MD_CTX, EVP_MAC_CTX) are allocated, copied to (from openssl_session), and then freed for every auth operation (ie. per packet). This is very inefficient, and avoidable. Make each openssl_session hold an array of structures, containing pointers to per-queue-pair cipher and auth context copies. These are populated on first use by allocating a new context and copying from the main context. These copies can then be used in a thread-safe manner by different worker lcores simultaneously. Consequently the auth context allocation and copy only has to happen once - the first time a given qp uses an openssl_session. This brings about a large performance boost. Throughput performance uplift measurements for HMAC-SHA1 generate on Ampere Altra Max platform: 1 worker lcore | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 0.63 | 1.42 | 123.5% | | 256 | 2.24 | 4.40 | 96.4% | | 1024 | 6.15 | 9.26 | 50.6% | | 2048 | 8.68 | 11.38 | 31.1% | | 4096 | 10.92 | 12.84 | 17.6% | 8 worker lcores | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 0.93 | 11.35 | 1122.5% | | 256 | 3.70 | 35.30 | 853.7% | | 1024 | 15.22 | 74.27 | 387.8% | | 2048 | 30.20 | 91.08 | 201.6% | | 4096 | 56.92 | 102.76 | 80.5% | Signed-off-by: Jack Bond-Preston Reviewed-by: Wathsala Vithanage --- drivers/crypto/openssl/compat.h | 26 +++ drivers/crypto/openssl/openssl_pmd_private.h | 25 ++- drivers/crypto/openssl/rte_openssl_pmd.c | 176 +++++++++++++++---- drivers/crypto/openssl/rte_openssl_pmd_ops.c | 7 +- 4 files changed, 193 insertions(+), 41 deletions(-) diff --git a/drivers/crypto/openssl/compat.h b/drivers/crypto/openssl/compa= t.h index 9f9167c4f1..4c5ddfbf3a 100644 --- a/drivers/crypto/openssl/compat.h +++ b/drivers/crypto/openssl/compat.h @@ -5,6 +5,32 @@ #ifndef __RTA_COMPAT_H__ #define __RTA_COMPAT_H__ +#if OPENSSL_VERSION_NUMBER > 0x30000000L +static __rte_always_inline void +free_hmac_ctx(EVP_MAC_CTX *ctx) +{ + EVP_MAC_CTX_free(ctx); +} + +static __rte_always_inline void +free_cmac_ctx(EVP_MAC_CTX *ctx) +{ + EVP_MAC_CTX_free(ctx); +} +#else +static __rte_always_inline void +free_hmac_ctx(HMAC_CTX *ctx) +{ + HMAC_CTX_free(ctx); +} + +static __rte_always_inline void +free_cmac_ctx(CMAC_CTX *ctx) +{ + CMAC_CTX_free(ctx); +} +#endif + #if (OPENSSL_VERSION_NUMBER < 0x10100000L) static __rte_always_inline int diff --git a/drivers/crypto/openssl/openssl_pmd_private.h b/drivers/crypto/= openssl/openssl_pmd_private.h index bad7dcf2f5..a50e4d4918 100644 --- a/drivers/crypto/openssl/openssl_pmd_private.h +++ b/drivers/crypto/openssl/openssl_pmd_private.h @@ -80,6 +80,20 @@ struct __rte_cache_aligned openssl_qp { */ }; +struct evp_ctx_pair { + EVP_CIPHER_CTX *cipher; + union { + EVP_MD_CTX *auth; +#if OPENSSL_VERSION_NUMBER >=3D 0x30000000L + EVP_MAC_CTX *hmac; + EVP_MAC_CTX *cmac; +#else + HMAC_CTX *hmac; + CMAC_CTX *cmac; +#endif + }; +}; + /** OPENSSL crypto private session structure */ struct __rte_cache_aligned openssl_session { enum openssl_chain_order chain_order; @@ -168,11 +182,12 @@ struct __rte_cache_aligned openssl_session { uint16_t ctx_copies_len; /* < number of entries in ctx_copies */ - EVP_CIPHER_CTX *qp_ctx[]; - /**< Flexible array member of per-queue-pair pointers to copies of = EVP - * context structure. Cipher contexts are not safe to use from mult= iple - * cores simultaneously, so maintaining these copies allows avoidin= g - * per-buffer copying into a temporary context. + struct evp_ctx_pair qp_ctx[]; + /**< Flexible array member of per-queue-pair structures, each conta= ining + * pointers to copies of the cipher and auth EVP contexts. Cipher + * contexts are not safe to use from multiple cores simultaneously,= so + * maintaining these copies allows avoiding per-buffer copying into= a + * temporary context. */ }; diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/open= ssl/rte_openssl_pmd.c index df44cc097e..7e2e505222 100644 --- a/drivers/crypto/openssl/rte_openssl_pmd.c +++ b/drivers/crypto/openssl/rte_openssl_pmd.c @@ -892,40 +892,45 @@ openssl_set_session_parameters(struct openssl_session= *sess, void openssl_reset_session(struct openssl_session *sess) { + /* Free all the qp_ctx entries. */ for (uint16_t i =3D 0; i < sess->ctx_copies_len; i++) { - if (sess->qp_ctx[i] !=3D NULL) { - EVP_CIPHER_CTX_free(sess->qp_ctx[i]); - sess->qp_ctx[i] =3D NULL; + if (sess->qp_ctx[i].cipher !=3D NULL) { + EVP_CIPHER_CTX_free(sess->qp_ctx[i].cipher); + sess->qp_ctx[i].cipher =3D NULL; + } + + switch (sess->auth.mode) { + case OPENSSL_AUTH_AS_AUTH: + EVP_MD_CTX_destroy(sess->qp_ctx[i].auth); + sess->qp_ctx[i].auth =3D NULL; + break; + case OPENSSL_AUTH_AS_HMAC: + free_hmac_ctx(sess->qp_ctx[i].hmac); + sess->qp_ctx[i].hmac =3D NULL; + break; + case OPENSSL_AUTH_AS_CMAC: + free_cmac_ctx(sess->qp_ctx[i].cmac); + sess->qp_ctx[i].cmac =3D NULL; + break; } } EVP_CIPHER_CTX_free(sess->cipher.ctx); - if (sess->chain_order =3D=3D OPENSSL_CHAIN_CIPHER_BPI) - EVP_CIPHER_CTX_free(sess->cipher.bpi_ctx); - switch (sess->auth.mode) { case OPENSSL_AUTH_AS_AUTH: EVP_MD_CTX_destroy(sess->auth.auth.ctx); break; case OPENSSL_AUTH_AS_HMAC: - EVP_PKEY_free(sess->auth.hmac.pkey); -# if OPENSSL_VERSION_NUMBER >=3D 0x30000000L - EVP_MAC_CTX_free(sess->auth.hmac.ctx); -# else - HMAC_CTX_free(sess->auth.hmac.ctx); -# endif + free_hmac_ctx(sess->auth.hmac.ctx); break; case OPENSSL_AUTH_AS_CMAC: -# if OPENSSL_VERSION_NUMBER >=3D 0x30000000L - EVP_MAC_CTX_free(sess->auth.cmac.ctx); -# else - CMAC_CTX_free(sess->auth.cmac.ctx); -# endif - break; - default: + free_cmac_ctx(sess->auth.cmac.ctx); break; } + + if (sess->chain_order =3D=3D OPENSSL_CHAIN_CIPHER_BPI) + EVP_CIPHER_CTX_free(sess->cipher.bpi_ctx); } /** Provide session for operation */ @@ -1471,6 +1476,9 @@ process_openssl_auth_mac(struct rte_mbuf *mbuf_src, u= int8_t *dst, int offset, if (m =3D=3D 0) goto process_auth_err; + if (EVP_MAC_init(ctx, NULL, 0, NULL) <=3D 0) + goto process_auth_err; + src =3D rte_pktmbuf_mtod_offset(m, uint8_t *, offset); l =3D rte_pktmbuf_data_len(m) - offset; @@ -1497,11 +1505,9 @@ process_openssl_auth_mac(struct rte_mbuf *mbuf_src, = uint8_t *dst, int offset, if (EVP_MAC_final(ctx, dst, &dstlen, DIGEST_LENGTH_MAX) !=3D 1) goto process_auth_err; - EVP_MAC_CTX_free(ctx); return 0; process_auth_err: - EVP_MAC_CTX_free(ctx); OPENSSL_LOG(ERR, "Process openssl auth failed"); return -EINVAL; } @@ -1620,7 +1626,7 @@ get_local_cipher_ctx(struct openssl_session *sess, st= ruct openssl_qp *qp) if (sess->ctx_copies_len =3D=3D 0) return sess->cipher.ctx; - EVP_CIPHER_CTX **lctx =3D &sess->qp_ctx[qp->id]; + EVP_CIPHER_CTX **lctx =3D &sess->qp_ctx[qp->id].cipher; if (unlikely(*lctx =3D=3D NULL)) { #if OPENSSL_VERSION_NUMBER >=3D 0x30200000L @@ -1647,6 +1653,112 @@ get_local_cipher_ctx(struct openssl_session *sess, = struct openssl_qp *qp) return *lctx; } +static inline EVP_MD_CTX * +get_local_auth_ctx(struct openssl_session *sess, struct openssl_qp *qp) +{ + /* If the array is not being used, just return the main context. */ + if (sess->ctx_copies_len =3D=3D 0) + return sess->auth.auth.ctx; + + EVP_MD_CTX **lctx =3D &sess->qp_ctx[qp->id].auth; + + if (unlikely(*lctx =3D=3D NULL)) { +#if OPENSSL_VERSION_NUMBER >=3D 0x30100000L + /* EVP_MD_CTX_dup() added in OSSL 3.1 */ + *lctx =3D EVP_MD_CTX_dup(sess->auth.auth.ctx); +#else + *lctx =3D EVP_MD_CTX_new(); + EVP_MD_CTX_copy(*lctx, sess->auth.auth.ctx); +#endif + } + + return *lctx; +} + +#if OPENSSL_VERSION_NUMBER >=3D 0x30000000L +static inline EVP_MAC_CTX * +#else +static inline HMAC_CTX * +#endif +get_local_hmac_ctx(struct openssl_session *sess, struct openssl_qp *qp) +{ +#if (OPENSSL_VERSION_NUMBER >=3D 0x30000000L && OPENSSL_VERSION_NUMBER < 0= x30003000L) + /* For OpenSSL versions 3.0.0 <=3D v < 3.0.3, re-initing of + * EVP_MAC_CTXs is broken, and doesn't actually reset their + * state. This was fixed in OSSL commit c9ddc5af5199 ("Avoid + * undefined behavior of provided macs on EVP_MAC + * reinitialization"). In cases where the fix is not present, + * fall back to duplicating the context every buffer as a + * workaround, at the cost of performance. + */ + RTE_SET_USED(qp); + return EVP_MAC_CTX_dup(sess->auth.hmac.ctx); +#else + if (sess->ctx_copies_len =3D=3D 0) + return sess->auth.hmac.ctx; + +#if OPENSSL_VERSION_NUMBER >=3D 0x30000000L + EVP_MAC_CTX **lctx =3D +#else + HMAC_CTX **lctx =3D +#endif + &sess->qp_ctx[qp->id].hmac; + + if (unlikely(*lctx =3D=3D NULL)) { +#if OPENSSL_VERSION_NUMBER >=3D 0x30000000L + *lctx =3D EVP_MAC_CTX_dup(sess->auth.hmac.ctx); +#else + *lctx =3D HMAC_CTX_new(); + HMAC_CTX_copy(*lctx, sess->auth.hmac.ctx); +#endif + } + + return *lctx; +#endif +} + +#if OPENSSL_VERSION_NUMBER >=3D 0x30000000L +static inline EVP_MAC_CTX * +#else +static inline CMAC_CTX * +#endif +get_local_cmac_ctx(struct openssl_session *sess, struct openssl_qp *qp) +{ +#if (OPENSSL_VERSION_NUMBER >=3D 0x30000000L && OPENSSL_VERSION_NUMBER < 0= x30003000L) + /* For OpenSSL versions 3.0.0 <=3D v < 3.0.3, re-initing of + * EVP_MAC_CTXs is broken, and doesn't actually reset their + * state. This was fixed in OSSL commit c9ddc5af5199 ("Avoid + * undefined behavior of provided macs on EVP_MAC + * reinitialization"). In cases where the fix is not present, + * fall back to duplicating the context every buffer as a + * workaround, at the cost of performance. + */ + RTE_SET_USED(qp); + return EVP_MAC_CTX_dup(sess->auth.cmac.ctx); +#else + if (sess->ctx_copies_len =3D=3D 0) + return sess->auth.cmac.ctx; + +#if OPENSSL_VERSION_NUMBER >=3D 0x30000000L + EVP_MAC_CTX **lctx =3D +#else + CMAC_CTX **lctx =3D +#endif + &sess->qp_ctx[qp->id].cmac; + + if (unlikely(*lctx =3D=3D NULL)) { +#if OPENSSL_VERSION_NUMBER >=3D 0x30000000L + *lctx =3D EVP_MAC_CTX_dup(sess->auth.cmac.ctx); +#else + *lctx =3D CMAC_CTX_new(); + CMAC_CTX_copy(*lctx, sess->auth.cmac.ctx); +#endif + } + + return *lctx; +#endif +} + /** Process auth/cipher combined operation */ static void process_openssl_combined_op(struct openssl_qp *qp, struct rte_crypto_op *o= p, @@ -1895,42 +2007,40 @@ process_openssl_auth_op(struct openssl_qp *qp, stru= ct rte_crypto_op *op, switch (sess->auth.mode) { case OPENSSL_AUTH_AS_AUTH: - ctx_a =3D EVP_MD_CTX_create(); - EVP_MD_CTX_copy_ex(ctx_a, sess->auth.auth.ctx); + ctx_a =3D get_local_auth_ctx(sess, qp); status =3D process_openssl_auth(mbuf_src, dst, op->sym->auth.data.offset, NULL, NULL, src= len, ctx_a, sess->auth.auth.evp_algo); - EVP_MD_CTX_destroy(ctx_a); break; case OPENSSL_AUTH_AS_HMAC: + ctx_h =3D get_local_hmac_ctx(sess, qp); # if OPENSSL_VERSION_NUMBER >=3D 0x30000000L - ctx_h =3D EVP_MAC_CTX_dup(sess->auth.hmac.ctx); status =3D process_openssl_auth_mac(mbuf_src, dst, op->sym->auth.data.offset, srclen, ctx_h); # else - ctx_h =3D HMAC_CTX_new(); - HMAC_CTX_copy(ctx_h, sess->auth.hmac.ctx); status =3D process_openssl_auth_hmac(mbuf_src, dst, op->sym->auth.data.offset, srclen, ctx_h); - HMAC_CTX_free(ctx_h); # endif +#if (OPENSSL_VERSION_NUMBER >=3D 0x30000000L && OPENSSL_VERSION_NUMBER < 0= x30003000L) + EVP_MAC_CTX_free(ctx_h); +#endif break; case OPENSSL_AUTH_AS_CMAC: + ctx_c =3D get_local_cmac_ctx(sess, qp); # if OPENSSL_VERSION_NUMBER >=3D 0x30000000L - ctx_c =3D EVP_MAC_CTX_dup(sess->auth.cmac.ctx); status =3D process_openssl_auth_mac(mbuf_src, dst, op->sym->auth.data.offset, srclen, ctx_c); # else - ctx_c =3D CMAC_CTX_new(); - CMAC_CTX_copy(ctx_c, sess->auth.cmac.ctx); status =3D process_openssl_auth_cmac(mbuf_src, dst, op->sym->auth.data.offset, srclen, ctx_c); - CMAC_CTX_free(ctx_c); # endif +#if (OPENSSL_VERSION_NUMBER >=3D 0x30000000L && OPENSSL_VERSION_NUMBER < 0= x30003000L) + EVP_MAC_CTX_free(ctx_c); +#endif break; default: status =3D -1; diff --git a/drivers/crypto/openssl/rte_openssl_pmd_ops.c b/drivers/crypto/= openssl/rte_openssl_pmd_ops.c index 4209c6ab6f..1bbb855a59 100644 --- a/drivers/crypto/openssl/rte_openssl_pmd_ops.c +++ b/drivers/crypto/openssl/rte_openssl_pmd_ops.c @@ -805,7 +805,7 @@ openssl_pmd_sym_session_get_size(struct rte_cryptodev *= dev) unsigned int max_nb_qps =3D ((struct openssl_private *) dev->data->dev_private)->max_nb_qpairs; return sizeof(struct openssl_session) + - (sizeof(void *) * max_nb_qps); + (sizeof(struct evp_ctx_pair) * max_nb_qps); } /* @@ -818,10 +818,11 @@ openssl_pmd_sym_session_get_size(struct rte_cryptodev= *dev) /* * Otherwise, the size of the flexible array member should be enou= gh to - * fit pointers to per-qp contexts. + * fit pointers to per-qp contexts. This is twice the number of que= ue + * pairs, to allow for auth and cipher contexts. */ return sizeof(struct openssl_session) + - (sizeof(void *) * dev->data->nb_queue_pairs); + (sizeof(struct evp_ctx_pair) * dev->data->nb_queue_pairs); } /** Returns the size of the asymmetric session structure */ -- 2.34.1 --_000_DS0PR11MB7458E2235E4EA742C4B4DB6181D32DS0PR11MB7458namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Acked-by: Kai Ji <kai.ji@intel.com>
From: Jack Bond-Preston <= ;jack.bond-preston@foss.arm.com>
Sent: 07 June 2024 13:47
To: Ji, Kai <kai.ji@intel.com>
Cc: dev@dpdk.org <dev@dpdk.org>; Wathsala Vithanage <wathsa= la.vithanage@arm.com>
Subject: [PATCH v4 4/5] crypto/openssl: per-qp auth context clones
 
Currently EVP auth ctxs (e.g. EVP_MD_CTX, EVP_MAC_= CTX) are allocated,
copied to (from openssl_session), and then freed for every auth
operation (ie. per packet). This is very inefficient, and avoidable.

Make each openssl_session hold an array of structures, containing
pointers to per-queue-pair cipher and auth context copies. These are
populated on first use by allocating a new context and copying from the
main context. These copies can then be used in a thread-safe manner by
different worker lcores simultaneously. Consequently the auth context
allocation and copy only has to happen once - the first time a given qp
uses an openssl_session. This brings about a large performance boost.

Throughput performance uplift measurements for HMAC-SHA1 generate on
Ampere Altra Max platform:
1 worker lcore
|   buffer sz (B) |   prev (Gbps) |   optimis= ed (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|            &n= bsp; 64 |          0.63 | = ;            &n= bsp; 1.42 |   123.5% |
|             2= 56 |          2.24 | &nbs= p;             = 4.40 |    96.4% |
|            1024 |&= nbsp;         6.15 |  &nb= sp;            9.26 = |    50.6% |
|            2048 |&= nbsp;         8.68 |  &nb= sp;           11.38 |&nbs= p;   31.1% |
|            4096 |&= nbsp;        10.92 |   &n= bsp;          12.84 | &nb= sp;  17.6% |

8 worker lcores
|   buffer sz (B) |   prev (Gbps) |   optimis= ed (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|            &n= bsp; 64 |          0.93 | = ;             1= 1.35 |  1122.5% |
|             2= 56 |          3.70 | &nbs= p;            35.30 = |   853.7% |
|            1024 |&= nbsp;        15.22 |   &n= bsp;          74.27 | &nb= sp; 387.8% |
|            2048 |&= nbsp;        30.20 |   &n= bsp;          91.08 | &nb= sp; 201.6% |
|            4096 |&= nbsp;        56.92 |   &n= bsp;         102.76 |  &n= bsp; 80.5% |

Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
---
 drivers/crypto/openssl/compat.h      &n= bsp;       |  26 +++
 drivers/crypto/openssl/openssl_pmd_private.h |  25 ++-
 drivers/crypto/openssl/rte_openssl_pmd.c     | 17= 6 +++++++++++++++----
 drivers/crypto/openssl/rte_openssl_pmd_ops.c |   7 +-
 4 files changed, 193 insertions(+), 41 deletions(-)

diff --git a/drivers/crypto/openssl/compat.h b/drivers/crypto/openssl/compa= t.h
index 9f9167c4f1..4c5ddfbf3a 100644
--- a/drivers/crypto/openssl/compat.h
+++ b/drivers/crypto/openssl/compat.h
@@ -5,6 +5,32 @@
 #ifndef __RTA_COMPAT_H__
 #define __RTA_COMPAT_H__
 
+#if OPENSSL_VERSION_NUMBER > 0x30000000L
+static __rte_always_inline void
+free_hmac_ctx(EVP_MAC_CTX *ctx)
+{
+       EVP_MAC_CTX_free(ctx);
+}
+
+static __rte_always_inline void
+free_cmac_ctx(EVP_MAC_CTX *ctx)
+{
+       EVP_MAC_CTX_free(ctx);
+}
+#else
+static __rte_always_inline void
+free_hmac_ctx(HMAC_CTX *ctx)
+{
+       HMAC_CTX_free(ctx);
+}
+
+static __rte_always_inline void
+free_cmac_ctx(CMAC_CTX *ctx)
+{
+       CMAC_CTX_free(ctx);
+}
+#endif
+
 #if (OPENSSL_VERSION_NUMBER < 0x10100000L)
 
 static __rte_always_inline int
diff --git a/drivers/crypto/openssl/openssl_pmd_private.h b/drivers/crypto/= openssl/openssl_pmd_private.h
index bad7dcf2f5..a50e4d4918 100644
--- a/drivers/crypto/openssl/openssl_pmd_private.h
+++ b/drivers/crypto/openssl/openssl_pmd_private.h
@@ -80,6 +80,20 @@ struct __rte_cache_aligned openssl_qp {
          */
 };
 
+struct evp_ctx_pair {
+       EVP_CIPHER_CTX *cipher;
+       union {
+            &n= bsp;  EVP_MD_CTX *auth;
+#if OPENSSL_VERSION_NUMBER >=3D 0x30000000L
+            &n= bsp;  EVP_MAC_CTX *hmac;
+            &n= bsp;  EVP_MAC_CTX *cmac;
+#else
+            &n= bsp;  HMAC_CTX *hmac;
+            &n= bsp;  CMAC_CTX *cmac;
+#endif
+       };
+};
+
 /** OPENSSL crypto private session structure */
 struct __rte_cache_aligned openssl_session {
         enum openssl_chain_order c= hain_order;
@@ -168,11 +182,12 @@ struct __rte_cache_aligned openssl_session {
 
         uint16_t ctx_copies_len;          /* < number of entries = in ctx_copies */
-       EVP_CIPHER_CTX *qp_ctx[];
-       /**< Flexible array member of per-= queue-pair pointers to copies of EVP
-        * context structure. Cipher con= texts are not safe to use from multiple
-        * cores simultaneously, so main= taining these copies allows avoiding
-        * per-buffer copying into a tem= porary context.
+       struct evp_ctx_pair qp_ctx[];
+       /**< Flexible array member of per-= queue-pair structures, each containing
+        * pointers to copies of the cip= her and auth EVP contexts. Cipher
+        * contexts are not safe to use = from multiple cores simultaneously, so
+        * maintaining these copies allo= ws avoiding per-buffer copying into a
+        * temporary context.
          */
 };
 
diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/open= ssl/rte_openssl_pmd.c
index df44cc097e..7e2e505222 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -892,40 +892,45 @@ openssl_set_session_parameters(struct openssl_session= *sess,
 void
 openssl_reset_session(struct openssl_session *sess)
 {
+       /* Free all the qp_ctx entries. */          for (uint16_t i =3D 0; i &= lt; sess->ctx_copies_len; i++) {
-            &n= bsp;  if (sess->qp_ctx[i] !=3D NULL) {
-            &n= bsp;          EVP_CIPHER_CTX_f= ree(sess->qp_ctx[i]);
-            &n= bsp;          sess->qp_ctx[= i] =3D NULL;
+            &n= bsp;  if (sess->qp_ctx[i].cipher !=3D NULL) {
+            &n= bsp;          EVP_CIPHER_CTX_f= ree(sess->qp_ctx[i].cipher);
+            &n= bsp;          sess->qp_ctx[= i].cipher =3D NULL;
+            &n= bsp;  }
+
+            &n= bsp;  switch (sess->auth.mode) {
+            &n= bsp;  case OPENSSL_AUTH_AS_AUTH:
+            &n= bsp;          EVP_MD_CTX_destr= oy(sess->qp_ctx[i].auth);
+            &n= bsp;          sess->qp_ctx[= i].auth =3D NULL;
+            &n= bsp;          break;
+            &n= bsp;  case OPENSSL_AUTH_AS_HMAC:
+            &n= bsp;          free_hmac_ctx(se= ss->qp_ctx[i].hmac);
+            &n= bsp;          sess->qp_ctx[= i].hmac =3D NULL;
+            &n= bsp;          break;
+            &n= bsp;  case OPENSSL_AUTH_AS_CMAC:
+            &n= bsp;          free_cmac_ctx(se= ss->qp_ctx[i].cmac);
+            &n= bsp;          sess->qp_ctx[= i].cmac =3D NULL;
+            &n= bsp;          break;
            &nb= sp;    }
         }
 
         EVP_CIPHER_CTX_free(sess-&= gt;cipher.ctx);
 
-       if (sess->chain_order =3D=3D OPENS= SL_CHAIN_CIPHER_BPI)
-            &n= bsp;  EVP_CIPHER_CTX_free(sess->cipher.bpi_ctx);
-
         switch (sess->auth.mode= ) {
         case OPENSSL_AUTH_AS_AUTH:=
            &nb= sp;    EVP_MD_CTX_destroy(sess->auth.auth.ctx);
            &nb= sp;    break;
         case OPENSSL_AUTH_AS_HMAC:=
-            &n= bsp;  EVP_PKEY_free(sess->auth.hmac.pkey);
-# if OPENSSL_VERSION_NUMBER >=3D 0x30000000L
-            &n= bsp;  EVP_MAC_CTX_free(sess->auth.hmac.ctx);
-# else
-            &n= bsp;  HMAC_CTX_free(sess->auth.hmac.ctx);
-# endif
+            &n= bsp;  free_hmac_ctx(sess->auth.hmac.ctx);
            &nb= sp;    break;
         case OPENSSL_AUTH_AS_CMAC:=
-# if OPENSSL_VERSION_NUMBER >=3D 0x30000000L
-            &n= bsp;  EVP_MAC_CTX_free(sess->auth.cmac.ctx);
-# else
-            &n= bsp;  CMAC_CTX_free(sess->auth.cmac.ctx);
-# endif
-            &n= bsp;  break;
-       default:
+            &n= bsp;  free_cmac_ctx(sess->auth.cmac.ctx);
            &nb= sp;    break;
         }
+
+       if (sess->chain_order =3D=3D OPENS= SL_CHAIN_CIPHER_BPI)
+            &n= bsp;  EVP_CIPHER_CTX_free(sess->cipher.bpi_ctx);
 }
 
 /** Provide session for operation */
@@ -1471,6 +1476,9 @@ process_openssl_auth_mac(struct rte_mbuf *mbuf_src, u= int8_t *dst, int offset,
         if (m =3D=3D 0)
            &nb= sp;    goto process_auth_err;
 
+       if (EVP_MAC_init(ctx, NULL, 0, NULL) = <=3D 0)
+            &n= bsp;  goto process_auth_err;
+
         src =3D rte_pktmbuf_mtod_o= ffset(m, uint8_t *, offset);
 
         l =3D rte_pktmbuf_data_len= (m) - offset;
@@ -1497,11 +1505,9 @@ process_openssl_auth_mac(struct rte_mbuf *mbuf_src, = uint8_t *dst, int offset,
         if (EVP_MAC_final(ctx, dst= , &dstlen, DIGEST_LENGTH_MAX) !=3D 1)
            &nb= sp;    goto process_auth_err;
 
-       EVP_MAC_CTX_free(ctx);
         return 0;
 
 process_auth_err:
-       EVP_MAC_CTX_free(ctx);
         OPENSSL_LOG(ERR, "Pro= cess openssl auth failed");
         return -EINVAL;
 }
@@ -1620,7 +1626,7 @@ get_local_cipher_ctx(struct openssl_session *sess, st= ruct openssl_qp *qp)
         if (sess->ctx_copies_le= n =3D=3D 0)
            &nb= sp;    return sess->cipher.ctx;
 
-       EVP_CIPHER_CTX **lctx =3D &sess-&= gt;qp_ctx[qp->id];
+       EVP_CIPHER_CTX **lctx =3D &sess-&= gt;qp_ctx[qp->id].cipher;
 
         if (unlikely(*lctx =3D=3D = NULL)) {
 #if OPENSSL_VERSION_NUMBER >=3D 0x30200000L
@@ -1647,6 +1653,112 @@ get_local_cipher_ctx(struct openssl_session *sess, = struct openssl_qp *qp)
         return *lctx;
 }
 
+static inline EVP_MD_CTX *
+get_local_auth_ctx(struct openssl_session *sess, struct openssl_qp *qp) +{
+       /* If the array is not being used, ju= st return the main context. */
+       if (sess->ctx_copies_len =3D=3D 0)=
+            &n= bsp;  return sess->auth.auth.ctx;
+
+       EVP_MD_CTX **lctx =3D &sess->q= p_ctx[qp->id].auth;
+
+       if (unlikely(*lctx =3D=3D NULL)) { +#if OPENSSL_VERSION_NUMBER >=3D 0x30100000L
+            &n= bsp;  /* EVP_MD_CTX_dup() added in OSSL 3.1 */
+            &n= bsp;  *lctx =3D EVP_MD_CTX_dup(sess->auth.auth.ctx);
+#else
+            &n= bsp;  *lctx =3D EVP_MD_CTX_new();
+            &n= bsp;  EVP_MD_CTX_copy(*lctx, sess->auth.auth.ctx);
+#endif
+       }
+
+       return *lctx;
+}
+
+#if OPENSSL_VERSION_NUMBER >=3D 0x30000000L
+static inline EVP_MAC_CTX *
+#else
+static inline HMAC_CTX *
+#endif
+get_local_hmac_ctx(struct openssl_session *sess, struct openssl_qp *qp) +{
+#if (OPENSSL_VERSION_NUMBER >=3D 0x30000000L && OPENSSL_VERSION= _NUMBER < 0x30003000L)
+       /* For OpenSSL versions 3.0.0 <=3D= v < 3.0.3, re-initing of
+        * EVP_MAC_CTXs is broken, and d= oesn't actually reset their
+        * state. This was fixed in OSSL= commit c9ddc5af5199 ("Avoid
+        * undefined behavior of provide= d macs on EVP_MAC
+        * reinitialization"). In c= ases where the fix is not present,
+        * fall back to duplicating the = context every buffer as a
+        * workaround, at the cost of pe= rformance.
+        */
+       RTE_SET_USED(qp);
+       return EVP_MAC_CTX_dup(sess->auth.= hmac.ctx);
+#else
+       if (sess->ctx_copies_len =3D=3D 0)=
+            &n= bsp;  return sess->auth.hmac.ctx;
+
+#if OPENSSL_VERSION_NUMBER >=3D 0x30000000L
+       EVP_MAC_CTX **lctx =3D
+#else
+       HMAC_CTX **lctx =3D
+#endif
+            &n= bsp;  &sess->qp_ctx[qp->id].hmac;
+
+       if (unlikely(*lctx =3D=3D NULL)) { +#if OPENSSL_VERSION_NUMBER >=3D 0x30000000L
+            &n= bsp;  *lctx =3D EVP_MAC_CTX_dup(sess->auth.hmac.ctx);
+#else
+            &n= bsp;  *lctx =3D HMAC_CTX_new();
+            &n= bsp;  HMAC_CTX_copy(*lctx, sess->auth.hmac.ctx);
+#endif
+       }
+
+       return *lctx;
+#endif
+}
+
+#if OPENSSL_VERSION_NUMBER >=3D 0x30000000L
+static inline EVP_MAC_CTX *
+#else
+static inline CMAC_CTX *
+#endif
+get_local_cmac_ctx(struct openssl_session *sess, struct openssl_qp *qp) +{
+#if (OPENSSL_VERSION_NUMBER >=3D 0x30000000L && OPENSSL_VERSION= _NUMBER < 0x30003000L)
+       /* For OpenSSL versions 3.0.0 <=3D= v < 3.0.3, re-initing of
+        * EVP_MAC_CTXs is broken, and d= oesn't actually reset their
+        * state. This was fixed in OSSL= commit c9ddc5af5199 ("Avoid
+        * undefined behavior of provide= d macs on EVP_MAC
+        * reinitialization"). In c= ases where the fix is not present,
+        * fall back to duplicating the = context every buffer as a
+        * workaround, at the cost of pe= rformance.
+        */
+       RTE_SET_USED(qp);
+       return EVP_MAC_CTX_dup(sess->auth.= cmac.ctx);
+#else
+       if (sess->ctx_copies_len =3D=3D 0)=
+            &n= bsp;  return sess->auth.cmac.ctx;
+
+#if OPENSSL_VERSION_NUMBER >=3D 0x30000000L
+       EVP_MAC_CTX **lctx =3D
+#else
+       CMAC_CTX **lctx =3D
+#endif
+            &n= bsp;  &sess->qp_ctx[qp->id].cmac;
+
+       if (unlikely(*lctx =3D=3D NULL)) { +#if OPENSSL_VERSION_NUMBER >=3D 0x30000000L
+            &n= bsp;  *lctx =3D EVP_MAC_CTX_dup(sess->auth.cmac.ctx);
+#else
+            &n= bsp;  *lctx =3D CMAC_CTX_new();
+            &n= bsp;  CMAC_CTX_copy(*lctx, sess->auth.cmac.ctx);
+#endif
+       }
+
+       return *lctx;
+#endif
+}
+
 /** Process auth/cipher combined operation */
 static void
 process_openssl_combined_op(struct openssl_qp *qp, struct rte_crypto_= op *op,
@@ -1895,42 +2007,40 @@ process_openssl_auth_op(struct openssl_qp *qp, stru= ct rte_crypto_op *op,
 
         switch (sess->auth.mode= ) {
         case OPENSSL_AUTH_AS_AUTH:=
-            &n= bsp;  ctx_a =3D EVP_MD_CTX_create();
-            &n= bsp;  EVP_MD_CTX_copy_ex(ctx_a, sess->auth.auth.ctx);
+            &n= bsp;  ctx_a =3D get_local_auth_ctx(sess, qp);
            &nb= sp;    status =3D process_openssl_auth(mbuf_src, dst,
            &nb= sp;            =         op->sym->auth.data.offset,= NULL, NULL, srclen,
            &nb= sp;            =         ctx_a, sess->auth.auth.evp_al= go);
-            &n= bsp;  EVP_MD_CTX_destroy(ctx_a);
            &nb= sp;    break;
         case OPENSSL_AUTH_AS_HMAC:=
+            &n= bsp;  ctx_h =3D get_local_hmac_ctx(sess, qp);
 # if OPENSSL_VERSION_NUMBER >=3D 0x30000000L
-            &n= bsp;  ctx_h =3D EVP_MAC_CTX_dup(sess->auth.hmac.ctx);
            &nb= sp;    status =3D process_openssl_auth_mac(mbuf_src, dst,             &nb= sp;            =         op->sym->auth.data.offset,= srclen,
            &nb= sp;            =         ctx_h);
 # else
-            &n= bsp;  ctx_h =3D HMAC_CTX_new();
-            &n= bsp;  HMAC_CTX_copy(ctx_h, sess->auth.hmac.ctx);
            &nb= sp;    status =3D process_openssl_auth_hmac(mbuf_src, dst,             &nb= sp;            =         op->sym->auth.data.offset,= srclen,
            &nb= sp;            =         ctx_h);
-            &n= bsp;  HMAC_CTX_free(ctx_h);
 # endif
+#if (OPENSSL_VERSION_NUMBER >=3D 0x30000000L && OPENSSL_VERSION= _NUMBER < 0x30003000L)
+            &n= bsp;  EVP_MAC_CTX_free(ctx_h);
+#endif
            &nb= sp;    break;
         case OPENSSL_AUTH_AS_CMAC:=
+            &n= bsp;  ctx_c =3D get_local_cmac_ctx(sess, qp);
 # if OPENSSL_VERSION_NUMBER >=3D 0x30000000L
-            &n= bsp;  ctx_c =3D EVP_MAC_CTX_dup(sess->auth.cmac.ctx);
            &nb= sp;    status =3D process_openssl_auth_mac(mbuf_src, dst,             &nb= sp;            =         op->sym->auth.data.offset,= srclen,
            &nb= sp;            =         ctx_c);
 # else
-            &n= bsp;  ctx_c =3D CMAC_CTX_new();
-            &n= bsp;  CMAC_CTX_copy(ctx_c, sess->auth.cmac.ctx);
            &nb= sp;    status =3D process_openssl_auth_cmac(mbuf_src, dst,             &nb= sp;            =         op->sym->auth.data.offset,= srclen,
            &nb= sp;            =         ctx_c);
-            &n= bsp;  CMAC_CTX_free(ctx_c);
 # endif
+#if (OPENSSL_VERSION_NUMBER >=3D 0x30000000L && OPENSSL_VERSION= _NUMBER < 0x30003000L)
+            &n= bsp;  EVP_MAC_CTX_free(ctx_c);
+#endif
            &nb= sp;    break;
         default:
            &nb= sp;    status =3D -1;
diff --git a/drivers/crypto/openssl/rte_openssl_pmd_ops.c b/drivers/crypto/= openssl/rte_openssl_pmd_ops.c
index 4209c6ab6f..1bbb855a59 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd_ops.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd_ops.c
@@ -805,7 +805,7 @@ openssl_pmd_sym_session_get_size(struct rte_cryptodev *= dev)
            &nb= sp;    unsigned int max_nb_qps =3D ((struct openssl_private = *)
            &nb= sp;            =         dev->data->dev_private)-&g= t;max_nb_qpairs;
            &nb= sp;    return sizeof(struct openssl_session) +
-            &n= bsp;            = ;      (sizeof(void *) * max_nb_qps);
+            &n= bsp;            = ;      (sizeof(struct evp_ctx_pair) * max_nb_qps);=
         }
 
         /*
@@ -818,10 +818,11 @@ openssl_pmd_sym_session_get_size(struct rte_cryptodev= *dev)
 
         /*
          * Otherwise, the siz= e of the flexible array member should be enough to
-        * fit pointers to per-qp contex= ts.
+        * fit pointers to per-qp contex= ts. This is twice the number of queue
+        * pairs, to allow for auth and = cipher contexts.
          */
         return sizeof(struct opens= sl_session) +
-            &n= bsp;  (sizeof(void *) * dev->data->nb_queue_pairs);
+            &n= bsp;  (sizeof(struct evp_ctx_pair) * dev->data->nb_queue_pairs);=
 }
 
 /** Returns the size of the asymmetric session structure */
--
2.34.1

--_000_DS0PR11MB7458E2235E4EA742C4B4DB6181D32DS0PR11MB7458namp_--