patches for DPDK stable branches
 help / color / mirror / Atom feed
* [PATCH 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs
       [not found] <20240603160119.1279476-1-jack.bond-preston@foss.arm.com>
@ 2024-06-03 16:01 ` Jack Bond-Preston
       [not found] ` <20240603184348.1310331-1-jack.bond-preston@foss.arm.com>
                   ` (4 subsequent siblings)
  5 siblings, 0 replies; 12+ messages in thread
From: Jack Bond-Preston @ 2024-06-03 16:01 UTC (permalink / raw)
  To: Kai Ji, Fan Zhang, Akhil Goyal; +Cc: dev, stable, Wathsala Vithanage

Commit 67ab783b5d70 ("crypto/openssl: use local copy for session
contexts") introduced a fix for concurrency bugs which could occur when
using one OpenSSL PMD session across multiple cores simultaneously. The
solution was to clone the EVP contexts per-buffer to avoid them being
used concurrently.

However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC
routine with 3.0 EVP API") reverted this fix, only for combined ops
(AES-GCM and AES-CCM), with no explanation. This commit fixes the issue
again, essentially reverting this part of the commit.

Throughput performance uplift measurements for AES-GCM-128 encrypt on
Ampere Altra Max platform:
1 worker lcore
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |          2.60 |               1.31 |   -49.5% |
|             256 |          7.69 |               4.45 |   -42.1% |
|            1024 |         15.33 |              11.30 |   -26.3% |
|            2048 |         18.74 |              15.37 |   -18.0% |
|            4096 |         21.11 |              18.80 |   -10.9% |

8 worker lcores
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |         19.94 |               2.83 |   -85.8% |
|             256 |         58.84 |              11.00 |   -81.3% |
|            1024 |        119.71 |              42.46 |   -64.5% |
|            2048 |        147.69 |              80.91 |   -45.2% |
|            4096 |        167.39 |             121.25 |   -27.6% |

Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API")
Cc: stable@dpdk.org
Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
---
 drivers/crypto/openssl/rte_openssl_pmd.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index e8cb09defc..ca7ed30ec4 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -1590,6 +1590,9 @@ process_openssl_combined_op
 		return;
 	}
 
+	EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
+	EVP_CIPHER_CTX_copy(ctx, sess->cipher.ctx);
+
 	iv = rte_crypto_op_ctod_offset(op, uint8_t *,
 			sess->iv.offset);
 	if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) {
@@ -1623,12 +1626,12 @@ process_openssl_combined_op
 			status = process_openssl_auth_encryption_gcm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, sess->cipher.ctx);
+					dst, tag, ctx);
 		else
 			status = process_openssl_auth_encryption_ccm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, taglen, sess->cipher.ctx);
+					dst, tag, taglen, ctx);
 
 	} else {
 		if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC ||
@@ -1636,14 +1639,16 @@ process_openssl_combined_op
 			status = process_openssl_auth_decryption_gcm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, sess->cipher.ctx);
+					dst, tag, ctx);
 		else
 			status = process_openssl_auth_decryption_ccm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, taglen, sess->cipher.ctx);
+					dst, tag, taglen, ctx);
 	}
 
+	EVP_CIPHER_CTX_free(ctx);
+
 	if (status != 0) {
 		if (status == (-EFAULT) &&
 				sess->auth.operation ==
-- 
2.34.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v2 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs
       [not found] ` <20240603184348.1310331-1-jack.bond-preston@foss.arm.com>
@ 2024-06-03 18:43   ` Jack Bond-Preston
  0 siblings, 0 replies; 12+ messages in thread
From: Jack Bond-Preston @ 2024-06-03 18:43 UTC (permalink / raw)
  To: Kai Ji, Fan Zhang, Akhil Goyal; +Cc: dev, stable, Wathsala Vithanage

Commit 67ab783b5d70 ("crypto/openssl: use local copy for session
contexts") introduced a fix for concurrency bugs which could occur when
using one OpenSSL PMD session across multiple cores simultaneously. The
solution was to clone the EVP contexts per-buffer to avoid them being
used concurrently.

However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC
routine with 3.0 EVP API") reverted this fix, only for combined ops
(AES-GCM and AES-CCM), with no explanation. This commit fixes the issue
again, essentially reverting this part of the commit.

Throughput performance uplift measurements for AES-GCM-128 encrypt on
Ampere Altra Max platform:
1 worker lcore
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |          2.60 |               1.31 |   -49.5% |
|             256 |          7.69 |               4.45 |   -42.1% |
|            1024 |         15.33 |              11.30 |   -26.3% |
|            2048 |         18.74 |              15.37 |   -18.0% |
|            4096 |         21.11 |              18.80 |   -10.9% |

8 worker lcores
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |         19.94 |               2.83 |   -85.8% |
|             256 |         58.84 |              11.00 |   -81.3% |
|            1024 |        119.71 |              42.46 |   -64.5% |
|            2048 |        147.69 |              80.91 |   -45.2% |
|            4096 |        167.39 |             121.25 |   -27.6% |

Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API")
Cc: stable@dpdk.org
Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
---
 drivers/crypto/openssl/rte_openssl_pmd.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index e8cb09defc..ca7ed30ec4 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -1590,6 +1590,9 @@ process_openssl_combined_op
 		return;
 	}
 
+	EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
+	EVP_CIPHER_CTX_copy(ctx, sess->cipher.ctx);
+
 	iv = rte_crypto_op_ctod_offset(op, uint8_t *,
 			sess->iv.offset);
 	if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) {
@@ -1623,12 +1626,12 @@ process_openssl_combined_op
 			status = process_openssl_auth_encryption_gcm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, sess->cipher.ctx);
+					dst, tag, ctx);
 		else
 			status = process_openssl_auth_encryption_ccm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, taglen, sess->cipher.ctx);
+					dst, tag, taglen, ctx);
 
 	} else {
 		if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC ||
@@ -1636,14 +1639,16 @@ process_openssl_combined_op
 			status = process_openssl_auth_decryption_gcm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, sess->cipher.ctx);
+					dst, tag, ctx);
 		else
 			status = process_openssl_auth_decryption_ccm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, taglen, sess->cipher.ctx);
+					dst, tag, taglen, ctx);
 	}
 
+	EVP_CIPHER_CTX_free(ctx);
+
 	if (status != 0) {
 		if (status == (-EFAULT) &&
 				sess->auth.operation ==
-- 
2.34.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v2 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs
       [not found] ` <20240603185939.1312680-1-jack.bond-preston@foss.arm.com>
@ 2024-06-03 18:59   ` Jack Bond-Preston
  0 siblings, 0 replies; 12+ messages in thread
From: Jack Bond-Preston @ 2024-06-03 18:59 UTC (permalink / raw)
  To: Kai Ji, Fan Zhang, Akhil Goyal; +Cc: dev, stable, Wathsala Vithanage

Commit 67ab783b5d70 ("crypto/openssl: use local copy for session
contexts") introduced a fix for concurrency bugs which could occur when
using one OpenSSL PMD session across multiple cores simultaneously. The
solution was to clone the EVP contexts per-buffer to avoid them being
used concurrently.

However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC
routine with 3.0 EVP API") reverted this fix, only for combined ops
(AES-GCM and AES-CCM), with no explanation. This commit fixes the issue
again, essentially reverting this part of the commit.

Throughput performance uplift measurements for AES-GCM-128 encrypt on
Ampere Altra Max platform:
1 worker lcore
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |          2.60 |               1.31 |   -49.5% |
|             256 |          7.69 |               4.45 |   -42.1% |
|            1024 |         15.33 |              11.30 |   -26.3% |
|            2048 |         18.74 |              15.37 |   -18.0% |
|            4096 |         21.11 |              18.80 |   -10.9% |

8 worker lcores
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |         19.94 |               2.83 |   -85.8% |
|             256 |         58.84 |              11.00 |   -81.3% |
|            1024 |        119.71 |              42.46 |   -64.5% |
|            2048 |        147.69 |              80.91 |   -45.2% |
|            4096 |        167.39 |             121.25 |   -27.6% |

Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API")
Cc: stable@dpdk.org
Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
---
 drivers/crypto/openssl/rte_openssl_pmd.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index e8cb09defc..ca7ed30ec4 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -1590,6 +1590,9 @@ process_openssl_combined_op
 		return;
 	}
 
+	EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
+	EVP_CIPHER_CTX_copy(ctx, sess->cipher.ctx);
+
 	iv = rte_crypto_op_ctod_offset(op, uint8_t *,
 			sess->iv.offset);
 	if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) {
@@ -1623,12 +1626,12 @@ process_openssl_combined_op
 			status = process_openssl_auth_encryption_gcm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, sess->cipher.ctx);
+					dst, tag, ctx);
 		else
 			status = process_openssl_auth_encryption_ccm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, taglen, sess->cipher.ctx);
+					dst, tag, taglen, ctx);
 
 	} else {
 		if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC ||
@@ -1636,14 +1639,16 @@ process_openssl_combined_op
 			status = process_openssl_auth_decryption_gcm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, sess->cipher.ctx);
+					dst, tag, ctx);
 		else
 			status = process_openssl_auth_decryption_ccm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, taglen, sess->cipher.ctx);
+					dst, tag, taglen, ctx);
 	}
 
+	EVP_CIPHER_CTX_free(ctx);
+
 	if (status != 0) {
 		if (status == (-EFAULT) &&
 				sess->auth.operation ==
-- 
2.34.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v3 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs
       [not found] ` <20240606102043.2926695-1-jack.bond-preston@foss.arm.com>
@ 2024-06-06 10:20   ` Jack Bond-Preston
  2024-06-06 10:44     ` [EXTERNAL] " Akhil Goyal
  0 siblings, 1 reply; 12+ messages in thread
From: Jack Bond-Preston @ 2024-06-06 10:20 UTC (permalink / raw)
  To: Kai Ji, Akhil Goyal, Fan Zhang; +Cc: dev, stable, Wathsala Vithanage

Commit 67ab783b5d70 ("crypto/openssl: use local copy for session
contexts") introduced a fix for concurrency bugs which could occur when
using one OpenSSL PMD session across multiple cores simultaneously. The
solution was to clone the EVP contexts per-buffer to avoid them being
used concurrently.

However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC
routine with 3.0 EVP API") reverted this fix, only for combined ops
(AES-GCM and AES-CCM).

Fix the concurrency issue by cloning EVP contexts per-buffer. An extra
workaround is required for OpenSSL versions which are >= 3.0.0, and
<= 3.2.0. This is because, prior to OpenSSL 3.2.0, EVP_CIPHER_CTX_copy()
is not implemented for AES-GCM or AES-CCM. When using these OpenSSL
versions, create and initialise the context from scratch, per-buffer.

Throughput performance uplift measurements for AES-GCM-128 encrypt on
Ampere Altra Max platform:
1 worker lcore
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |          2.60 |               1.31 |   -49.5% |
|             256 |          7.69 |               4.45 |   -42.1% |
|            1024 |         15.33 |              11.30 |   -26.3% |
|            2048 |         18.74 |              15.37 |   -18.0% |
|            4096 |         21.11 |              18.80 |   -10.9% |

8 worker lcores
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |         19.94 |               2.83 |   -85.8% |
|             256 |         58.84 |              11.00 |   -81.3% |
|            1024 |        119.71 |              42.46 |   -64.5% |
|            2048 |        147.69 |              80.91 |   -45.2% |
|            4096 |        167.39 |             121.25 |   -27.6% |

Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API")
Cc: stable@dpdk.org
Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
---
 drivers/crypto/openssl/rte_openssl_pmd.c | 84 ++++++++++++++++++------
 1 file changed, 64 insertions(+), 20 deletions(-)

diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index e8cb09defc..3f7f4d8c37 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -350,7 +350,8 @@ get_aead_algo(enum rte_crypto_aead_algorithm sess_algo, size_t keylen,
 static int
 openssl_set_sess_aead_enc_param(struct openssl_session *sess,
 		enum rte_crypto_aead_algorithm algo,
-		uint8_t tag_len, const uint8_t *key)
+		uint8_t tag_len, const uint8_t *key,
+		EVP_CIPHER_CTX **ctx)
 {
 	int iv_type = 0;
 	unsigned int do_ccm;
@@ -378,7 +379,7 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess,
 	}
 
 	sess->cipher.mode = OPENSSL_CIPHER_LIB;
-	sess->cipher.ctx = EVP_CIPHER_CTX_new();
+	*ctx = EVP_CIPHER_CTX_new();
 
 	if (get_aead_algo(algo, sess->cipher.key.length,
 			&sess->cipher.evp_algo) != 0)
@@ -388,19 +389,19 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess,
 
 	sess->chain_order = OPENSSL_CHAIN_COMBINED;
 
-	if (EVP_EncryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo,
+	if (EVP_EncryptInit_ex(*ctx, sess->cipher.evp_algo,
 			NULL, NULL, NULL) <= 0)
 		return -EINVAL;
 
-	if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type, sess->iv.length,
+	if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type, sess->iv.length,
 			NULL) <= 0)
 		return -EINVAL;
 
 	if (do_ccm)
-		EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG,
+		EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG,
 				tag_len, NULL);
 
-	if (EVP_EncryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0)
+	if (EVP_EncryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0)
 		return -EINVAL;
 
 	return 0;
@@ -410,7 +411,8 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess,
 static int
 openssl_set_sess_aead_dec_param(struct openssl_session *sess,
 		enum rte_crypto_aead_algorithm algo,
-		uint8_t tag_len, const uint8_t *key)
+		uint8_t tag_len, const uint8_t *key,
+		EVP_CIPHER_CTX **ctx)
 {
 	int iv_type = 0;
 	unsigned int do_ccm = 0;
@@ -437,7 +439,7 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess,
 	}
 
 	sess->cipher.mode = OPENSSL_CIPHER_LIB;
-	sess->cipher.ctx = EVP_CIPHER_CTX_new();
+	*ctx = EVP_CIPHER_CTX_new();
 
 	if (get_aead_algo(algo, sess->cipher.key.length,
 			&sess->cipher.evp_algo) != 0)
@@ -447,24 +449,54 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess,
 
 	sess->chain_order = OPENSSL_CHAIN_COMBINED;
 
-	if (EVP_DecryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo,
+	if (EVP_DecryptInit_ex(*ctx, sess->cipher.evp_algo,
 			NULL, NULL, NULL) <= 0)
 		return -EINVAL;
 
-	if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type,
+	if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type,
 			sess->iv.length, NULL) <= 0)
 		return -EINVAL;
 
 	if (do_ccm)
-		EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG,
+		EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG,
 				tag_len, NULL);
 
-	if (EVP_DecryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0)
+	if (EVP_DecryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0)
 		return -EINVAL;
 
 	return 0;
 }
 
+static int openssl_aesni_ctx_clone(EVP_CIPHER_CTX **dest,
+		struct openssl_session *sess)
+{
+#if (OPENSSL_VERSION_NUMBER > 0x30200000L)
+	*dest = EVP_CIPHER_CTX_dup(sess->ctx);
+	return 0;
+#elif (OPENSSL_VERSION_NUMBER > 0x30000000L)
+	/* OpenSSL versions 3.0.0 <= V < 3.2.0 have no dupctx() implementation
+	 * for AES-GCM and AES-CCM. In this case, we have to create new empty
+	 * contexts and initialise, as we did the original context.
+	 */
+	if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC)
+		sess->aead_algo = RTE_CRYPTO_AEAD_AES_GCM;
+
+	if (sess->cipher.direction == RTE_CRYPTO_CIPHER_OP_ENCRYPT)
+		return openssl_set_sess_aead_enc_param(sess, sess->aead_algo,
+				sess->auth.digest_length, sess->cipher.key.data,
+				dest);
+	else
+		return openssl_set_sess_aead_dec_param(sess, sess->aead_algo,
+				sess->auth.digest_length, sess->cipher.key.data,
+				dest);
+#else
+	*dest = EVP_CIPHER_CTX_new();
+	if (EVP_CIPHER_CTX_copy(*dest, sess->cipher.ctx) != 1)
+		return -EINVAL;
+	return 0;
+#endif
+}
+
 /** Set session cipher parameters */
 static int
 openssl_set_session_cipher_parameters(struct openssl_session *sess,
@@ -623,12 +655,14 @@ openssl_set_session_auth_parameters(struct openssl_session *sess,
 			return openssl_set_sess_aead_enc_param(sess,
 						RTE_CRYPTO_AEAD_AES_GCM,
 						xform->auth.digest_length,
-						xform->auth.key.data);
+						xform->auth.key.data,
+						&sess->cipher.ctx);
 		else
 			return openssl_set_sess_aead_dec_param(sess,
 						RTE_CRYPTO_AEAD_AES_GCM,
 						xform->auth.digest_length,
-						xform->auth.key.data);
+						xform->auth.key.data,
+						&sess->cipher.ctx);
 		break;
 
 	case RTE_CRYPTO_AUTH_MD5:
@@ -770,10 +804,12 @@ openssl_set_session_aead_parameters(struct openssl_session *sess,
 	/* Select cipher direction */
 	if (xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT)
 		return openssl_set_sess_aead_enc_param(sess, xform->aead.algo,
-				xform->aead.digest_length, xform->aead.key.data);
+				xform->aead.digest_length, xform->aead.key.data,
+				&sess->cipher.ctx);
 	else
 		return openssl_set_sess_aead_dec_param(sess, xform->aead.algo,
-				xform->aead.digest_length, xform->aead.key.data);
+				xform->aead.digest_length, xform->aead.key.data,
+				&sess->cipher.ctx);
 }
 
 /** Parse crypto xform chain and set private session parameters */
@@ -1590,6 +1626,12 @@ process_openssl_combined_op
 		return;
 	}
 
+	EVP_CIPHER_CTX *ctx;
+	if (openssl_aesni_ctx_clone(&ctx, sess) != 0) {
+		op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+		return;
+	}
+
 	iv = rte_crypto_op_ctod_offset(op, uint8_t *,
 			sess->iv.offset);
 	if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) {
@@ -1623,12 +1665,12 @@ process_openssl_combined_op
 			status = process_openssl_auth_encryption_gcm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, sess->cipher.ctx);
+					dst, tag, ctx);
 		else
 			status = process_openssl_auth_encryption_ccm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, taglen, sess->cipher.ctx);
+					dst, tag, taglen, ctx);
 
 	} else {
 		if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC ||
@@ -1636,14 +1678,16 @@ process_openssl_combined_op
 			status = process_openssl_auth_decryption_gcm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, sess->cipher.ctx);
+					dst, tag, ctx);
 		else
 			status = process_openssl_auth_decryption_ccm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, taglen, sess->cipher.ctx);
+					dst, tag, taglen, ctx);
 	}
 
+	EVP_CIPHER_CTX_free(ctx);
+
 	if (status != 0) {
 		if (status == (-EFAULT) &&
 				sess->auth.operation ==
-- 
2.34.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* RE: [EXTERNAL] [PATCH v3 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs
  2024-06-06 10:20   ` [PATCH v3 " Jack Bond-Preston
@ 2024-06-06 10:44     ` Akhil Goyal
  0 siblings, 0 replies; 12+ messages in thread
From: Akhil Goyal @ 2024-06-06 10:44 UTC (permalink / raw)
  To: Jack Bond-Preston, Kai Ji, Fan Zhang; +Cc: dev, stable, Wathsala Vithanage

> Subject: [EXTERNAL] [PATCH v3 1/5] crypto/openssl: fix GCM and CCM thread
> unsafe ctxs
> 
> Commit 67ab783b5d70 ("crypto/openssl: use local copy for session
> contexts") introduced a fix for concurrency bugs which could occur when
> using one OpenSSL PMD session across multiple cores simultaneously. The
> solution was to clone the EVP contexts per-buffer to avoid them being
> used concurrently.
> 
> However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC
> routine with 3.0 EVP API") reverted this fix, only for combined ops
> (AES-GCM and AES-CCM).
> 
> Fix the concurrency issue by cloning EVP contexts per-buffer. An extra
> workaround is required for OpenSSL versions which are >= 3.0.0, and
> <= 3.2.0. This is because, prior to OpenSSL 3.2.0, EVP_CIPHER_CTX_copy()
> is not implemented for AES-GCM or AES-CCM. When using these OpenSSL
> versions, create and initialise the context from scratch, per-buffer.
> 
> Throughput performance uplift measurements for AES-GCM-128 encrypt on
> Ampere Altra Max platform:
> 1 worker lcore
> |   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
> |-----------------+---------------+--------------------+----------|
> |              64 |          2.60 |               1.31 |   -49.5% |
> |             256 |          7.69 |               4.45 |   -42.1% |
> |            1024 |         15.33 |              11.30 |   -26.3% |
> |            2048 |         18.74 |              15.37 |   -18.0% |
> |            4096 |         21.11 |              18.80 |   -10.9% |
> 
> 8 worker lcores
> |   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
> |-----------------+---------------+--------------------+----------|
> |              64 |         19.94 |               2.83 |   -85.8% |
> |             256 |         58.84 |              11.00 |   -81.3% |
> |            1024 |        119.71 |              42.46 |   -64.5% |
> |            2048 |        147.69 |              80.91 |   -45.2% |
> |            4096 |        167.39 |             121.25 |   -27.6% |
> 
> Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API")
> Cc: stable@dpdk.org
> Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
> Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
Hi Kai,

Please review these optimizations.

Regards,
Akhil

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v4 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs
       [not found] ` <20240607124756.3968704-1-jack.bond-preston@foss.arm.com>
@ 2024-06-07 12:47   ` Jack Bond-Preston
  2024-07-01 12:55     ` Ji, Kai
  2024-07-02 15:39     ` [EXTERNAL] " Akhil Goyal
  0 siblings, 2 replies; 12+ messages in thread
From: Jack Bond-Preston @ 2024-06-07 12:47 UTC (permalink / raw)
  To: Kai Ji, Fan Zhang, Akhil Goyal; +Cc: dev, stable, Wathsala Vithanage

Commit 67ab783b5d70 ("crypto/openssl: use local copy for session
contexts") introduced a fix for concurrency bugs which could occur when
using one OpenSSL PMD session across multiple cores simultaneously. The
solution was to clone the EVP contexts per-buffer to avoid them being
used concurrently.

However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC
routine with 3.0 EVP API") reverted this fix, only for combined ops
(AES-GCM and AES-CCM).

Fix the concurrency issue by cloning EVP contexts per-buffer. An extra
workaround is required for OpenSSL versions which are >= 3.0.0, and
<= 3.2.0. This is because, prior to OpenSSL 3.2.0, EVP_CIPHER_CTX_copy()
is not implemented for AES-GCM or AES-CCM. When using these OpenSSL
versions, create and initialise the context from scratch, per-buffer.

Throughput performance uplift measurements for AES-GCM-128 encrypt on
Ampere Altra Max platform:
1 worker lcore
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |          2.60 |               1.31 |   -49.5% |
|             256 |          7.69 |               4.45 |   -42.1% |
|            1024 |         15.33 |              11.30 |   -26.3% |
|            2048 |         18.74 |              15.37 |   -18.0% |
|            4096 |         21.11 |              18.80 |   -10.9% |

8 worker lcores
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |         19.94 |               2.83 |   -85.8% |
|             256 |         58.84 |              11.00 |   -81.3% |
|            1024 |        119.71 |              42.46 |   -64.5% |
|            2048 |        147.69 |              80.91 |   -45.2% |
|            4096 |        167.39 |             121.25 |   -27.6% |

Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API")
Cc: stable@dpdk.org
Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
---
 drivers/crypto/openssl/rte_openssl_pmd.c | 84 ++++++++++++++++++------
 1 file changed, 64 insertions(+), 20 deletions(-)

diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index e8cb09defc..3f7f4d8c37 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -350,7 +350,8 @@ get_aead_algo(enum rte_crypto_aead_algorithm sess_algo, size_t keylen,
 static int
 openssl_set_sess_aead_enc_param(struct openssl_session *sess,
 		enum rte_crypto_aead_algorithm algo,
-		uint8_t tag_len, const uint8_t *key)
+		uint8_t tag_len, const uint8_t *key,
+		EVP_CIPHER_CTX **ctx)
 {
 	int iv_type = 0;
 	unsigned int do_ccm;
@@ -378,7 +379,7 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess,
 	}
 
 	sess->cipher.mode = OPENSSL_CIPHER_LIB;
-	sess->cipher.ctx = EVP_CIPHER_CTX_new();
+	*ctx = EVP_CIPHER_CTX_new();
 
 	if (get_aead_algo(algo, sess->cipher.key.length,
 			&sess->cipher.evp_algo) != 0)
@@ -388,19 +389,19 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess,
 
 	sess->chain_order = OPENSSL_CHAIN_COMBINED;
 
-	if (EVP_EncryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo,
+	if (EVP_EncryptInit_ex(*ctx, sess->cipher.evp_algo,
 			NULL, NULL, NULL) <= 0)
 		return -EINVAL;
 
-	if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type, sess->iv.length,
+	if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type, sess->iv.length,
 			NULL) <= 0)
 		return -EINVAL;
 
 	if (do_ccm)
-		EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG,
+		EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG,
 				tag_len, NULL);
 
-	if (EVP_EncryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0)
+	if (EVP_EncryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0)
 		return -EINVAL;
 
 	return 0;
@@ -410,7 +411,8 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess,
 static int
 openssl_set_sess_aead_dec_param(struct openssl_session *sess,
 		enum rte_crypto_aead_algorithm algo,
-		uint8_t tag_len, const uint8_t *key)
+		uint8_t tag_len, const uint8_t *key,
+		EVP_CIPHER_CTX **ctx)
 {
 	int iv_type = 0;
 	unsigned int do_ccm = 0;
@@ -437,7 +439,7 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess,
 	}
 
 	sess->cipher.mode = OPENSSL_CIPHER_LIB;
-	sess->cipher.ctx = EVP_CIPHER_CTX_new();
+	*ctx = EVP_CIPHER_CTX_new();
 
 	if (get_aead_algo(algo, sess->cipher.key.length,
 			&sess->cipher.evp_algo) != 0)
@@ -447,24 +449,54 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess,
 
 	sess->chain_order = OPENSSL_CHAIN_COMBINED;
 
-	if (EVP_DecryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo,
+	if (EVP_DecryptInit_ex(*ctx, sess->cipher.evp_algo,
 			NULL, NULL, NULL) <= 0)
 		return -EINVAL;
 
-	if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type,
+	if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type,
 			sess->iv.length, NULL) <= 0)
 		return -EINVAL;
 
 	if (do_ccm)
-		EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG,
+		EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG,
 				tag_len, NULL);
 
-	if (EVP_DecryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0)
+	if (EVP_DecryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0)
 		return -EINVAL;
 
 	return 0;
 }
 
+static int openssl_aesni_ctx_clone(EVP_CIPHER_CTX **dest,
+		struct openssl_session *sess)
+{
+#if (OPENSSL_VERSION_NUMBER > 0x30200000L)
+	*dest = EVP_CIPHER_CTX_dup(sess->ctx);
+	return 0;
+#elif (OPENSSL_VERSION_NUMBER > 0x30000000L)
+	/* OpenSSL versions 3.0.0 <= V < 3.2.0 have no dupctx() implementation
+	 * for AES-GCM and AES-CCM. In this case, we have to create new empty
+	 * contexts and initialise, as we did the original context.
+	 */
+	if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC)
+		sess->aead_algo = RTE_CRYPTO_AEAD_AES_GCM;
+
+	if (sess->cipher.direction == RTE_CRYPTO_CIPHER_OP_ENCRYPT)
+		return openssl_set_sess_aead_enc_param(sess, sess->aead_algo,
+				sess->auth.digest_length, sess->cipher.key.data,
+				dest);
+	else
+		return openssl_set_sess_aead_dec_param(sess, sess->aead_algo,
+				sess->auth.digest_length, sess->cipher.key.data,
+				dest);
+#else
+	*dest = EVP_CIPHER_CTX_new();
+	if (EVP_CIPHER_CTX_copy(*dest, sess->cipher.ctx) != 1)
+		return -EINVAL;
+	return 0;
+#endif
+}
+
 /** Set session cipher parameters */
 static int
 openssl_set_session_cipher_parameters(struct openssl_session *sess,
@@ -623,12 +655,14 @@ openssl_set_session_auth_parameters(struct openssl_session *sess,
 			return openssl_set_sess_aead_enc_param(sess,
 						RTE_CRYPTO_AEAD_AES_GCM,
 						xform->auth.digest_length,
-						xform->auth.key.data);
+						xform->auth.key.data,
+						&sess->cipher.ctx);
 		else
 			return openssl_set_sess_aead_dec_param(sess,
 						RTE_CRYPTO_AEAD_AES_GCM,
 						xform->auth.digest_length,
-						xform->auth.key.data);
+						xform->auth.key.data,
+						&sess->cipher.ctx);
 		break;
 
 	case RTE_CRYPTO_AUTH_MD5:
@@ -770,10 +804,12 @@ openssl_set_session_aead_parameters(struct openssl_session *sess,
 	/* Select cipher direction */
 	if (xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT)
 		return openssl_set_sess_aead_enc_param(sess, xform->aead.algo,
-				xform->aead.digest_length, xform->aead.key.data);
+				xform->aead.digest_length, xform->aead.key.data,
+				&sess->cipher.ctx);
 	else
 		return openssl_set_sess_aead_dec_param(sess, xform->aead.algo,
-				xform->aead.digest_length, xform->aead.key.data);
+				xform->aead.digest_length, xform->aead.key.data,
+				&sess->cipher.ctx);
 }
 
 /** Parse crypto xform chain and set private session parameters */
@@ -1590,6 +1626,12 @@ process_openssl_combined_op
 		return;
 	}
 
+	EVP_CIPHER_CTX *ctx;
+	if (openssl_aesni_ctx_clone(&ctx, sess) != 0) {
+		op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+		return;
+	}
+
 	iv = rte_crypto_op_ctod_offset(op, uint8_t *,
 			sess->iv.offset);
 	if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) {
@@ -1623,12 +1665,12 @@ process_openssl_combined_op
 			status = process_openssl_auth_encryption_gcm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, sess->cipher.ctx);
+					dst, tag, ctx);
 		else
 			status = process_openssl_auth_encryption_ccm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, taglen, sess->cipher.ctx);
+					dst, tag, taglen, ctx);
 
 	} else {
 		if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC ||
@@ -1636,14 +1678,16 @@ process_openssl_combined_op
 			status = process_openssl_auth_decryption_gcm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, sess->cipher.ctx);
+					dst, tag, ctx);
 		else
 			status = process_openssl_auth_decryption_ccm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, taglen, sess->cipher.ctx);
+					dst, tag, taglen, ctx);
 	}
 
+	EVP_CIPHER_CTX_free(ctx);
+
 	if (status != 0) {
 		if (status == (-EFAULT) &&
 				sess->auth.operation ==
-- 
2.34.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH v4 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs
  2024-06-07 12:47   ` [PATCH v4 " Jack Bond-Preston
@ 2024-07-01 12:55     ` Ji, Kai
  2024-07-02 15:39     ` [EXTERNAL] " Akhil Goyal
  1 sibling, 0 replies; 12+ messages in thread
From: Ji, Kai @ 2024-07-01 12:55 UTC (permalink / raw)
  To: Jack Bond-Preston, Fan Zhang, Akhil Goyal; +Cc: dev, stable, Wathsala Vithanage

[-- Attachment #1: Type: text/plain, Size: 12073 bytes --]

Acked-by: Kai Ji <kai.ji@intel.com>

________________________________
From: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Sent: 07 June 2024 13:47
To: Ji, Kai <kai.ji@intel.com>; Fan Zhang <fanzhang.oss@gmail.com>; Akhil Goyal <gakhil@marvell.com>
Cc: dev@dpdk.org <dev@dpdk.org>; stable@dpdk.org <stable@dpdk.org>; Wathsala Vithanage <wathsala.vithanage@arm.com>
Subject: [PATCH v4 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs

Commit 67ab783b5d70 ("crypto/openssl: use local copy for session
contexts") introduced a fix for concurrency bugs which could occur when
using one OpenSSL PMD session across multiple cores simultaneously. The
solution was to clone the EVP contexts per-buffer to avoid them being
used concurrently.

However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC
routine with 3.0 EVP API") reverted this fix, only for combined ops
(AES-GCM and AES-CCM).

Fix the concurrency issue by cloning EVP contexts per-buffer. An extra
workaround is required for OpenSSL versions which are >= 3.0.0, and
<= 3.2.0. This is because, prior to OpenSSL 3.2.0, EVP_CIPHER_CTX_copy()
is not implemented for AES-GCM or AES-CCM. When using these OpenSSL
versions, create and initialise the context from scratch, per-buffer.

Throughput performance uplift measurements for AES-GCM-128 encrypt on
Ampere Altra Max platform:
1 worker lcore
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |          2.60 |               1.31 |   -49.5% |
|             256 |          7.69 |               4.45 |   -42.1% |
|            1024 |         15.33 |              11.30 |   -26.3% |
|            2048 |         18.74 |              15.37 |   -18.0% |
|            4096 |         21.11 |              18.80 |   -10.9% |

8 worker lcores
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |         19.94 |               2.83 |   -85.8% |
|             256 |         58.84 |              11.00 |   -81.3% |
|            1024 |        119.71 |              42.46 |   -64.5% |
|            2048 |        147.69 |              80.91 |   -45.2% |
|            4096 |        167.39 |             121.25 |   -27.6% |

Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API")
Cc: stable@dpdk.org
Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
---
 drivers/crypto/openssl/rte_openssl_pmd.c | 84 ++++++++++++++++++------
 1 file changed, 64 insertions(+), 20 deletions(-)

diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index e8cb09defc..3f7f4d8c37 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -350,7 +350,8 @@ get_aead_algo(enum rte_crypto_aead_algorithm sess_algo, size_t keylen,
 static int
 openssl_set_sess_aead_enc_param(struct openssl_session *sess,
                 enum rte_crypto_aead_algorithm algo,
-               uint8_t tag_len, const uint8_t *key)
+               uint8_t tag_len, const uint8_t *key,
+               EVP_CIPHER_CTX **ctx)
 {
         int iv_type = 0;
         unsigned int do_ccm;
@@ -378,7 +379,7 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess,
         }

         sess->cipher.mode = OPENSSL_CIPHER_LIB;
-       sess->cipher.ctx = EVP_CIPHER_CTX_new();
+       *ctx = EVP_CIPHER_CTX_new();

         if (get_aead_algo(algo, sess->cipher.key.length,
                         &sess->cipher.evp_algo) != 0)
@@ -388,19 +389,19 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess,

         sess->chain_order = OPENSSL_CHAIN_COMBINED;

-       if (EVP_EncryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo,
+       if (EVP_EncryptInit_ex(*ctx, sess->cipher.evp_algo,
                         NULL, NULL, NULL) <= 0)
                 return -EINVAL;

-       if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type, sess->iv.length,
+       if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type, sess->iv.length,
                         NULL) <= 0)
                 return -EINVAL;

         if (do_ccm)
-               EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG,
+               EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG,
                                 tag_len, NULL);

-       if (EVP_EncryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0)
+       if (EVP_EncryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0)
                 return -EINVAL;

         return 0;
@@ -410,7 +411,8 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess,
 static int
 openssl_set_sess_aead_dec_param(struct openssl_session *sess,
                 enum rte_crypto_aead_algorithm algo,
-               uint8_t tag_len, const uint8_t *key)
+               uint8_t tag_len, const uint8_t *key,
+               EVP_CIPHER_CTX **ctx)
 {
         int iv_type = 0;
         unsigned int do_ccm = 0;
@@ -437,7 +439,7 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess,
         }

         sess->cipher.mode = OPENSSL_CIPHER_LIB;
-       sess->cipher.ctx = EVP_CIPHER_CTX_new();
+       *ctx = EVP_CIPHER_CTX_new();

         if (get_aead_algo(algo, sess->cipher.key.length,
                         &sess->cipher.evp_algo) != 0)
@@ -447,24 +449,54 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess,

         sess->chain_order = OPENSSL_CHAIN_COMBINED;

-       if (EVP_DecryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo,
+       if (EVP_DecryptInit_ex(*ctx, sess->cipher.evp_algo,
                         NULL, NULL, NULL) <= 0)
                 return -EINVAL;

-       if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type,
+       if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type,
                         sess->iv.length, NULL) <= 0)
                 return -EINVAL;

         if (do_ccm)
-               EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG,
+               EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG,
                                 tag_len, NULL);

-       if (EVP_DecryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0)
+       if (EVP_DecryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0)
                 return -EINVAL;

         return 0;
 }

+static int openssl_aesni_ctx_clone(EVP_CIPHER_CTX **dest,
+               struct openssl_session *sess)
+{
+#if (OPENSSL_VERSION_NUMBER > 0x30200000L)
+       *dest = EVP_CIPHER_CTX_dup(sess->ctx);
+       return 0;
+#elif (OPENSSL_VERSION_NUMBER > 0x30000000L)
+       /* OpenSSL versions 3.0.0 <= V < 3.2.0 have no dupctx() implementation
+        * for AES-GCM and AES-CCM. In this case, we have to create new empty
+        * contexts and initialise, as we did the original context.
+        */
+       if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC)
+               sess->aead_algo = RTE_CRYPTO_AEAD_AES_GCM;
+
+       if (sess->cipher.direction == RTE_CRYPTO_CIPHER_OP_ENCRYPT)
+               return openssl_set_sess_aead_enc_param(sess, sess->aead_algo,
+                               sess->auth.digest_length, sess->cipher.key.data,
+                               dest);
+       else
+               return openssl_set_sess_aead_dec_param(sess, sess->aead_algo,
+                               sess->auth.digest_length, sess->cipher.key.data,
+                               dest);
+#else
+       *dest = EVP_CIPHER_CTX_new();
+       if (EVP_CIPHER_CTX_copy(*dest, sess->cipher.ctx) != 1)
+               return -EINVAL;
+       return 0;
+#endif
+}
+
 /** Set session cipher parameters */
 static int
 openssl_set_session_cipher_parameters(struct openssl_session *sess,
@@ -623,12 +655,14 @@ openssl_set_session_auth_parameters(struct openssl_session *sess,
                         return openssl_set_sess_aead_enc_param(sess,
                                                 RTE_CRYPTO_AEAD_AES_GCM,
                                                 xform->auth.digest_length,
-                                               xform->auth.key.data);
+                                               xform->auth.key.data,
+                                               &sess->cipher.ctx);
                 else
                         return openssl_set_sess_aead_dec_param(sess,
                                                 RTE_CRYPTO_AEAD_AES_GCM,
                                                 xform->auth.digest_length,
-                                               xform->auth.key.data);
+                                               xform->auth.key.data,
+                                               &sess->cipher.ctx);
                 break;

         case RTE_CRYPTO_AUTH_MD5:
@@ -770,10 +804,12 @@ openssl_set_session_aead_parameters(struct openssl_session *sess,
         /* Select cipher direction */
         if (xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT)
                 return openssl_set_sess_aead_enc_param(sess, xform->aead.algo,
-                               xform->aead.digest_length, xform->aead.key.data);
+                               xform->aead.digest_length, xform->aead.key.data,
+                               &sess->cipher.ctx);
         else
                 return openssl_set_sess_aead_dec_param(sess, xform->aead.algo,
-                               xform->aead.digest_length, xform->aead.key.data);
+                               xform->aead.digest_length, xform->aead.key.data,
+                               &sess->cipher.ctx);
 }

 /** Parse crypto xform chain and set private session parameters */
@@ -1590,6 +1626,12 @@ process_openssl_combined_op
                 return;
         }

+       EVP_CIPHER_CTX *ctx;
+       if (openssl_aesni_ctx_clone(&ctx, sess) != 0) {
+               op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+               return;
+       }
+
         iv = rte_crypto_op_ctod_offset(op, uint8_t *,
                         sess->iv.offset);
         if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) {
@@ -1623,12 +1665,12 @@ process_openssl_combined_op
                         status = process_openssl_auth_encryption_gcm(
                                         mbuf_src, offset, srclen,
                                         aad, aadlen, iv,
-                                       dst, tag, sess->cipher.ctx);
+                                       dst, tag, ctx);
                 else
                         status = process_openssl_auth_encryption_ccm(
                                         mbuf_src, offset, srclen,
                                         aad, aadlen, iv,
-                                       dst, tag, taglen, sess->cipher.ctx);
+                                       dst, tag, taglen, ctx);

         } else {
                 if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC ||
@@ -1636,14 +1678,16 @@ process_openssl_combined_op
                         status = process_openssl_auth_decryption_gcm(
                                         mbuf_src, offset, srclen,
                                         aad, aadlen, iv,
-                                       dst, tag, sess->cipher.ctx);
+                                       dst, tag, ctx);
                 else
                         status = process_openssl_auth_decryption_ccm(
                                         mbuf_src, offset, srclen,
                                         aad, aadlen, iv,
-                                       dst, tag, taglen, sess->cipher.ctx);
+                                       dst, tag, taglen, ctx);
         }

+       EVP_CIPHER_CTX_free(ctx);
+
         if (status != 0) {
                 if (status == (-EFAULT) &&
                                 sess->auth.operation ==
--
2.34.1


[-- Attachment #2: Type: text/html, Size: 29860 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* RE: [EXTERNAL] [PATCH v4 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs
  2024-06-07 12:47   ` [PATCH v4 " Jack Bond-Preston
  2024-07-01 12:55     ` Ji, Kai
@ 2024-07-02 15:39     ` Akhil Goyal
  2024-07-03 10:49       ` Jack Bond-Preston
  1 sibling, 1 reply; 12+ messages in thread
From: Akhil Goyal @ 2024-07-02 15:39 UTC (permalink / raw)
  To: Jack Bond-Preston, Kai Ji, Fan Zhang; +Cc: dev, stable, Wathsala Vithanage

> Commit 67ab783b5d70 ("crypto/openssl: use local copy for session
> contexts") introduced a fix for concurrency bugs which could occur when
> using one OpenSSL PMD session across multiple cores simultaneously. The
> solution was to clone the EVP contexts per-buffer to avoid them being
> used concurrently.
> 
> However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC
> routine with 3.0 EVP API") reverted this fix, only for combined ops
> (AES-GCM and AES-CCM).
> 
> Fix the concurrency issue by cloning EVP contexts per-buffer. An extra
> workaround is required for OpenSSL versions which are >= 3.0.0, and
> <= 3.2.0. This is because, prior to OpenSSL 3.2.0, EVP_CIPHER_CTX_copy()
> is not implemented for AES-GCM or AES-CCM. When using these OpenSSL
> versions, create and initialise the context from scratch, per-buffer.
> 
> Throughput performance uplift measurements for AES-GCM-128 encrypt on
> Ampere Altra Max platform:
> 1 worker lcore
> |   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
> |-----------------+---------------+--------------------+----------|
> |              64 |          2.60 |               1.31 |   -49.5% |
> |             256 |          7.69 |               4.45 |   -42.1% |
> |            1024 |         15.33 |              11.30 |   -26.3% |
> |            2048 |         18.74 |              15.37 |   -18.0% |
> |            4096 |         21.11 |              18.80 |   -10.9% |
> 
> 8 worker lcores
> |   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
> |-----------------+---------------+--------------------+----------|
> |              64 |         19.94 |               2.83 |   -85.8% |
> |             256 |         58.84 |              11.00 |   -81.3% |
> |            1024 |        119.71 |              42.46 |   -64.5% |
> |            2048 |        147.69 |              80.91 |   -45.2% |
> |            4096 |        167.39 |             121.25 |   -27.6% |
> 
> Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API")
> Cc: stable@dpdk.org
> Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
> Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com>


I am seeing below errors when it is compiled with openssl 3.0

[1/30] Compiling C object 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd_ops.c.o'.
FAILED: drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd_ops.c.o
ccache gcc -Idrivers/a715181@@tmp_rte_crypto_openssl@sta -Idrivers -I../drivers -Idrivers/crypto/openssl -I../drivers/crypto/openssl -Ilib/cryptodev -I../lib/cryptodev -I. -I../ -Iconfig -I../config -Ilib/eal/include -I../lib/eal/include -Ilib/eal/linux/include -I../lib/eal/linux/include -Ilib/eal/x86/include -I../lib/eal/x86/include -Ilib/eal/common -I../lib/eal/common -Ilib/eal -I../lib/eal -Ilib/kvargs -I../lib/kvargs -Ilib/log -I../lib/log -Ilib/telemetry/../metrics -I../lib/telemetry/../metrics -Ilib/telemetry -I../lib/telemetry -Ilib/mbuf -I../lib/mbuf -Ilib/mempool -I../lib/mempool -Ilib/ring -I../lib/ring -Ilib/rcu -I../lib/rcu -Idrivers/bus/vdev -I../drivers/bus/vdev -I/usr/local/include -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Werror -std=c11 -O2 -g -include rte_config.h -Wcast-qual -Wdeprecated -Wformat -Wformat-nonliteral -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -Wwrite-strings -Wno-address-of-packed-member -Wno-packed-not-aligned -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native -mrtm -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -Wno-format-truncation -DRTE_LOG_DEFAULT_LOGTYPE=pmd.crypto.openssl -MD -MQ 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd_ops.c.o' -MF 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd_ops.c.o.d' -o 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd_ops.c.o' -c ../drivers/crypto/openssl/rte_openssl_pmd_ops.c
In file included from ../drivers/crypto/openssl/rte_openssl_pmd_ops.c:12:
../drivers/crypto/openssl/compat.h: In function 'free_hmac_ctx':
../drivers/crypto/openssl/compat.h:24:2: error: 'HMAC_CTX_free' is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations]
   24 |  HMAC_CTX_free(ctx);
      |  ^~~~~~~~~~~~~
In file included from ../drivers/crypto/openssl/openssl_pmd_private.h:10,
                 from ../drivers/crypto/openssl/rte_openssl_pmd_ops.c:11:
/usr/local/include/openssl/hmac.h:35:28: note: declared here
   35 | OSSL_DEPRECATEDIN_3_0 void HMAC_CTX_free(HMAC_CTX *ctx);
      |                            ^~~~~~~~~~~~~
In file included from ../drivers/crypto/openssl/rte_openssl_pmd_ops.c:12:
../drivers/crypto/openssl/compat.h: In function 'free_cmac_ctx':
../drivers/crypto/openssl/compat.h:30:2: error: 'CMAC_CTX_free' is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations]
   30 |  CMAC_CTX_free(ctx);
      |  ^~~~~~~~~~~~~
In file included from ../drivers/crypto/openssl/openssl_pmd_private.h:9,
                 from ../drivers/crypto/openssl/rte_openssl_pmd_ops.c:11:
/usr/local/include/openssl/cmac.h:34:28: note: declared here
   34 | OSSL_DEPRECATEDIN_3_0 void CMAC_CTX_free(CMAC_CTX *ctx);
      |                            ^~~~~~~~~~~~~
cc1: all warnings being treated as errors
[2/30] Compiling C object 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd.c.o'.
FAILED: drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd.c.o
ccache gcc -Idrivers/a715181@@tmp_rte_crypto_openssl@sta -Idrivers -I../drivers -Idrivers/crypto/openssl -I../drivers/crypto/openssl -Ilib/cryptodev -I../lib/cryptodev -I. -I../ -Iconfig -I../config -Ilib/eal/include -I../lib/eal/include -Ilib/eal/linux/include -I../lib/eal/linux/include -Ilib/eal/x86/include -I../lib/eal/x86/include -Ilib/eal/common -I../lib/eal/common -Ilib/eal -I../lib/eal -Ilib/kvargs -I../lib/kvargs -Ilib/log -I../lib/log -Ilib/telemetry/../metrics -I../lib/telemetry/../metrics -Ilib/telemetry -I../lib/telemetry -Ilib/mbuf -I../lib/mbuf -Ilib/mempool -I../lib/mempool -Ilib/ring -I../lib/ring -Ilib/rcu -I../lib/rcu -Idrivers/bus/vdev -I../drivers/bus/vdev -I/usr/local/include -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Werror -std=c11 -O2 -g -include rte_config.h -Wcast-qual -Wdeprecated -Wformat -Wformat-nonliteral -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -Wwrite-strings -Wno-address-of-packed-member -Wno-packed-not-aligned -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native -mrtm -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -Wno-format-truncation -DRTE_LOG_DEFAULT_LOGTYPE=pmd.crypto.openssl -MD -MQ 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd.c.o' -MF 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd.c.o.d' -o 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd.c.o' -c ../drivers/crypto/openssl/rte_openssl_pmd.c
In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:19:
../drivers/crypto/openssl/compat.h: In function 'free_hmac_ctx':
../drivers/crypto/openssl/compat.h:24:2: error: 'HMAC_CTX_free' is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations]
   24 |  HMAC_CTX_free(ctx);
      |  ^~~~~~~~~~~~~
In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:14:
/usr/local/include/openssl/hmac.h:35:28: note: declared here
   35 | OSSL_DEPRECATEDIN_3_0 void HMAC_CTX_free(HMAC_CTX *ctx);
      |                            ^~~~~~~~~~~~~
In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:19:
../drivers/crypto/openssl/compat.h: In function 'free_cmac_ctx':
../drivers/crypto/openssl/compat.h:30:2: error: 'CMAC_CTX_free' is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations]
   30 |  CMAC_CTX_free(ctx);
      |  ^~~~~~~~~~~~~
In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:13:
/usr/local/include/openssl/cmac.h:34:28: note: declared here
   34 | OSSL_DEPRECATEDIN_3_0 void CMAC_CTX_free(CMAC_CTX *ctx);
      |                            ^~~~~~~~~~~~~
../drivers/crypto/openssl/rte_openssl_pmd.c: In function 'openssl_reset_session':
../drivers/crypto/openssl/rte_openssl_pmd.c:908:33: error: passing argument 1 of 'free_hmac_ctx' from incompatible pointer type [-Werror=incompatible-pointer-types]
  908 |    free_hmac_ctx(sess->qp_ctx[i].hmac);
      |                  ~~~~~~~~~~~~~~~^~~~~
      |                                 |
      |                                 EVP_MAC_CTX * {aka struct evp_mac_ctx_st *}
In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:19:
../drivers/crypto/openssl/compat.h:22:25: note: expected 'HMAC_CTX *' {aka 'struct hmac_ctx_st *'} but argument is of type 'EVP_MAC_CTX *' {aka 'struct evp_mac_ctx_st *'}
   22 | free_hmac_ctx(HMAC_CTX *ctx)
      |               ~~~~~~~~~~^~~
../drivers/crypto/openssl/rte_openssl_pmd.c:912:33: error: passing argument 1 of 'free_cmac_ctx' from incompatible pointer type [-Werror=incompatible-pointer-types]
  912 |    free_cmac_ctx(sess->qp_ctx[i].cmac);
      |                  ~~~~~~~~~~~~~~~^~~~~
      |                                 |
      |                                 EVP_MAC_CTX * {aka struct evp_mac_ctx_st *}
In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:19:
../drivers/crypto/openssl/compat.h:28:25: note: expected 'CMAC_CTX *' {aka 'struct CMAC_CTX_st *'} but argument is of type 'EVP_MAC_CTX *' {aka 'struct evp_mac_ctx_st *'}
   28 | free_cmac_ctx(CMAC_CTX *ctx)
      |               ~~~~~~~~~~^~~
../drivers/crypto/openssl/rte_openssl_pmd.c:925:32: error: passing argument 1 of 'free_hmac_ctx' from incompatible pointer type [-Werror=incompatible-pointer-types]
  925 |   free_hmac_ctx(sess->auth.hmac.ctx);
      |                 ~~~~~~~~~~~~~~~^~~~
      |                                |
      |                                EVP_MAC_CTX * {aka struct evp_mac_ctx_st *}
In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:19:
../drivers/crypto/openssl/compat.h:22:25: note: expected 'HMAC_CTX *' {aka 'struct hmac_ctx_st *'} but argument is of type 'EVP_MAC_CTX *' {aka 'struct evp_mac_ctx_st *'}
   22 | free_hmac_ctx(HMAC_CTX *ctx)
      |               ~~~~~~~~~~^~~
../drivers/crypto/openssl/rte_openssl_pmd.c:928:32: error: passing argument 1 of 'free_cmac_ctx' from incompatible pointer type [-Werror=incompatible-pointer-types]
  928 |   free_cmac_ctx(sess->auth.cmac.ctx);
      |                 ~~~~~~~~~~~~~~~^~~~
      |                                |
      |                                EVP_MAC_CTX * {aka struct evp_mac_ctx_st *}
In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:19:
../drivers/crypto/openssl/compat.h:28:25: note: expected 'CMAC_CTX *' {aka 'struct CMAC_CTX_st *'} but argument is of type 'EVP_MAC_CTX *' {aka 'struct evp_mac_ctx_st *'}
   28 | free_cmac_ctx(CMAC_CTX *ctx)
      |               ~~~~~~~~~~^~~
cc1: all warnings being treated as errors

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [EXTERNAL] [PATCH v4 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs
  2024-07-02 15:39     ` [EXTERNAL] " Akhil Goyal
@ 2024-07-03 10:49       ` Jack Bond-Preston
  0 siblings, 0 replies; 12+ messages in thread
From: Jack Bond-Preston @ 2024-07-03 10:49 UTC (permalink / raw)
  To: Akhil Goyal, Kai Ji, Fan Zhang; +Cc: dev, stable, Wathsala Vithanage

On 02/07/2024 16:39, Akhil Goyal wrote:
> 
> I am seeing below errors when it is compiled with openssl 3.0
> 
> [1/30] Compiling C object 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd_ops.c.o'.
> FAILED: drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd_ops.c.o
> <snip>
Ah, that's my mistake. The OpenSSL version comparison macros should be 
 >= 3.0.0, not > (in patches 1/5 and 3/5). I will push the amended 
patchset ASAP.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v5 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs
       [not found] ` <20240703134552.1439633-1-jack.bond-preston@foss.arm.com>
@ 2024-07-03 13:45   ` Jack Bond-Preston
  2024-07-03 15:20     ` [EXTERNAL] " Akhil Goyal
  0 siblings, 1 reply; 12+ messages in thread
From: Jack Bond-Preston @ 2024-07-03 13:45 UTC (permalink / raw)
  To: Kai Ji, Fan Zhang, Akhil Goyal; +Cc: dev, stable, Wathsala Vithanage

Commit 67ab783b5d70 ("crypto/openssl: use local copy for session
contexts") introduced a fix for concurrency bugs which could occur when
using one OpenSSL PMD session across multiple cores simultaneously. The
solution was to clone the EVP contexts per-buffer to avoid them being
used concurrently.

However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC
routine with 3.0 EVP API") reverted this fix, only for combined ops
(AES-GCM and AES-CCM).

Fix the concurrency issue by cloning EVP contexts per-buffer. An extra
workaround is required for OpenSSL versions which are >= 3.0.0, and
<= 3.2.0. This is because, prior to OpenSSL 3.2.0, EVP_CIPHER_CTX_copy()
is not implemented for AES-GCM or AES-CCM. When using these OpenSSL
versions, create and initialise the context from scratch, per-buffer.

Throughput performance uplift measurements for AES-GCM-128 encrypt on
Ampere Altra Max platform:
1 worker lcore
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |          2.60 |               1.31 |   -49.5% |
|             256 |          7.69 |               4.45 |   -42.1% |
|            1024 |         15.33 |              11.30 |   -26.3% |
|            2048 |         18.74 |              15.37 |   -18.0% |
|            4096 |         21.11 |              18.80 |   -10.9% |

8 worker lcores
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |         19.94 |               2.83 |   -85.8% |
|             256 |         58.84 |              11.00 |   -81.3% |
|            1024 |        119.71 |              42.46 |   -64.5% |
|            2048 |        147.69 |              80.91 |   -45.2% |
|            4096 |        167.39 |             121.25 |   -27.6% |

Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API")
Cc: stable@dpdk.org
Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Acked-by: Kai Ji <kai.ji@intel.com>
Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
---
 drivers/crypto/openssl/rte_openssl_pmd.c | 84 ++++++++++++++++++------
 1 file changed, 64 insertions(+), 20 deletions(-)

diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index e8cb09defc..c661528738 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -350,7 +350,8 @@ get_aead_algo(enum rte_crypto_aead_algorithm sess_algo, size_t keylen,
 static int
 openssl_set_sess_aead_enc_param(struct openssl_session *sess,
 		enum rte_crypto_aead_algorithm algo,
-		uint8_t tag_len, const uint8_t *key)
+		uint8_t tag_len, const uint8_t *key,
+		EVP_CIPHER_CTX **ctx)
 {
 	int iv_type = 0;
 	unsigned int do_ccm;
@@ -378,7 +379,7 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess,
 	}
 
 	sess->cipher.mode = OPENSSL_CIPHER_LIB;
-	sess->cipher.ctx = EVP_CIPHER_CTX_new();
+	*ctx = EVP_CIPHER_CTX_new();
 
 	if (get_aead_algo(algo, sess->cipher.key.length,
 			&sess->cipher.evp_algo) != 0)
@@ -388,19 +389,19 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess,
 
 	sess->chain_order = OPENSSL_CHAIN_COMBINED;
 
-	if (EVP_EncryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo,
+	if (EVP_EncryptInit_ex(*ctx, sess->cipher.evp_algo,
 			NULL, NULL, NULL) <= 0)
 		return -EINVAL;
 
-	if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type, sess->iv.length,
+	if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type, sess->iv.length,
 			NULL) <= 0)
 		return -EINVAL;
 
 	if (do_ccm)
-		EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG,
+		EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG,
 				tag_len, NULL);
 
-	if (EVP_EncryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0)
+	if (EVP_EncryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0)
 		return -EINVAL;
 
 	return 0;
@@ -410,7 +411,8 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess,
 static int
 openssl_set_sess_aead_dec_param(struct openssl_session *sess,
 		enum rte_crypto_aead_algorithm algo,
-		uint8_t tag_len, const uint8_t *key)
+		uint8_t tag_len, const uint8_t *key,
+		EVP_CIPHER_CTX **ctx)
 {
 	int iv_type = 0;
 	unsigned int do_ccm = 0;
@@ -437,7 +439,7 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess,
 	}
 
 	sess->cipher.mode = OPENSSL_CIPHER_LIB;
-	sess->cipher.ctx = EVP_CIPHER_CTX_new();
+	*ctx = EVP_CIPHER_CTX_new();
 
 	if (get_aead_algo(algo, sess->cipher.key.length,
 			&sess->cipher.evp_algo) != 0)
@@ -447,24 +449,54 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess,
 
 	sess->chain_order = OPENSSL_CHAIN_COMBINED;
 
-	if (EVP_DecryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo,
+	if (EVP_DecryptInit_ex(*ctx, sess->cipher.evp_algo,
 			NULL, NULL, NULL) <= 0)
 		return -EINVAL;
 
-	if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type,
+	if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type,
 			sess->iv.length, NULL) <= 0)
 		return -EINVAL;
 
 	if (do_ccm)
-		EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG,
+		EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG,
 				tag_len, NULL);
 
-	if (EVP_DecryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0)
+	if (EVP_DecryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0)
 		return -EINVAL;
 
 	return 0;
 }
 
+static int openssl_aesni_ctx_clone(EVP_CIPHER_CTX **dest,
+		struct openssl_session *sess)
+{
+#if (OPENSSL_VERSION_NUMBER >= 0x30200000L)
+	*dest = EVP_CIPHER_CTX_dup(sess->ctx);
+	return 0;
+#elif (OPENSSL_VERSION_NUMBER >= 0x30000000L)
+	/* OpenSSL versions 3.0.0 <= V < 3.2.0 have no dupctx() implementation
+	 * for AES-GCM and AES-CCM. In this case, we have to create new empty
+	 * contexts and initialise, as we did the original context.
+	 */
+	if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC)
+		sess->aead_algo = RTE_CRYPTO_AEAD_AES_GCM;
+
+	if (sess->cipher.direction == RTE_CRYPTO_CIPHER_OP_ENCRYPT)
+		return openssl_set_sess_aead_enc_param(sess, sess->aead_algo,
+				sess->auth.digest_length, sess->cipher.key.data,
+				dest);
+	else
+		return openssl_set_sess_aead_dec_param(sess, sess->aead_algo,
+				sess->auth.digest_length, sess->cipher.key.data,
+				dest);
+#else
+	*dest = EVP_CIPHER_CTX_new();
+	if (EVP_CIPHER_CTX_copy(*dest, sess->cipher.ctx) != 1)
+		return -EINVAL;
+	return 0;
+#endif
+}
+
 /** Set session cipher parameters */
 static int
 openssl_set_session_cipher_parameters(struct openssl_session *sess,
@@ -623,12 +655,14 @@ openssl_set_session_auth_parameters(struct openssl_session *sess,
 			return openssl_set_sess_aead_enc_param(sess,
 						RTE_CRYPTO_AEAD_AES_GCM,
 						xform->auth.digest_length,
-						xform->auth.key.data);
+						xform->auth.key.data,
+						&sess->cipher.ctx);
 		else
 			return openssl_set_sess_aead_dec_param(sess,
 						RTE_CRYPTO_AEAD_AES_GCM,
 						xform->auth.digest_length,
-						xform->auth.key.data);
+						xform->auth.key.data,
+						&sess->cipher.ctx);
 		break;
 
 	case RTE_CRYPTO_AUTH_MD5:
@@ -770,10 +804,12 @@ openssl_set_session_aead_parameters(struct openssl_session *sess,
 	/* Select cipher direction */
 	if (xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT)
 		return openssl_set_sess_aead_enc_param(sess, xform->aead.algo,
-				xform->aead.digest_length, xform->aead.key.data);
+				xform->aead.digest_length, xform->aead.key.data,
+				&sess->cipher.ctx);
 	else
 		return openssl_set_sess_aead_dec_param(sess, xform->aead.algo,
-				xform->aead.digest_length, xform->aead.key.data);
+				xform->aead.digest_length, xform->aead.key.data,
+				&sess->cipher.ctx);
 }
 
 /** Parse crypto xform chain and set private session parameters */
@@ -1590,6 +1626,12 @@ process_openssl_combined_op
 		return;
 	}
 
+	EVP_CIPHER_CTX *ctx;
+	if (openssl_aesni_ctx_clone(&ctx, sess) != 0) {
+		op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+		return;
+	}
+
 	iv = rte_crypto_op_ctod_offset(op, uint8_t *,
 			sess->iv.offset);
 	if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) {
@@ -1623,12 +1665,12 @@ process_openssl_combined_op
 			status = process_openssl_auth_encryption_gcm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, sess->cipher.ctx);
+					dst, tag, ctx);
 		else
 			status = process_openssl_auth_encryption_ccm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, taglen, sess->cipher.ctx);
+					dst, tag, taglen, ctx);
 
 	} else {
 		if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC ||
@@ -1636,14 +1678,16 @@ process_openssl_combined_op
 			status = process_openssl_auth_decryption_gcm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, sess->cipher.ctx);
+					dst, tag, ctx);
 		else
 			status = process_openssl_auth_decryption_ccm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, taglen, sess->cipher.ctx);
+					dst, tag, taglen, ctx);
 	}
 
+	EVP_CIPHER_CTX_free(ctx);
+
 	if (status != 0) {
 		if (status == (-EFAULT) &&
 				sess->auth.operation ==
-- 
2.34.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* RE: [EXTERNAL] [PATCH v5 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs
  2024-07-03 13:45   ` [PATCH v5 " Jack Bond-Preston
@ 2024-07-03 15:20     ` Akhil Goyal
  0 siblings, 0 replies; 12+ messages in thread
From: Akhil Goyal @ 2024-07-03 15:20 UTC (permalink / raw)
  To: Jack Bond-Preston, Kai Ji, Fan Zhang; +Cc: dev, stable, Wathsala Vithanage

> Subject: [EXTERNAL] [PATCH v5 1/5] crypto/openssl: fix GCM and CCM thread
> unsafe ctxs
> 
> Commit 67ab783b5d70 ("crypto/openssl: use local copy for session
> contexts") introduced a fix for concurrency bugs which could occur when
> using one OpenSSL PMD session across multiple cores simultaneously. The
> solution was to clone the EVP contexts per-buffer to avoid them being
> used concurrently.
> 
> However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC
> routine with 3.0 EVP API") reverted this fix, only for combined ops
> (AES-GCM and AES-CCM).
> 
> Fix the concurrency issue by cloning EVP contexts per-buffer. An extra
> workaround is required for OpenSSL versions which are >= 3.0.0, and
> <= 3.2.0. This is because, prior to OpenSSL 3.2.0, EVP_CIPHER_CTX_copy()
> is not implemented for AES-GCM or AES-CCM. When using these OpenSSL
> versions, create and initialise the context from scratch, per-buffer.
> 
> Throughput performance uplift measurements for AES-GCM-128 encrypt on
> Ampere Altra Max platform:
> 1 worker lcore
> |   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
> |-----------------+---------------+--------------------+----------|
> |              64 |          2.60 |               1.31 |   -49.5% |
> |             256 |          7.69 |               4.45 |   -42.1% |
> |            1024 |         15.33 |              11.30 |   -26.3% |
> |            2048 |         18.74 |              15.37 |   -18.0% |
> |            4096 |         21.11 |              18.80 |   -10.9% |
> 
> 8 worker lcores
> |   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
> |-----------------+---------------+--------------------+----------|
> |              64 |         19.94 |               2.83 |   -85.8% |
> |             256 |         58.84 |              11.00 |   -81.3% |
> |            1024 |        119.71 |              42.46 |   -64.5% |
> |            2048 |        147.69 |              80.91 |   -45.2% |
> |            4096 |        167.39 |             121.25 |   -27.6% |
> 
> Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API")
> Cc: stable@dpdk.org

Since this fix is reducing the performance significantly.
Backporting it alone would reduce performance on stable releases.

I would suggest applying all the patches in this series for backports.


> Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
> Acked-by: Kai Ji <kai.ji@intel.com>
> Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
Applied to dpdk-next-crypto
Thanks for the optimizations.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs
       [not found] <20240603155555.1278010-1-jack.bond-preston@foss.arm.com>
@ 2024-06-03 15:55 ` Jack Bond-Preston
  0 siblings, 0 replies; 12+ messages in thread
From: Jack Bond-Preston @ 2024-06-03 15:55 UTC (permalink / raw)
  To: jack.bond-preston; +Cc: stable, Wathsala Vithanage

Commit 67ab783b5d70 ("crypto/openssl: use local copy for session
contexts") introduced a fix for concurrency bugs which could occur when
using one OpenSSL PMD session across multiple cores simultaneously. The
solution was to clone the EVP contexts per-buffer to avoid them being
used concurrently.

However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC
routine with 3.0 EVP API") reverted this fix, only for combined ops
(AES-GCM and AES-CCM), with no explanation. This commit fixes the issue
again, essentially reverting this part of the commit.

Throughput performance uplift measurements for AES-GCM-128 encrypt on
Ampere Altra Max platform:
1 worker lcore
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |          2.60 |               1.31 |   -49.5% |
|             256 |          7.69 |               4.45 |   -42.1% |
|            1024 |         15.33 |              11.30 |   -26.3% |
|            2048 |         18.74 |              15.37 |   -18.0% |
|            4096 |         21.11 |              18.80 |   -10.9% |

8 worker lcores
|   buffer sz (B) |   prev (Gbps) |   optimised (Gbps) |   uplift |
|-----------------+---------------+--------------------+----------|
|              64 |         19.94 |               2.83 |   -85.8% |
|             256 |         58.84 |              11.00 |   -81.3% |
|            1024 |        119.71 |              42.46 |   -64.5% |
|            2048 |        147.69 |              80.91 |   -45.2% |
|            4096 |        167.39 |             121.25 |   -27.6% |

Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API")
Cc: stable@dpdk.org
Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
---
 drivers/crypto/openssl/rte_openssl_pmd.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index e8cb09defc..ca7ed30ec4 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -1590,6 +1590,9 @@ process_openssl_combined_op
 		return;
 	}
 
+	EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
+	EVP_CIPHER_CTX_copy(ctx, sess->cipher.ctx);
+
 	iv = rte_crypto_op_ctod_offset(op, uint8_t *,
 			sess->iv.offset);
 	if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) {
@@ -1623,12 +1626,12 @@ process_openssl_combined_op
 			status = process_openssl_auth_encryption_gcm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, sess->cipher.ctx);
+					dst, tag, ctx);
 		else
 			status = process_openssl_auth_encryption_ccm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, taglen, sess->cipher.ctx);
+					dst, tag, taglen, ctx);
 
 	} else {
 		if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC ||
@@ -1636,14 +1639,16 @@ process_openssl_combined_op
 			status = process_openssl_auth_decryption_gcm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, sess->cipher.ctx);
+					dst, tag, ctx);
 		else
 			status = process_openssl_auth_decryption_ccm(
 					mbuf_src, offset, srclen,
 					aad, aadlen, iv,
-					dst, tag, taglen, sess->cipher.ctx);
+					dst, tag, taglen, ctx);
 	}
 
+	EVP_CIPHER_CTX_free(ctx);
+
 	if (status != 0) {
 		if (status == (-EFAULT) &&
 				sess->auth.operation ==
-- 
2.34.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2024-07-03 15:20 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <20240603160119.1279476-1-jack.bond-preston@foss.arm.com>
2024-06-03 16:01 ` [PATCH 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs Jack Bond-Preston
     [not found] ` <20240603184348.1310331-1-jack.bond-preston@foss.arm.com>
2024-06-03 18:43   ` [PATCH v2 " Jack Bond-Preston
     [not found] ` <20240603185939.1312680-1-jack.bond-preston@foss.arm.com>
2024-06-03 18:59   ` Jack Bond-Preston
     [not found] ` <20240606102043.2926695-1-jack.bond-preston@foss.arm.com>
2024-06-06 10:20   ` [PATCH v3 " Jack Bond-Preston
2024-06-06 10:44     ` [EXTERNAL] " Akhil Goyal
     [not found] ` <20240607124756.3968704-1-jack.bond-preston@foss.arm.com>
2024-06-07 12:47   ` [PATCH v4 " Jack Bond-Preston
2024-07-01 12:55     ` Ji, Kai
2024-07-02 15:39     ` [EXTERNAL] " Akhil Goyal
2024-07-03 10:49       ` Jack Bond-Preston
     [not found] ` <20240703134552.1439633-1-jack.bond-preston@foss.arm.com>
2024-07-03 13:45   ` [PATCH v5 " Jack Bond-Preston
2024-07-03 15:20     ` [EXTERNAL] " Akhil Goyal
     [not found] <20240603155555.1278010-1-jack.bond-preston@foss.arm.com>
2024-06-03 15:55 ` [PATCH " Jack Bond-Preston

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).