* [PATCH 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs [not found] <20240603160119.1279476-1-jack.bond-preston@foss.arm.com> @ 2024-06-03 16:01 ` Jack Bond-Preston [not found] ` <20240603184348.1310331-1-jack.bond-preston@foss.arm.com> ` (4 subsequent siblings) 5 siblings, 0 replies; 11+ messages in thread From: Jack Bond-Preston @ 2024-06-03 16:01 UTC (permalink / raw) To: Kai Ji, Fan Zhang, Akhil Goyal; +Cc: dev, stable, Wathsala Vithanage Commit 67ab783b5d70 ("crypto/openssl: use local copy for session contexts") introduced a fix for concurrency bugs which could occur when using one OpenSSL PMD session across multiple cores simultaneously. The solution was to clone the EVP contexts per-buffer to avoid them being used concurrently. However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") reverted this fix, only for combined ops (AES-GCM and AES-CCM), with no explanation. This commit fixes the issue again, essentially reverting this part of the commit. Throughput performance uplift measurements for AES-GCM-128 encrypt on Ampere Altra Max platform: 1 worker lcore | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 2.60 | 1.31 | -49.5% | | 256 | 7.69 | 4.45 | -42.1% | | 1024 | 15.33 | 11.30 | -26.3% | | 2048 | 18.74 | 15.37 | -18.0% | | 4096 | 21.11 | 18.80 | -10.9% | 8 worker lcores | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 19.94 | 2.83 | -85.8% | | 256 | 58.84 | 11.00 | -81.3% | | 1024 | 119.71 | 42.46 | -64.5% | | 2048 | 147.69 | 80.91 | -45.2% | | 4096 | 167.39 | 121.25 | -27.6% | Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") Cc: stable@dpdk.org Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com> Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com> --- drivers/crypto/openssl/rte_openssl_pmd.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c index e8cb09defc..ca7ed30ec4 100644 --- a/drivers/crypto/openssl/rte_openssl_pmd.c +++ b/drivers/crypto/openssl/rte_openssl_pmd.c @@ -1590,6 +1590,9 @@ process_openssl_combined_op return; } + EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new(); + EVP_CIPHER_CTX_copy(ctx, sess->cipher.ctx); + iv = rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv.offset); if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) { @@ -1623,12 +1626,12 @@ process_openssl_combined_op status = process_openssl_auth_encryption_gcm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx); else status = process_openssl_auth_encryption_ccm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx); } else { if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC || @@ -1636,14 +1639,16 @@ process_openssl_combined_op status = process_openssl_auth_decryption_gcm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx); else status = process_openssl_auth_decryption_ccm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx); } + EVP_CIPHER_CTX_free(ctx); + if (status != 0) { if (status == (-EFAULT) && sess->auth.operation == -- 2.34.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <20240603184348.1310331-1-jack.bond-preston@foss.arm.com>]
* [PATCH v2 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs [not found] ` <20240603184348.1310331-1-jack.bond-preston@foss.arm.com> @ 2024-06-03 18:43 ` Jack Bond-Preston 0 siblings, 0 replies; 11+ messages in thread From: Jack Bond-Preston @ 2024-06-03 18:43 UTC (permalink / raw) To: Kai Ji, Fan Zhang, Akhil Goyal; +Cc: dev, stable, Wathsala Vithanage Commit 67ab783b5d70 ("crypto/openssl: use local copy for session contexts") introduced a fix for concurrency bugs which could occur when using one OpenSSL PMD session across multiple cores simultaneously. The solution was to clone the EVP contexts per-buffer to avoid them being used concurrently. However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") reverted this fix, only for combined ops (AES-GCM and AES-CCM), with no explanation. This commit fixes the issue again, essentially reverting this part of the commit. Throughput performance uplift measurements for AES-GCM-128 encrypt on Ampere Altra Max platform: 1 worker lcore | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 2.60 | 1.31 | -49.5% | | 256 | 7.69 | 4.45 | -42.1% | | 1024 | 15.33 | 11.30 | -26.3% | | 2048 | 18.74 | 15.37 | -18.0% | | 4096 | 21.11 | 18.80 | -10.9% | 8 worker lcores | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 19.94 | 2.83 | -85.8% | | 256 | 58.84 | 11.00 | -81.3% | | 1024 | 119.71 | 42.46 | -64.5% | | 2048 | 147.69 | 80.91 | -45.2% | | 4096 | 167.39 | 121.25 | -27.6% | Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") Cc: stable@dpdk.org Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com> Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com> --- drivers/crypto/openssl/rte_openssl_pmd.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c index e8cb09defc..ca7ed30ec4 100644 --- a/drivers/crypto/openssl/rte_openssl_pmd.c +++ b/drivers/crypto/openssl/rte_openssl_pmd.c @@ -1590,6 +1590,9 @@ process_openssl_combined_op return; } + EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new(); + EVP_CIPHER_CTX_copy(ctx, sess->cipher.ctx); + iv = rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv.offset); if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) { @@ -1623,12 +1626,12 @@ process_openssl_combined_op status = process_openssl_auth_encryption_gcm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx); else status = process_openssl_auth_encryption_ccm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx); } else { if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC || @@ -1636,14 +1639,16 @@ process_openssl_combined_op status = process_openssl_auth_decryption_gcm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx); else status = process_openssl_auth_decryption_ccm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx); } + EVP_CIPHER_CTX_free(ctx); + if (status != 0) { if (status == (-EFAULT) && sess->auth.operation == -- 2.34.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <20240603185939.1312680-1-jack.bond-preston@foss.arm.com>]
* [PATCH v2 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs [not found] ` <20240603185939.1312680-1-jack.bond-preston@foss.arm.com> @ 2024-06-03 18:59 ` Jack Bond-Preston 0 siblings, 0 replies; 11+ messages in thread From: Jack Bond-Preston @ 2024-06-03 18:59 UTC (permalink / raw) To: Kai Ji, Fan Zhang, Akhil Goyal; +Cc: dev, stable, Wathsala Vithanage Commit 67ab783b5d70 ("crypto/openssl: use local copy for session contexts") introduced a fix for concurrency bugs which could occur when using one OpenSSL PMD session across multiple cores simultaneously. The solution was to clone the EVP contexts per-buffer to avoid them being used concurrently. However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") reverted this fix, only for combined ops (AES-GCM and AES-CCM), with no explanation. This commit fixes the issue again, essentially reverting this part of the commit. Throughput performance uplift measurements for AES-GCM-128 encrypt on Ampere Altra Max platform: 1 worker lcore | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 2.60 | 1.31 | -49.5% | | 256 | 7.69 | 4.45 | -42.1% | | 1024 | 15.33 | 11.30 | -26.3% | | 2048 | 18.74 | 15.37 | -18.0% | | 4096 | 21.11 | 18.80 | -10.9% | 8 worker lcores | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 19.94 | 2.83 | -85.8% | | 256 | 58.84 | 11.00 | -81.3% | | 1024 | 119.71 | 42.46 | -64.5% | | 2048 | 147.69 | 80.91 | -45.2% | | 4096 | 167.39 | 121.25 | -27.6% | Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") Cc: stable@dpdk.org Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com> Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com> --- drivers/crypto/openssl/rte_openssl_pmd.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c index e8cb09defc..ca7ed30ec4 100644 --- a/drivers/crypto/openssl/rte_openssl_pmd.c +++ b/drivers/crypto/openssl/rte_openssl_pmd.c @@ -1590,6 +1590,9 @@ process_openssl_combined_op return; } + EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new(); + EVP_CIPHER_CTX_copy(ctx, sess->cipher.ctx); + iv = rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv.offset); if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) { @@ -1623,12 +1626,12 @@ process_openssl_combined_op status = process_openssl_auth_encryption_gcm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx); else status = process_openssl_auth_encryption_ccm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx); } else { if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC || @@ -1636,14 +1639,16 @@ process_openssl_combined_op status = process_openssl_auth_decryption_gcm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx); else status = process_openssl_auth_decryption_ccm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx); } + EVP_CIPHER_CTX_free(ctx); + if (status != 0) { if (status == (-EFAULT) && sess->auth.operation == -- 2.34.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <20240606102043.2926695-1-jack.bond-preston@foss.arm.com>]
* [PATCH v3 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs [not found] ` <20240606102043.2926695-1-jack.bond-preston@foss.arm.com> @ 2024-06-06 10:20 ` Jack Bond-Preston 2024-06-06 10:44 ` [EXTERNAL] " Akhil Goyal 0 siblings, 1 reply; 11+ messages in thread From: Jack Bond-Preston @ 2024-06-06 10:20 UTC (permalink / raw) To: Kai Ji, Akhil Goyal, Fan Zhang; +Cc: dev, stable, Wathsala Vithanage Commit 67ab783b5d70 ("crypto/openssl: use local copy for session contexts") introduced a fix for concurrency bugs which could occur when using one OpenSSL PMD session across multiple cores simultaneously. The solution was to clone the EVP contexts per-buffer to avoid them being used concurrently. However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") reverted this fix, only for combined ops (AES-GCM and AES-CCM). Fix the concurrency issue by cloning EVP contexts per-buffer. An extra workaround is required for OpenSSL versions which are >= 3.0.0, and <= 3.2.0. This is because, prior to OpenSSL 3.2.0, EVP_CIPHER_CTX_copy() is not implemented for AES-GCM or AES-CCM. When using these OpenSSL versions, create and initialise the context from scratch, per-buffer. Throughput performance uplift measurements for AES-GCM-128 encrypt on Ampere Altra Max platform: 1 worker lcore | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 2.60 | 1.31 | -49.5% | | 256 | 7.69 | 4.45 | -42.1% | | 1024 | 15.33 | 11.30 | -26.3% | | 2048 | 18.74 | 15.37 | -18.0% | | 4096 | 21.11 | 18.80 | -10.9% | 8 worker lcores | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 19.94 | 2.83 | -85.8% | | 256 | 58.84 | 11.00 | -81.3% | | 1024 | 119.71 | 42.46 | -64.5% | | 2048 | 147.69 | 80.91 | -45.2% | | 4096 | 167.39 | 121.25 | -27.6% | Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") Cc: stable@dpdk.org Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com> Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com> --- drivers/crypto/openssl/rte_openssl_pmd.c | 84 ++++++++++++++++++------ 1 file changed, 64 insertions(+), 20 deletions(-) diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c index e8cb09defc..3f7f4d8c37 100644 --- a/drivers/crypto/openssl/rte_openssl_pmd.c +++ b/drivers/crypto/openssl/rte_openssl_pmd.c @@ -350,7 +350,8 @@ get_aead_algo(enum rte_crypto_aead_algorithm sess_algo, size_t keylen, static int openssl_set_sess_aead_enc_param(struct openssl_session *sess, enum rte_crypto_aead_algorithm algo, - uint8_t tag_len, const uint8_t *key) + uint8_t tag_len, const uint8_t *key, + EVP_CIPHER_CTX **ctx) { int iv_type = 0; unsigned int do_ccm; @@ -378,7 +379,7 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess, } sess->cipher.mode = OPENSSL_CIPHER_LIB; - sess->cipher.ctx = EVP_CIPHER_CTX_new(); + *ctx = EVP_CIPHER_CTX_new(); if (get_aead_algo(algo, sess->cipher.key.length, &sess->cipher.evp_algo) != 0) @@ -388,19 +389,19 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess, sess->chain_order = OPENSSL_CHAIN_COMBINED; - if (EVP_EncryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo, + if (EVP_EncryptInit_ex(*ctx, sess->cipher.evp_algo, NULL, NULL, NULL) <= 0) return -EINVAL; - if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type, sess->iv.length, + if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type, sess->iv.length, NULL) <= 0) return -EINVAL; if (do_ccm) - EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG, + EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG, tag_len, NULL); - if (EVP_EncryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0) + if (EVP_EncryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0) return -EINVAL; return 0; @@ -410,7 +411,8 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess, static int openssl_set_sess_aead_dec_param(struct openssl_session *sess, enum rte_crypto_aead_algorithm algo, - uint8_t tag_len, const uint8_t *key) + uint8_t tag_len, const uint8_t *key, + EVP_CIPHER_CTX **ctx) { int iv_type = 0; unsigned int do_ccm = 0; @@ -437,7 +439,7 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess, } sess->cipher.mode = OPENSSL_CIPHER_LIB; - sess->cipher.ctx = EVP_CIPHER_CTX_new(); + *ctx = EVP_CIPHER_CTX_new(); if (get_aead_algo(algo, sess->cipher.key.length, &sess->cipher.evp_algo) != 0) @@ -447,24 +449,54 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess, sess->chain_order = OPENSSL_CHAIN_COMBINED; - if (EVP_DecryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo, + if (EVP_DecryptInit_ex(*ctx, sess->cipher.evp_algo, NULL, NULL, NULL) <= 0) return -EINVAL; - if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type, + if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type, sess->iv.length, NULL) <= 0) return -EINVAL; if (do_ccm) - EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG, + EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG, tag_len, NULL); - if (EVP_DecryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0) + if (EVP_DecryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0) return -EINVAL; return 0; } +static int openssl_aesni_ctx_clone(EVP_CIPHER_CTX **dest, + struct openssl_session *sess) +{ +#if (OPENSSL_VERSION_NUMBER > 0x30200000L) + *dest = EVP_CIPHER_CTX_dup(sess->ctx); + return 0; +#elif (OPENSSL_VERSION_NUMBER > 0x30000000L) + /* OpenSSL versions 3.0.0 <= V < 3.2.0 have no dupctx() implementation + * for AES-GCM and AES-CCM. In this case, we have to create new empty + * contexts and initialise, as we did the original context. + */ + if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) + sess->aead_algo = RTE_CRYPTO_AEAD_AES_GCM; + + if (sess->cipher.direction == RTE_CRYPTO_CIPHER_OP_ENCRYPT) + return openssl_set_sess_aead_enc_param(sess, sess->aead_algo, + sess->auth.digest_length, sess->cipher.key.data, + dest); + else + return openssl_set_sess_aead_dec_param(sess, sess->aead_algo, + sess->auth.digest_length, sess->cipher.key.data, + dest); +#else + *dest = EVP_CIPHER_CTX_new(); + if (EVP_CIPHER_CTX_copy(*dest, sess->cipher.ctx) != 1) + return -EINVAL; + return 0; +#endif +} + /** Set session cipher parameters */ static int openssl_set_session_cipher_parameters(struct openssl_session *sess, @@ -623,12 +655,14 @@ openssl_set_session_auth_parameters(struct openssl_session *sess, return openssl_set_sess_aead_enc_param(sess, RTE_CRYPTO_AEAD_AES_GCM, xform->auth.digest_length, - xform->auth.key.data); + xform->auth.key.data, + &sess->cipher.ctx); else return openssl_set_sess_aead_dec_param(sess, RTE_CRYPTO_AEAD_AES_GCM, xform->auth.digest_length, - xform->auth.key.data); + xform->auth.key.data, + &sess->cipher.ctx); break; case RTE_CRYPTO_AUTH_MD5: @@ -770,10 +804,12 @@ openssl_set_session_aead_parameters(struct openssl_session *sess, /* Select cipher direction */ if (xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT) return openssl_set_sess_aead_enc_param(sess, xform->aead.algo, - xform->aead.digest_length, xform->aead.key.data); + xform->aead.digest_length, xform->aead.key.data, + &sess->cipher.ctx); else return openssl_set_sess_aead_dec_param(sess, xform->aead.algo, - xform->aead.digest_length, xform->aead.key.data); + xform->aead.digest_length, xform->aead.key.data, + &sess->cipher.ctx); } /** Parse crypto xform chain and set private session parameters */ @@ -1590,6 +1626,12 @@ process_openssl_combined_op return; } + EVP_CIPHER_CTX *ctx; + if (openssl_aesni_ctx_clone(&ctx, sess) != 0) { + op->status = RTE_CRYPTO_OP_STATUS_ERROR; + return; + } + iv = rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv.offset); if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) { @@ -1623,12 +1665,12 @@ process_openssl_combined_op status = process_openssl_auth_encryption_gcm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx); else status = process_openssl_auth_encryption_ccm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx); } else { if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC || @@ -1636,14 +1678,16 @@ process_openssl_combined_op status = process_openssl_auth_decryption_gcm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx); else status = process_openssl_auth_decryption_ccm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx); } + EVP_CIPHER_CTX_free(ctx); + if (status != 0) { if (status == (-EFAULT) && sess->auth.operation == -- 2.34.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [EXTERNAL] [PATCH v3 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs 2024-06-06 10:20 ` [PATCH v3 " Jack Bond-Preston @ 2024-06-06 10:44 ` Akhil Goyal 0 siblings, 0 replies; 11+ messages in thread From: Akhil Goyal @ 2024-06-06 10:44 UTC (permalink / raw) To: Jack Bond-Preston, Kai Ji, Fan Zhang; +Cc: dev, stable, Wathsala Vithanage > Subject: [EXTERNAL] [PATCH v3 1/5] crypto/openssl: fix GCM and CCM thread > unsafe ctxs > > Commit 67ab783b5d70 ("crypto/openssl: use local copy for session > contexts") introduced a fix for concurrency bugs which could occur when > using one OpenSSL PMD session across multiple cores simultaneously. The > solution was to clone the EVP contexts per-buffer to avoid them being > used concurrently. > > However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC > routine with 3.0 EVP API") reverted this fix, only for combined ops > (AES-GCM and AES-CCM). > > Fix the concurrency issue by cloning EVP contexts per-buffer. An extra > workaround is required for OpenSSL versions which are >= 3.0.0, and > <= 3.2.0. This is because, prior to OpenSSL 3.2.0, EVP_CIPHER_CTX_copy() > is not implemented for AES-GCM or AES-CCM. When using these OpenSSL > versions, create and initialise the context from scratch, per-buffer. > > Throughput performance uplift measurements for AES-GCM-128 encrypt on > Ampere Altra Max platform: > 1 worker lcore > | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | > |-----------------+---------------+--------------------+----------| > | 64 | 2.60 | 1.31 | -49.5% | > | 256 | 7.69 | 4.45 | -42.1% | > | 1024 | 15.33 | 11.30 | -26.3% | > | 2048 | 18.74 | 15.37 | -18.0% | > | 4096 | 21.11 | 18.80 | -10.9% | > > 8 worker lcores > | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | > |-----------------+---------------+--------------------+----------| > | 64 | 19.94 | 2.83 | -85.8% | > | 256 | 58.84 | 11.00 | -81.3% | > | 1024 | 119.71 | 42.46 | -64.5% | > | 2048 | 147.69 | 80.91 | -45.2% | > | 4096 | 167.39 | 121.25 | -27.6% | > > Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") > Cc: stable@dpdk.org > Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com> > Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com> Hi Kai, Please review these optimizations. Regards, Akhil ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <20240607124756.3968704-1-jack.bond-preston@foss.arm.com>]
* [PATCH v4 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs [not found] ` <20240607124756.3968704-1-jack.bond-preston@foss.arm.com> @ 2024-06-07 12:47 ` Jack Bond-Preston 2024-07-01 12:55 ` Ji, Kai 2024-07-02 15:39 ` [EXTERNAL] " Akhil Goyal 0 siblings, 2 replies; 11+ messages in thread From: Jack Bond-Preston @ 2024-06-07 12:47 UTC (permalink / raw) To: Kai Ji, Fan Zhang, Akhil Goyal; +Cc: dev, stable, Wathsala Vithanage Commit 67ab783b5d70 ("crypto/openssl: use local copy for session contexts") introduced a fix for concurrency bugs which could occur when using one OpenSSL PMD session across multiple cores simultaneously. The solution was to clone the EVP contexts per-buffer to avoid them being used concurrently. However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") reverted this fix, only for combined ops (AES-GCM and AES-CCM). Fix the concurrency issue by cloning EVP contexts per-buffer. An extra workaround is required for OpenSSL versions which are >= 3.0.0, and <= 3.2.0. This is because, prior to OpenSSL 3.2.0, EVP_CIPHER_CTX_copy() is not implemented for AES-GCM or AES-CCM. When using these OpenSSL versions, create and initialise the context from scratch, per-buffer. Throughput performance uplift measurements for AES-GCM-128 encrypt on Ampere Altra Max platform: 1 worker lcore | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 2.60 | 1.31 | -49.5% | | 256 | 7.69 | 4.45 | -42.1% | | 1024 | 15.33 | 11.30 | -26.3% | | 2048 | 18.74 | 15.37 | -18.0% | | 4096 | 21.11 | 18.80 | -10.9% | 8 worker lcores | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 19.94 | 2.83 | -85.8% | | 256 | 58.84 | 11.00 | -81.3% | | 1024 | 119.71 | 42.46 | -64.5% | | 2048 | 147.69 | 80.91 | -45.2% | | 4096 | 167.39 | 121.25 | -27.6% | Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") Cc: stable@dpdk.org Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com> Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com> --- drivers/crypto/openssl/rte_openssl_pmd.c | 84 ++++++++++++++++++------ 1 file changed, 64 insertions(+), 20 deletions(-) diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c index e8cb09defc..3f7f4d8c37 100644 --- a/drivers/crypto/openssl/rte_openssl_pmd.c +++ b/drivers/crypto/openssl/rte_openssl_pmd.c @@ -350,7 +350,8 @@ get_aead_algo(enum rte_crypto_aead_algorithm sess_algo, size_t keylen, static int openssl_set_sess_aead_enc_param(struct openssl_session *sess, enum rte_crypto_aead_algorithm algo, - uint8_t tag_len, const uint8_t *key) + uint8_t tag_len, const uint8_t *key, + EVP_CIPHER_CTX **ctx) { int iv_type = 0; unsigned int do_ccm; @@ -378,7 +379,7 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess, } sess->cipher.mode = OPENSSL_CIPHER_LIB; - sess->cipher.ctx = EVP_CIPHER_CTX_new(); + *ctx = EVP_CIPHER_CTX_new(); if (get_aead_algo(algo, sess->cipher.key.length, &sess->cipher.evp_algo) != 0) @@ -388,19 +389,19 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess, sess->chain_order = OPENSSL_CHAIN_COMBINED; - if (EVP_EncryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo, + if (EVP_EncryptInit_ex(*ctx, sess->cipher.evp_algo, NULL, NULL, NULL) <= 0) return -EINVAL; - if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type, sess->iv.length, + if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type, sess->iv.length, NULL) <= 0) return -EINVAL; if (do_ccm) - EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG, + EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG, tag_len, NULL); - if (EVP_EncryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0) + if (EVP_EncryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0) return -EINVAL; return 0; @@ -410,7 +411,8 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess, static int openssl_set_sess_aead_dec_param(struct openssl_session *sess, enum rte_crypto_aead_algorithm algo, - uint8_t tag_len, const uint8_t *key) + uint8_t tag_len, const uint8_t *key, + EVP_CIPHER_CTX **ctx) { int iv_type = 0; unsigned int do_ccm = 0; @@ -437,7 +439,7 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess, } sess->cipher.mode = OPENSSL_CIPHER_LIB; - sess->cipher.ctx = EVP_CIPHER_CTX_new(); + *ctx = EVP_CIPHER_CTX_new(); if (get_aead_algo(algo, sess->cipher.key.length, &sess->cipher.evp_algo) != 0) @@ -447,24 +449,54 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess, sess->chain_order = OPENSSL_CHAIN_COMBINED; - if (EVP_DecryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo, + if (EVP_DecryptInit_ex(*ctx, sess->cipher.evp_algo, NULL, NULL, NULL) <= 0) return -EINVAL; - if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type, + if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type, sess->iv.length, NULL) <= 0) return -EINVAL; if (do_ccm) - EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG, + EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG, tag_len, NULL); - if (EVP_DecryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0) + if (EVP_DecryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0) return -EINVAL; return 0; } +static int openssl_aesni_ctx_clone(EVP_CIPHER_CTX **dest, + struct openssl_session *sess) +{ +#if (OPENSSL_VERSION_NUMBER > 0x30200000L) + *dest = EVP_CIPHER_CTX_dup(sess->ctx); + return 0; +#elif (OPENSSL_VERSION_NUMBER > 0x30000000L) + /* OpenSSL versions 3.0.0 <= V < 3.2.0 have no dupctx() implementation + * for AES-GCM and AES-CCM. In this case, we have to create new empty + * contexts and initialise, as we did the original context. + */ + if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) + sess->aead_algo = RTE_CRYPTO_AEAD_AES_GCM; + + if (sess->cipher.direction == RTE_CRYPTO_CIPHER_OP_ENCRYPT) + return openssl_set_sess_aead_enc_param(sess, sess->aead_algo, + sess->auth.digest_length, sess->cipher.key.data, + dest); + else + return openssl_set_sess_aead_dec_param(sess, sess->aead_algo, + sess->auth.digest_length, sess->cipher.key.data, + dest); +#else + *dest = EVP_CIPHER_CTX_new(); + if (EVP_CIPHER_CTX_copy(*dest, sess->cipher.ctx) != 1) + return -EINVAL; + return 0; +#endif +} + /** Set session cipher parameters */ static int openssl_set_session_cipher_parameters(struct openssl_session *sess, @@ -623,12 +655,14 @@ openssl_set_session_auth_parameters(struct openssl_session *sess, return openssl_set_sess_aead_enc_param(sess, RTE_CRYPTO_AEAD_AES_GCM, xform->auth.digest_length, - xform->auth.key.data); + xform->auth.key.data, + &sess->cipher.ctx); else return openssl_set_sess_aead_dec_param(sess, RTE_CRYPTO_AEAD_AES_GCM, xform->auth.digest_length, - xform->auth.key.data); + xform->auth.key.data, + &sess->cipher.ctx); break; case RTE_CRYPTO_AUTH_MD5: @@ -770,10 +804,12 @@ openssl_set_session_aead_parameters(struct openssl_session *sess, /* Select cipher direction */ if (xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT) return openssl_set_sess_aead_enc_param(sess, xform->aead.algo, - xform->aead.digest_length, xform->aead.key.data); + xform->aead.digest_length, xform->aead.key.data, + &sess->cipher.ctx); else return openssl_set_sess_aead_dec_param(sess, xform->aead.algo, - xform->aead.digest_length, xform->aead.key.data); + xform->aead.digest_length, xform->aead.key.data, + &sess->cipher.ctx); } /** Parse crypto xform chain and set private session parameters */ @@ -1590,6 +1626,12 @@ process_openssl_combined_op return; } + EVP_CIPHER_CTX *ctx; + if (openssl_aesni_ctx_clone(&ctx, sess) != 0) { + op->status = RTE_CRYPTO_OP_STATUS_ERROR; + return; + } + iv = rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv.offset); if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) { @@ -1623,12 +1665,12 @@ process_openssl_combined_op status = process_openssl_auth_encryption_gcm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx); else status = process_openssl_auth_encryption_ccm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx); } else { if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC || @@ -1636,14 +1678,16 @@ process_openssl_combined_op status = process_openssl_auth_decryption_gcm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx); else status = process_openssl_auth_decryption_ccm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx); } + EVP_CIPHER_CTX_free(ctx); + if (status != 0) { if (status == (-EFAULT) && sess->auth.operation == -- 2.34.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v4 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs 2024-06-07 12:47 ` [PATCH v4 " Jack Bond-Preston @ 2024-07-01 12:55 ` Ji, Kai 2024-07-02 15:39 ` [EXTERNAL] " Akhil Goyal 1 sibling, 0 replies; 11+ messages in thread From: Ji, Kai @ 2024-07-01 12:55 UTC (permalink / raw) To: Jack Bond-Preston, Fan Zhang, Akhil Goyal; +Cc: dev, stable, Wathsala Vithanage [-- Attachment #1: Type: text/plain, Size: 12073 bytes --] Acked-by: Kai Ji <kai.ji@intel.com> ________________________________ From: Jack Bond-Preston <jack.bond-preston@foss.arm.com> Sent: 07 June 2024 13:47 To: Ji, Kai <kai.ji@intel.com>; Fan Zhang <fanzhang.oss@gmail.com>; Akhil Goyal <gakhil@marvell.com> Cc: dev@dpdk.org <dev@dpdk.org>; stable@dpdk.org <stable@dpdk.org>; Wathsala Vithanage <wathsala.vithanage@arm.com> Subject: [PATCH v4 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs Commit 67ab783b5d70 ("crypto/openssl: use local copy for session contexts") introduced a fix for concurrency bugs which could occur when using one OpenSSL PMD session across multiple cores simultaneously. The solution was to clone the EVP contexts per-buffer to avoid them being used concurrently. However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") reverted this fix, only for combined ops (AES-GCM and AES-CCM). Fix the concurrency issue by cloning EVP contexts per-buffer. An extra workaround is required for OpenSSL versions which are >= 3.0.0, and <= 3.2.0. This is because, prior to OpenSSL 3.2.0, EVP_CIPHER_CTX_copy() is not implemented for AES-GCM or AES-CCM. When using these OpenSSL versions, create and initialise the context from scratch, per-buffer. Throughput performance uplift measurements for AES-GCM-128 encrypt on Ampere Altra Max platform: 1 worker lcore | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 2.60 | 1.31 | -49.5% | | 256 | 7.69 | 4.45 | -42.1% | | 1024 | 15.33 | 11.30 | -26.3% | | 2048 | 18.74 | 15.37 | -18.0% | | 4096 | 21.11 | 18.80 | -10.9% | 8 worker lcores | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 19.94 | 2.83 | -85.8% | | 256 | 58.84 | 11.00 | -81.3% | | 1024 | 119.71 | 42.46 | -64.5% | | 2048 | 147.69 | 80.91 | -45.2% | | 4096 | 167.39 | 121.25 | -27.6% | Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") Cc: stable@dpdk.org Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com> Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com> --- drivers/crypto/openssl/rte_openssl_pmd.c | 84 ++++++++++++++++++------ 1 file changed, 64 insertions(+), 20 deletions(-) diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c index e8cb09defc..3f7f4d8c37 100644 --- a/drivers/crypto/openssl/rte_openssl_pmd.c +++ b/drivers/crypto/openssl/rte_openssl_pmd.c @@ -350,7 +350,8 @@ get_aead_algo(enum rte_crypto_aead_algorithm sess_algo, size_t keylen, static int openssl_set_sess_aead_enc_param(struct openssl_session *sess, enum rte_crypto_aead_algorithm algo, - uint8_t tag_len, const uint8_t *key) + uint8_t tag_len, const uint8_t *key, + EVP_CIPHER_CTX **ctx) { int iv_type = 0; unsigned int do_ccm; @@ -378,7 +379,7 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess, } sess->cipher.mode = OPENSSL_CIPHER_LIB; - sess->cipher.ctx = EVP_CIPHER_CTX_new(); + *ctx = EVP_CIPHER_CTX_new(); if (get_aead_algo(algo, sess->cipher.key.length, &sess->cipher.evp_algo) != 0) @@ -388,19 +389,19 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess, sess->chain_order = OPENSSL_CHAIN_COMBINED; - if (EVP_EncryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo, + if (EVP_EncryptInit_ex(*ctx, sess->cipher.evp_algo, NULL, NULL, NULL) <= 0) return -EINVAL; - if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type, sess->iv.length, + if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type, sess->iv.length, NULL) <= 0) return -EINVAL; if (do_ccm) - EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG, + EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG, tag_len, NULL); - if (EVP_EncryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0) + if (EVP_EncryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0) return -EINVAL; return 0; @@ -410,7 +411,8 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess, static int openssl_set_sess_aead_dec_param(struct openssl_session *sess, enum rte_crypto_aead_algorithm algo, - uint8_t tag_len, const uint8_t *key) + uint8_t tag_len, const uint8_t *key, + EVP_CIPHER_CTX **ctx) { int iv_type = 0; unsigned int do_ccm = 0; @@ -437,7 +439,7 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess, } sess->cipher.mode = OPENSSL_CIPHER_LIB; - sess->cipher.ctx = EVP_CIPHER_CTX_new(); + *ctx = EVP_CIPHER_CTX_new(); if (get_aead_algo(algo, sess->cipher.key.length, &sess->cipher.evp_algo) != 0) @@ -447,24 +449,54 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess, sess->chain_order = OPENSSL_CHAIN_COMBINED; - if (EVP_DecryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo, + if (EVP_DecryptInit_ex(*ctx, sess->cipher.evp_algo, NULL, NULL, NULL) <= 0) return -EINVAL; - if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type, + if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type, sess->iv.length, NULL) <= 0) return -EINVAL; if (do_ccm) - EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG, + EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG, tag_len, NULL); - if (EVP_DecryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0) + if (EVP_DecryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0) return -EINVAL; return 0; } +static int openssl_aesni_ctx_clone(EVP_CIPHER_CTX **dest, + struct openssl_session *sess) +{ +#if (OPENSSL_VERSION_NUMBER > 0x30200000L) + *dest = EVP_CIPHER_CTX_dup(sess->ctx); + return 0; +#elif (OPENSSL_VERSION_NUMBER > 0x30000000L) + /* OpenSSL versions 3.0.0 <= V < 3.2.0 have no dupctx() implementation + * for AES-GCM and AES-CCM. In this case, we have to create new empty + * contexts and initialise, as we did the original context. + */ + if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) + sess->aead_algo = RTE_CRYPTO_AEAD_AES_GCM; + + if (sess->cipher.direction == RTE_CRYPTO_CIPHER_OP_ENCRYPT) + return openssl_set_sess_aead_enc_param(sess, sess->aead_algo, + sess->auth.digest_length, sess->cipher.key.data, + dest); + else + return openssl_set_sess_aead_dec_param(sess, sess->aead_algo, + sess->auth.digest_length, sess->cipher.key.data, + dest); +#else + *dest = EVP_CIPHER_CTX_new(); + if (EVP_CIPHER_CTX_copy(*dest, sess->cipher.ctx) != 1) + return -EINVAL; + return 0; +#endif +} + /** Set session cipher parameters */ static int openssl_set_session_cipher_parameters(struct openssl_session *sess, @@ -623,12 +655,14 @@ openssl_set_session_auth_parameters(struct openssl_session *sess, return openssl_set_sess_aead_enc_param(sess, RTE_CRYPTO_AEAD_AES_GCM, xform->auth.digest_length, - xform->auth.key.data); + xform->auth.key.data, + &sess->cipher.ctx); else return openssl_set_sess_aead_dec_param(sess, RTE_CRYPTO_AEAD_AES_GCM, xform->auth.digest_length, - xform->auth.key.data); + xform->auth.key.data, + &sess->cipher.ctx); break; case RTE_CRYPTO_AUTH_MD5: @@ -770,10 +804,12 @@ openssl_set_session_aead_parameters(struct openssl_session *sess, /* Select cipher direction */ if (xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT) return openssl_set_sess_aead_enc_param(sess, xform->aead.algo, - xform->aead.digest_length, xform->aead.key.data); + xform->aead.digest_length, xform->aead.key.data, + &sess->cipher.ctx); else return openssl_set_sess_aead_dec_param(sess, xform->aead.algo, - xform->aead.digest_length, xform->aead.key.data); + xform->aead.digest_length, xform->aead.key.data, + &sess->cipher.ctx); } /** Parse crypto xform chain and set private session parameters */ @@ -1590,6 +1626,12 @@ process_openssl_combined_op return; } + EVP_CIPHER_CTX *ctx; + if (openssl_aesni_ctx_clone(&ctx, sess) != 0) { + op->status = RTE_CRYPTO_OP_STATUS_ERROR; + return; + } + iv = rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv.offset); if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) { @@ -1623,12 +1665,12 @@ process_openssl_combined_op status = process_openssl_auth_encryption_gcm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx); else status = process_openssl_auth_encryption_ccm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx); } else { if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC || @@ -1636,14 +1678,16 @@ process_openssl_combined_op status = process_openssl_auth_decryption_gcm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx); else status = process_openssl_auth_decryption_ccm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx); } + EVP_CIPHER_CTX_free(ctx); + if (status != 0) { if (status == (-EFAULT) && sess->auth.operation == -- 2.34.1 [-- Attachment #2: Type: text/html, Size: 29860 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [EXTERNAL] [PATCH v4 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs 2024-06-07 12:47 ` [PATCH v4 " Jack Bond-Preston 2024-07-01 12:55 ` Ji, Kai @ 2024-07-02 15:39 ` Akhil Goyal 2024-07-03 10:49 ` Jack Bond-Preston 1 sibling, 1 reply; 11+ messages in thread From: Akhil Goyal @ 2024-07-02 15:39 UTC (permalink / raw) To: Jack Bond-Preston, Kai Ji, Fan Zhang; +Cc: dev, stable, Wathsala Vithanage > Commit 67ab783b5d70 ("crypto/openssl: use local copy for session > contexts") introduced a fix for concurrency bugs which could occur when > using one OpenSSL PMD session across multiple cores simultaneously. The > solution was to clone the EVP contexts per-buffer to avoid them being > used concurrently. > > However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC > routine with 3.0 EVP API") reverted this fix, only for combined ops > (AES-GCM and AES-CCM). > > Fix the concurrency issue by cloning EVP contexts per-buffer. An extra > workaround is required for OpenSSL versions which are >= 3.0.0, and > <= 3.2.0. This is because, prior to OpenSSL 3.2.0, EVP_CIPHER_CTX_copy() > is not implemented for AES-GCM or AES-CCM. When using these OpenSSL > versions, create and initialise the context from scratch, per-buffer. > > Throughput performance uplift measurements for AES-GCM-128 encrypt on > Ampere Altra Max platform: > 1 worker lcore > | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | > |-----------------+---------------+--------------------+----------| > | 64 | 2.60 | 1.31 | -49.5% | > | 256 | 7.69 | 4.45 | -42.1% | > | 1024 | 15.33 | 11.30 | -26.3% | > | 2048 | 18.74 | 15.37 | -18.0% | > | 4096 | 21.11 | 18.80 | -10.9% | > > 8 worker lcores > | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | > |-----------------+---------------+--------------------+----------| > | 64 | 19.94 | 2.83 | -85.8% | > | 256 | 58.84 | 11.00 | -81.3% | > | 1024 | 119.71 | 42.46 | -64.5% | > | 2048 | 147.69 | 80.91 | -45.2% | > | 4096 | 167.39 | 121.25 | -27.6% | > > Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") > Cc: stable@dpdk.org > Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com> > Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com> I am seeing below errors when it is compiled with openssl 3.0 [1/30] Compiling C object 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd_ops.c.o'. FAILED: drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd_ops.c.o ccache gcc -Idrivers/a715181@@tmp_rte_crypto_openssl@sta -Idrivers -I../drivers -Idrivers/crypto/openssl -I../drivers/crypto/openssl -Ilib/cryptodev -I../lib/cryptodev -I. -I../ -Iconfig -I../config -Ilib/eal/include -I../lib/eal/include -Ilib/eal/linux/include -I../lib/eal/linux/include -Ilib/eal/x86/include -I../lib/eal/x86/include -Ilib/eal/common -I../lib/eal/common -Ilib/eal -I../lib/eal -Ilib/kvargs -I../lib/kvargs -Ilib/log -I../lib/log -Ilib/telemetry/../metrics -I../lib/telemetry/../metrics -Ilib/telemetry -I../lib/telemetry -Ilib/mbuf -I../lib/mbuf -Ilib/mempool -I../lib/mempool -Ilib/ring -I../lib/ring -Ilib/rcu -I../lib/rcu -Idrivers/bus/vdev -I../drivers/bus/vdev -I/usr/local/include -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Werror -std=c11 -O2 -g -include rte_config.h -Wcast-qual -Wdeprecated -Wformat -Wformat-nonliteral -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -Wwrite-strings -Wno-address-of-packed-member -Wno-packed-not-aligned -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native -mrtm -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -Wno-format-truncation -DRTE_LOG_DEFAULT_LOGTYPE=pmd.crypto.openssl -MD -MQ 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd_ops.c.o' -MF 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd_ops.c.o.d' -o 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd_ops.c.o' -c ../drivers/crypto/openssl/rte_openssl_pmd_ops.c In file included from ../drivers/crypto/openssl/rte_openssl_pmd_ops.c:12: ../drivers/crypto/openssl/compat.h: In function 'free_hmac_ctx': ../drivers/crypto/openssl/compat.h:24:2: error: 'HMAC_CTX_free' is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations] 24 | HMAC_CTX_free(ctx); | ^~~~~~~~~~~~~ In file included from ../drivers/crypto/openssl/openssl_pmd_private.h:10, from ../drivers/crypto/openssl/rte_openssl_pmd_ops.c:11: /usr/local/include/openssl/hmac.h:35:28: note: declared here 35 | OSSL_DEPRECATEDIN_3_0 void HMAC_CTX_free(HMAC_CTX *ctx); | ^~~~~~~~~~~~~ In file included from ../drivers/crypto/openssl/rte_openssl_pmd_ops.c:12: ../drivers/crypto/openssl/compat.h: In function 'free_cmac_ctx': ../drivers/crypto/openssl/compat.h:30:2: error: 'CMAC_CTX_free' is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations] 30 | CMAC_CTX_free(ctx); | ^~~~~~~~~~~~~ In file included from ../drivers/crypto/openssl/openssl_pmd_private.h:9, from ../drivers/crypto/openssl/rte_openssl_pmd_ops.c:11: /usr/local/include/openssl/cmac.h:34:28: note: declared here 34 | OSSL_DEPRECATEDIN_3_0 void CMAC_CTX_free(CMAC_CTX *ctx); | ^~~~~~~~~~~~~ cc1: all warnings being treated as errors [2/30] Compiling C object 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd.c.o'. FAILED: drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd.c.o ccache gcc -Idrivers/a715181@@tmp_rte_crypto_openssl@sta -Idrivers -I../drivers -Idrivers/crypto/openssl -I../drivers/crypto/openssl -Ilib/cryptodev -I../lib/cryptodev -I. -I../ -Iconfig -I../config -Ilib/eal/include -I../lib/eal/include -Ilib/eal/linux/include -I../lib/eal/linux/include -Ilib/eal/x86/include -I../lib/eal/x86/include -Ilib/eal/common -I../lib/eal/common -Ilib/eal -I../lib/eal -Ilib/kvargs -I../lib/kvargs -Ilib/log -I../lib/log -Ilib/telemetry/../metrics -I../lib/telemetry/../metrics -Ilib/telemetry -I../lib/telemetry -Ilib/mbuf -I../lib/mbuf -Ilib/mempool -I../lib/mempool -Ilib/ring -I../lib/ring -Ilib/rcu -I../lib/rcu -Idrivers/bus/vdev -I../drivers/bus/vdev -I/usr/local/include -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Werror -std=c11 -O2 -g -include rte_config.h -Wcast-qual -Wdeprecated -Wformat -Wformat-nonliteral -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -Wwrite-strings -Wno-address-of-packed-member -Wno-packed-not-aligned -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native -mrtm -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -Wno-format-truncation -DRTE_LOG_DEFAULT_LOGTYPE=pmd.crypto.openssl -MD -MQ 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd.c.o' -MF 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd.c.o.d' -o 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd.c.o' -c ../drivers/crypto/openssl/rte_openssl_pmd.c In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:19: ../drivers/crypto/openssl/compat.h: In function 'free_hmac_ctx': ../drivers/crypto/openssl/compat.h:24:2: error: 'HMAC_CTX_free' is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations] 24 | HMAC_CTX_free(ctx); | ^~~~~~~~~~~~~ In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:14: /usr/local/include/openssl/hmac.h:35:28: note: declared here 35 | OSSL_DEPRECATEDIN_3_0 void HMAC_CTX_free(HMAC_CTX *ctx); | ^~~~~~~~~~~~~ In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:19: ../drivers/crypto/openssl/compat.h: In function 'free_cmac_ctx': ../drivers/crypto/openssl/compat.h:30:2: error: 'CMAC_CTX_free' is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations] 30 | CMAC_CTX_free(ctx); | ^~~~~~~~~~~~~ In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:13: /usr/local/include/openssl/cmac.h:34:28: note: declared here 34 | OSSL_DEPRECATEDIN_3_0 void CMAC_CTX_free(CMAC_CTX *ctx); | ^~~~~~~~~~~~~ ../drivers/crypto/openssl/rte_openssl_pmd.c: In function 'openssl_reset_session': ../drivers/crypto/openssl/rte_openssl_pmd.c:908:33: error: passing argument 1 of 'free_hmac_ctx' from incompatible pointer type [-Werror=incompatible-pointer-types] 908 | free_hmac_ctx(sess->qp_ctx[i].hmac); | ~~~~~~~~~~~~~~~^~~~~ | | | EVP_MAC_CTX * {aka struct evp_mac_ctx_st *} In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:19: ../drivers/crypto/openssl/compat.h:22:25: note: expected 'HMAC_CTX *' {aka 'struct hmac_ctx_st *'} but argument is of type 'EVP_MAC_CTX *' {aka 'struct evp_mac_ctx_st *'} 22 | free_hmac_ctx(HMAC_CTX *ctx) | ~~~~~~~~~~^~~ ../drivers/crypto/openssl/rte_openssl_pmd.c:912:33: error: passing argument 1 of 'free_cmac_ctx' from incompatible pointer type [-Werror=incompatible-pointer-types] 912 | free_cmac_ctx(sess->qp_ctx[i].cmac); | ~~~~~~~~~~~~~~~^~~~~ | | | EVP_MAC_CTX * {aka struct evp_mac_ctx_st *} In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:19: ../drivers/crypto/openssl/compat.h:28:25: note: expected 'CMAC_CTX *' {aka 'struct CMAC_CTX_st *'} but argument is of type 'EVP_MAC_CTX *' {aka 'struct evp_mac_ctx_st *'} 28 | free_cmac_ctx(CMAC_CTX *ctx) | ~~~~~~~~~~^~~ ../drivers/crypto/openssl/rte_openssl_pmd.c:925:32: error: passing argument 1 of 'free_hmac_ctx' from incompatible pointer type [-Werror=incompatible-pointer-types] 925 | free_hmac_ctx(sess->auth.hmac.ctx); | ~~~~~~~~~~~~~~~^~~~ | | | EVP_MAC_CTX * {aka struct evp_mac_ctx_st *} In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:19: ../drivers/crypto/openssl/compat.h:22:25: note: expected 'HMAC_CTX *' {aka 'struct hmac_ctx_st *'} but argument is of type 'EVP_MAC_CTX *' {aka 'struct evp_mac_ctx_st *'} 22 | free_hmac_ctx(HMAC_CTX *ctx) | ~~~~~~~~~~^~~ ../drivers/crypto/openssl/rte_openssl_pmd.c:928:32: error: passing argument 1 of 'free_cmac_ctx' from incompatible pointer type [-Werror=incompatible-pointer-types] 928 | free_cmac_ctx(sess->auth.cmac.ctx); | ~~~~~~~~~~~~~~~^~~~ | | | EVP_MAC_CTX * {aka struct evp_mac_ctx_st *} In file included from ../drivers/crypto/openssl/rte_openssl_pmd.c:19: ../drivers/crypto/openssl/compat.h:28:25: note: expected 'CMAC_CTX *' {aka 'struct CMAC_CTX_st *'} but argument is of type 'EVP_MAC_CTX *' {aka 'struct evp_mac_ctx_st *'} 28 | free_cmac_ctx(CMAC_CTX *ctx) | ~~~~~~~~~~^~~ cc1: all warnings being treated as errors ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [EXTERNAL] [PATCH v4 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs 2024-07-02 15:39 ` [EXTERNAL] " Akhil Goyal @ 2024-07-03 10:49 ` Jack Bond-Preston 0 siblings, 0 replies; 11+ messages in thread From: Jack Bond-Preston @ 2024-07-03 10:49 UTC (permalink / raw) To: Akhil Goyal, Kai Ji, Fan Zhang; +Cc: dev, stable, Wathsala Vithanage On 02/07/2024 16:39, Akhil Goyal wrote: > > I am seeing below errors when it is compiled with openssl 3.0 > > [1/30] Compiling C object 'drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd_ops.c.o'. > FAILED: drivers/a715181@@tmp_rte_crypto_openssl@sta/crypto_openssl_rte_openssl_pmd_ops.c.o > <snip> Ah, that's my mistake. The OpenSSL version comparison macros should be >= 3.0.0, not > (in patches 1/5 and 3/5). I will push the amended patchset ASAP. ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <20240703134552.1439633-1-jack.bond-preston@foss.arm.com>]
* [PATCH v5 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs [not found] ` <20240703134552.1439633-1-jack.bond-preston@foss.arm.com> @ 2024-07-03 13:45 ` Jack Bond-Preston 2024-07-03 15:20 ` [EXTERNAL] " Akhil Goyal 0 siblings, 1 reply; 11+ messages in thread From: Jack Bond-Preston @ 2024-07-03 13:45 UTC (permalink / raw) To: Kai Ji, Fan Zhang, Akhil Goyal; +Cc: dev, stable, Wathsala Vithanage Commit 67ab783b5d70 ("crypto/openssl: use local copy for session contexts") introduced a fix for concurrency bugs which could occur when using one OpenSSL PMD session across multiple cores simultaneously. The solution was to clone the EVP contexts per-buffer to avoid them being used concurrently. However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") reverted this fix, only for combined ops (AES-GCM and AES-CCM). Fix the concurrency issue by cloning EVP contexts per-buffer. An extra workaround is required for OpenSSL versions which are >= 3.0.0, and <= 3.2.0. This is because, prior to OpenSSL 3.2.0, EVP_CIPHER_CTX_copy() is not implemented for AES-GCM or AES-CCM. When using these OpenSSL versions, create and initialise the context from scratch, per-buffer. Throughput performance uplift measurements for AES-GCM-128 encrypt on Ampere Altra Max platform: 1 worker lcore | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 2.60 | 1.31 | -49.5% | | 256 | 7.69 | 4.45 | -42.1% | | 1024 | 15.33 | 11.30 | -26.3% | | 2048 | 18.74 | 15.37 | -18.0% | | 4096 | 21.11 | 18.80 | -10.9% | 8 worker lcores | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | |-----------------+---------------+--------------------+----------| | 64 | 19.94 | 2.83 | -85.8% | | 256 | 58.84 | 11.00 | -81.3% | | 1024 | 119.71 | 42.46 | -64.5% | | 2048 | 147.69 | 80.91 | -45.2% | | 4096 | 167.39 | 121.25 | -27.6% | Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") Cc: stable@dpdk.org Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com> Acked-by: Kai Ji <kai.ji@intel.com> Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com> --- drivers/crypto/openssl/rte_openssl_pmd.c | 84 ++++++++++++++++++------ 1 file changed, 64 insertions(+), 20 deletions(-) diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c index e8cb09defc..c661528738 100644 --- a/drivers/crypto/openssl/rte_openssl_pmd.c +++ b/drivers/crypto/openssl/rte_openssl_pmd.c @@ -350,7 +350,8 @@ get_aead_algo(enum rte_crypto_aead_algorithm sess_algo, size_t keylen, static int openssl_set_sess_aead_enc_param(struct openssl_session *sess, enum rte_crypto_aead_algorithm algo, - uint8_t tag_len, const uint8_t *key) + uint8_t tag_len, const uint8_t *key, + EVP_CIPHER_CTX **ctx) { int iv_type = 0; unsigned int do_ccm; @@ -378,7 +379,7 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess, } sess->cipher.mode = OPENSSL_CIPHER_LIB; - sess->cipher.ctx = EVP_CIPHER_CTX_new(); + *ctx = EVP_CIPHER_CTX_new(); if (get_aead_algo(algo, sess->cipher.key.length, &sess->cipher.evp_algo) != 0) @@ -388,19 +389,19 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess, sess->chain_order = OPENSSL_CHAIN_COMBINED; - if (EVP_EncryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo, + if (EVP_EncryptInit_ex(*ctx, sess->cipher.evp_algo, NULL, NULL, NULL) <= 0) return -EINVAL; - if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type, sess->iv.length, + if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type, sess->iv.length, NULL) <= 0) return -EINVAL; if (do_ccm) - EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG, + EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG, tag_len, NULL); - if (EVP_EncryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0) + if (EVP_EncryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0) return -EINVAL; return 0; @@ -410,7 +411,8 @@ openssl_set_sess_aead_enc_param(struct openssl_session *sess, static int openssl_set_sess_aead_dec_param(struct openssl_session *sess, enum rte_crypto_aead_algorithm algo, - uint8_t tag_len, const uint8_t *key) + uint8_t tag_len, const uint8_t *key, + EVP_CIPHER_CTX **ctx) { int iv_type = 0; unsigned int do_ccm = 0; @@ -437,7 +439,7 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess, } sess->cipher.mode = OPENSSL_CIPHER_LIB; - sess->cipher.ctx = EVP_CIPHER_CTX_new(); + *ctx = EVP_CIPHER_CTX_new(); if (get_aead_algo(algo, sess->cipher.key.length, &sess->cipher.evp_algo) != 0) @@ -447,24 +449,54 @@ openssl_set_sess_aead_dec_param(struct openssl_session *sess, sess->chain_order = OPENSSL_CHAIN_COMBINED; - if (EVP_DecryptInit_ex(sess->cipher.ctx, sess->cipher.evp_algo, + if (EVP_DecryptInit_ex(*ctx, sess->cipher.evp_algo, NULL, NULL, NULL) <= 0) return -EINVAL; - if (EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, iv_type, + if (EVP_CIPHER_CTX_ctrl(*ctx, iv_type, sess->iv.length, NULL) <= 0) return -EINVAL; if (do_ccm) - EVP_CIPHER_CTX_ctrl(sess->cipher.ctx, EVP_CTRL_CCM_SET_TAG, + EVP_CIPHER_CTX_ctrl(*ctx, EVP_CTRL_CCM_SET_TAG, tag_len, NULL); - if (EVP_DecryptInit_ex(sess->cipher.ctx, NULL, NULL, key, NULL) <= 0) + if (EVP_DecryptInit_ex(*ctx, NULL, NULL, key, NULL) <= 0) return -EINVAL; return 0; } +static int openssl_aesni_ctx_clone(EVP_CIPHER_CTX **dest, + struct openssl_session *sess) +{ +#if (OPENSSL_VERSION_NUMBER >= 0x30200000L) + *dest = EVP_CIPHER_CTX_dup(sess->ctx); + return 0; +#elif (OPENSSL_VERSION_NUMBER >= 0x30000000L) + /* OpenSSL versions 3.0.0 <= V < 3.2.0 have no dupctx() implementation + * for AES-GCM and AES-CCM. In this case, we have to create new empty + * contexts and initialise, as we did the original context. + */ + if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) + sess->aead_algo = RTE_CRYPTO_AEAD_AES_GCM; + + if (sess->cipher.direction == RTE_CRYPTO_CIPHER_OP_ENCRYPT) + return openssl_set_sess_aead_enc_param(sess, sess->aead_algo, + sess->auth.digest_length, sess->cipher.key.data, + dest); + else + return openssl_set_sess_aead_dec_param(sess, sess->aead_algo, + sess->auth.digest_length, sess->cipher.key.data, + dest); +#else + *dest = EVP_CIPHER_CTX_new(); + if (EVP_CIPHER_CTX_copy(*dest, sess->cipher.ctx) != 1) + return -EINVAL; + return 0; +#endif +} + /** Set session cipher parameters */ static int openssl_set_session_cipher_parameters(struct openssl_session *sess, @@ -623,12 +655,14 @@ openssl_set_session_auth_parameters(struct openssl_session *sess, return openssl_set_sess_aead_enc_param(sess, RTE_CRYPTO_AEAD_AES_GCM, xform->auth.digest_length, - xform->auth.key.data); + xform->auth.key.data, + &sess->cipher.ctx); else return openssl_set_sess_aead_dec_param(sess, RTE_CRYPTO_AEAD_AES_GCM, xform->auth.digest_length, - xform->auth.key.data); + xform->auth.key.data, + &sess->cipher.ctx); break; case RTE_CRYPTO_AUTH_MD5: @@ -770,10 +804,12 @@ openssl_set_session_aead_parameters(struct openssl_session *sess, /* Select cipher direction */ if (xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT) return openssl_set_sess_aead_enc_param(sess, xform->aead.algo, - xform->aead.digest_length, xform->aead.key.data); + xform->aead.digest_length, xform->aead.key.data, + &sess->cipher.ctx); else return openssl_set_sess_aead_dec_param(sess, xform->aead.algo, - xform->aead.digest_length, xform->aead.key.data); + xform->aead.digest_length, xform->aead.key.data, + &sess->cipher.ctx); } /** Parse crypto xform chain and set private session parameters */ @@ -1590,6 +1626,12 @@ process_openssl_combined_op return; } + EVP_CIPHER_CTX *ctx; + if (openssl_aesni_ctx_clone(&ctx, sess) != 0) { + op->status = RTE_CRYPTO_OP_STATUS_ERROR; + return; + } + iv = rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv.offset); if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) { @@ -1623,12 +1665,12 @@ process_openssl_combined_op status = process_openssl_auth_encryption_gcm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx); else status = process_openssl_auth_encryption_ccm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx); } else { if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC || @@ -1636,14 +1678,16 @@ process_openssl_combined_op status = process_openssl_auth_decryption_gcm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx); else status = process_openssl_auth_decryption_ccm( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx); } + EVP_CIPHER_CTX_free(ctx); + if (status != 0) { if (status == (-EFAULT) && sess->auth.operation == -- 2.34.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [EXTERNAL] [PATCH v5 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs 2024-07-03 13:45 ` [PATCH v5 " Jack Bond-Preston @ 2024-07-03 15:20 ` Akhil Goyal 0 siblings, 0 replies; 11+ messages in thread From: Akhil Goyal @ 2024-07-03 15:20 UTC (permalink / raw) To: Jack Bond-Preston, Kai Ji, Fan Zhang; +Cc: dev, stable, Wathsala Vithanage > Subject: [EXTERNAL] [PATCH v5 1/5] crypto/openssl: fix GCM and CCM thread > unsafe ctxs > > Commit 67ab783b5d70 ("crypto/openssl: use local copy for session > contexts") introduced a fix for concurrency bugs which could occur when > using one OpenSSL PMD session across multiple cores simultaneously. The > solution was to clone the EVP contexts per-buffer to avoid them being > used concurrently. > > However, part of commit 75adf1eae44f ("crypto/openssl: update HMAC > routine with 3.0 EVP API") reverted this fix, only for combined ops > (AES-GCM and AES-CCM). > > Fix the concurrency issue by cloning EVP contexts per-buffer. An extra > workaround is required for OpenSSL versions which are >= 3.0.0, and > <= 3.2.0. This is because, prior to OpenSSL 3.2.0, EVP_CIPHER_CTX_copy() > is not implemented for AES-GCM or AES-CCM. When using these OpenSSL > versions, create and initialise the context from scratch, per-buffer. > > Throughput performance uplift measurements for AES-GCM-128 encrypt on > Ampere Altra Max platform: > 1 worker lcore > | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | > |-----------------+---------------+--------------------+----------| > | 64 | 2.60 | 1.31 | -49.5% | > | 256 | 7.69 | 4.45 | -42.1% | > | 1024 | 15.33 | 11.30 | -26.3% | > | 2048 | 18.74 | 15.37 | -18.0% | > | 4096 | 21.11 | 18.80 | -10.9% | > > 8 worker lcores > | buffer sz (B) | prev (Gbps) | optimised (Gbps) | uplift | > |-----------------+---------------+--------------------+----------| > | 64 | 19.94 | 2.83 | -85.8% | > | 256 | 58.84 | 11.00 | -81.3% | > | 1024 | 119.71 | 42.46 | -64.5% | > | 2048 | 147.69 | 80.91 | -45.2% | > | 4096 | 167.39 | 121.25 | -27.6% | > > Fixes: 75adf1eae44f ("crypto/openssl: update HMAC routine with 3.0 EVP API") > Cc: stable@dpdk.org Since this fix is reducing the performance significantly. Backporting it alone would reduce performance on stable releases. I would suggest applying all the patches in this series for backports. > Signed-off-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com> > Acked-by: Kai Ji <kai.ji@intel.com> > Reviewed-by: Wathsala Vithanage <wathsala.vithanage@arm.com> Applied to dpdk-next-crypto Thanks for the optimizations. ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2024-07-03 15:20 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20240603160119.1279476-1-jack.bond-preston@foss.arm.com>
2024-06-03 16:01 ` [PATCH 1/5] crypto/openssl: fix GCM and CCM thread unsafe ctxs Jack Bond-Preston
[not found] ` <20240603184348.1310331-1-jack.bond-preston@foss.arm.com>
2024-06-03 18:43 ` [PATCH v2 " Jack Bond-Preston
[not found] ` <20240603185939.1312680-1-jack.bond-preston@foss.arm.com>
2024-06-03 18:59 ` Jack Bond-Preston
[not found] ` <20240606102043.2926695-1-jack.bond-preston@foss.arm.com>
2024-06-06 10:20 ` [PATCH v3 " Jack Bond-Preston
2024-06-06 10:44 ` [EXTERNAL] " Akhil Goyal
[not found] ` <20240607124756.3968704-1-jack.bond-preston@foss.arm.com>
2024-06-07 12:47 ` [PATCH v4 " Jack Bond-Preston
2024-07-01 12:55 ` Ji, Kai
2024-07-02 15:39 ` [EXTERNAL] " Akhil Goyal
2024-07-03 10:49 ` Jack Bond-Preston
[not found] ` <20240703134552.1439633-1-jack.bond-preston@foss.arm.com>
2024-07-03 13:45 ` [PATCH v5 " Jack Bond-Preston
2024-07-03 15:20 ` [EXTERNAL] " Akhil Goyal
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).