From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 97273A0C45; Fri, 15 Oct 2021 16:41:51 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 83A13412A2; Fri, 15 Oct 2021 16:40:49 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id DBD144129B for ; Fri, 15 Oct 2021 16:40:46 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816198" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816198" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542910" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:43 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu, =?UTF-8?q?Ciara=C2=A0Power?= , Thomas Monjalon Date: Fri, 15 Oct 2021 14:39:55 +0000 Message-Id: <20211015143957.842499-13-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v4 12/14] drivers/crypto: move zuc PMD to IPsec-mb framework X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Piotr Bronowski This patch removes the crypto/zuc folder and gathers all zuc PMD implementation specific details into two files, pmd_zuc.c and pmd_zuc_priv.h in the crypto/ipsec_mb folder. Signed-off-by: Piotr Bronowski Signed-off-by: Ciara Power Acked-by: Ray Kinsella --- v4: - Fixed some function names. - Removed unnecessary enqueued counts from process_ops function. - Added release note. - Removed security includes. - Split into private header file. v2: Updated maintainers file. --- MAINTAINERS | 8 +- doc/guides/cryptodevs/zuc.rst | 3 +- doc/guides/rel_notes/release_21_11.rst | 1 + drivers/crypto/ipsec_mb/ipsec_mb_private.h | 7 + drivers/crypto/ipsec_mb/meson.build | 3 +- drivers/crypto/ipsec_mb/pmd_zuc.c | 403 ++++++++++++++ drivers/crypto/ipsec_mb/pmd_zuc_priv.h | 85 +++ drivers/crypto/meson.build | 1 - drivers/crypto/zuc/meson.build | 24 - drivers/crypto/zuc/rte_zuc_pmd.c | 583 --------------------- drivers/crypto/zuc/rte_zuc_pmd_ops.c | 322 ------------ drivers/crypto/zuc/version.map | 3 - drivers/crypto/zuc/zuc_pmd_private.h | 83 --- 13 files changed, 502 insertions(+), 1024 deletions(-) create mode 100644 drivers/crypto/ipsec_mb/pmd_zuc.c create mode 100644 drivers/crypto/ipsec_mb/pmd_zuc_priv.h delete mode 100644 drivers/crypto/zuc/meson.build delete mode 100644 drivers/crypto/zuc/rte_zuc_pmd.c delete mode 100644 drivers/crypto/zuc/rte_zuc_pmd_ops.c delete mode 100644 drivers/crypto/zuc/version.map delete mode 100644 drivers/crypto/zuc/zuc_pmd_private.h diff --git a/MAINTAINERS b/MAINTAINERS index cff55e4525..1c269bd991 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1061,10 +1061,12 @@ F: doc/guides/cryptodevs/aesni_gcm.rst F: doc/guides/cryptodevs/aesni_mb.rst F: doc/guides/cryptodevs/kasumi.rst F: doc/guides/cryptodevs/snow3g.rst +F: doc/guides/cryptodevs/zuc.rst F: doc/guides/cryptodevs/features/aesni_gcm.ini F: doc/guides/cryptodevs/features/aesni_mb.ini F: doc/guides/cryptodevs/features/kasumi.ini F: doc/guides/cryptodevs/features/snow3g.ini +F: doc/guides/cryptodevs/features/zuc.ini Marvell cnxk crypto M: Ankur Dwivedi @@ -1141,12 +1143,6 @@ F: drivers/crypto/virtio/ F: doc/guides/cryptodevs/virtio.rst F: doc/guides/cryptodevs/features/virtio.ini -ZUC -M: Pablo de Lara -F: drivers/crypto/zuc/ -F: doc/guides/cryptodevs/zuc.rst -F: doc/guides/cryptodevs/features/zuc.ini - Compression Drivers ------------------- diff --git a/doc/guides/cryptodevs/zuc.rst b/doc/guides/cryptodevs/zuc.rst index 988a79bc26..8c925665b5 100644 --- a/doc/guides/cryptodevs/zuc.rst +++ b/doc/guides/cryptodevs/zuc.rst @@ -77,7 +77,8 @@ and the external crypto libraries supported by them: DPDK version Crypto library version ============= ================================ 16.11 - 19.11 LibSSO ZUC - 20.02+ Multi-buffer library 0.53 - 1.0* + 20.02 - 21.08 Multi-buffer library 0.53 - 1.0* + 21.11+ Multi-buffer library 1.0* ============= ================================ \* Multi-buffer library 1.0 or newer only works for Meson but not Make build system. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 3ba3aad180..52ac148884 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -108,6 +108,7 @@ New Features * AESNI_GCM PMD. * KASUMI PMD. * SNOW3G PMD. + * ZUC PMD. * **Updated the aesni_mb crypto PMD.** diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_private.h b/drivers/crypto/ipsec_mb/ipsec_mb_private.h index 9ee23d8fb5..fef3fcab68 100644 --- a/drivers/crypto/ipsec_mb/ipsec_mb_private.h +++ b/drivers/crypto/ipsec_mb/ipsec_mb_private.h @@ -46,6 +46,9 @@ extern RTE_DEFINE_PER_LCORE(IMB_MGR *, mb_mgr); #define CRYPTODEV_NAME_SNOW3G_PMD crypto_snow3g /**< IPSEC Multi buffer PMD snow3g device name */ +#define CRYPTODEV_NAME_ZUC_PMD crypto_zuc +/**< IPSEC Multi buffer PMD zuc device name */ + /** PMD LOGTYPE DRIVER, common to all PMDs */ extern int ipsec_mb_logtype_driver; #define IPSEC_MB_LOG(level, fmt, ...) \ @@ -58,6 +61,7 @@ enum ipsec_mb_pmd_types { IPSEC_MB_PMD_TYPE_AESNI_GCM, IPSEC_MB_PMD_TYPE_KASUMI, IPSEC_MB_PMD_TYPE_SNOW3G, + IPSEC_MB_PMD_TYPE_ZUC, IPSEC_MB_N_PMD_TYPES }; @@ -80,6 +84,7 @@ extern uint8_t pmd_driver_id_aesni_mb; extern uint8_t pmd_driver_id_aesni_gcm; extern uint8_t pmd_driver_id_kasumi; extern uint8_t pmd_driver_id_snow3g; +extern uint8_t pmd_driver_id_zuc; /** Helper function. Gets driver ID based on PMD type */ static __rte_always_inline uint8_t @@ -94,6 +99,8 @@ ipsec_mb_get_driver_id(enum ipsec_mb_pmd_types pmd_type) return pmd_driver_id_kasumi; case IPSEC_MB_PMD_TYPE_SNOW3G: return pmd_driver_id_snow3g; + case IPSEC_MB_PMD_TYPE_ZUC: + return pmd_driver_id_zuc; default: break; } diff --git a/drivers/crypto/ipsec_mb/meson.build b/drivers/crypto/ipsec_mb/meson.build index 0c074d789d..c712c2adb1 100644 --- a/drivers/crypto/ipsec_mb/meson.build +++ b/drivers/crypto/ipsec_mb/meson.build @@ -26,6 +26,7 @@ sources = files('ipsec_mb_private.c', 'pmd_aesni_mb.c', 'pmd_aesni_gcm.c', 'pmd_kasumi.c', - 'pmd_snow3g.c' + 'pmd_snow3g.c', + 'pmd_zuc.c' ) deps += ['bus_vdev', 'net', 'security'] diff --git a/drivers/crypto/ipsec_mb/pmd_zuc.c b/drivers/crypto/ipsec_mb/pmd_zuc.c new file mode 100644 index 0000000000..b542264069 --- /dev/null +++ b/drivers/crypto/ipsec_mb/pmd_zuc.c @@ -0,0 +1,403 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2016-2021 Intel Corporation + */ + +#include "pmd_zuc_priv.h" + +/** Parse crypto xform chain and set private session parameters. */ +static int +zuc_session_configure(__rte_unused IMB_MGR * mgr, void *zuc_sess, + const struct rte_crypto_sym_xform *xform) +{ + struct zuc_session *sess = (struct zuc_session *) zuc_sess; + const struct rte_crypto_sym_xform *auth_xform = NULL; + const struct rte_crypto_sym_xform *cipher_xform = NULL; + enum ipsec_mb_operation mode; + /* Select Crypto operation - hash then cipher / cipher then hash */ + int ret = ipsec_mb_parse_xform(xform, &mode, &auth_xform, + &cipher_xform, NULL); + + if (ret) + return ret; + + if (cipher_xform) { + /* Only ZUC EEA3 supported */ + if (cipher_xform->cipher.algo != RTE_CRYPTO_CIPHER_ZUC_EEA3) + return -ENOTSUP; + + if (cipher_xform->cipher.iv.length != ZUC_IV_KEY_LENGTH) { + IPSEC_MB_LOG(ERR, "Wrong IV length"); + return -EINVAL; + } + sess->cipher_iv_offset = cipher_xform->cipher.iv.offset; + + /* Copy the key */ + memcpy(sess->pKey_cipher, cipher_xform->cipher.key.data, + ZUC_IV_KEY_LENGTH); + } + + if (auth_xform) { + /* Only ZUC EIA3 supported */ + if (auth_xform->auth.algo != RTE_CRYPTO_AUTH_ZUC_EIA3) + return -ENOTSUP; + + if (auth_xform->auth.digest_length != ZUC_DIGEST_LENGTH) { + IPSEC_MB_LOG(ERR, "Wrong digest length"); + return -EINVAL; + } + + sess->auth_op = auth_xform->auth.op; + + if (auth_xform->auth.iv.length != ZUC_IV_KEY_LENGTH) { + IPSEC_MB_LOG(ERR, "Wrong IV length"); + return -EINVAL; + } + sess->auth_iv_offset = auth_xform->auth.iv.offset; + + /* Copy the key */ + memcpy(sess->pKey_hash, auth_xform->auth.key.data, + ZUC_IV_KEY_LENGTH); + } + + sess->op = mode; + return 0; +} + +/** Encrypt/decrypt mbufs. */ +static uint8_t +process_zuc_cipher_op(struct ipsec_mb_qp *qp, struct rte_crypto_op **ops, + struct zuc_session **sessions, + uint8_t num_ops) +{ + unsigned int i; + uint8_t processed_ops = 0; + const void *src[ZUC_MAX_BURST]; + void *dst[ZUC_MAX_BURST]; + const void *iv[ZUC_MAX_BURST]; + uint32_t num_bytes[ZUC_MAX_BURST]; + const void *cipher_keys[ZUC_MAX_BURST]; + struct zuc_session *sess; + + for (i = 0; i < num_ops; i++) { + if (((ops[i]->sym->cipher.data.length % BYTE_LEN) != 0) + || ((ops[i]->sym->cipher.data.offset + % BYTE_LEN) != 0)) { + ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + IPSEC_MB_LOG(ERR, "Data Length or offset"); + break; + } + + sess = sessions[i]; + +#ifdef RTE_LIBRTE_PMD_ZUC_DEBUG + if (!rte_pktmbuf_is_contiguous(ops[i]->sym->m_src) || + (ops[i]->sym->m_dst != NULL && + !rte_pktmbuf_is_contiguous( + ops[i]->sym->m_dst))) { + IPSEC_MB_LOG(ERR, "PMD supports only " + " contiguous mbufs, op (%p) " + "provides noncontiguous mbuf " + "as source/destination buffer.\n", + "PMD supports only contiguous mbufs, " + "op (%p) provides noncontiguous mbuf " + "as source/destination buffer.\n", + ops[i]); + ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + break; + } +#endif + + src[i] = rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + + (ops[i]->sym->cipher.data.offset >> 3); + dst[i] = ops[i]->sym->m_dst ? + rte_pktmbuf_mtod(ops[i]->sym->m_dst, uint8_t *) + + (ops[i]->sym->cipher.data.offset >> 3) : + rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + + (ops[i]->sym->cipher.data.offset >> 3); + iv[i] = rte_crypto_op_ctod_offset(ops[i], uint8_t *, + sess->cipher_iv_offset); + num_bytes[i] = ops[i]->sym->cipher.data.length >> 3; + + cipher_keys[i] = sess->pKey_cipher; + + processed_ops++; + } + + IMB_ZUC_EEA3_N_BUFFER(qp->mb_mgr, (const void **)cipher_keys, + (const void **)iv, (const void **)src, (void **)dst, + num_bytes, processed_ops); + + return processed_ops; +} + +/** Generate/verify hash from mbufs. */ +static int +process_zuc_hash_op(struct ipsec_mb_qp *qp, struct rte_crypto_op **ops, + struct zuc_session **sessions, + uint8_t num_ops) +{ + unsigned int i; + uint8_t processed_ops = 0; + uint8_t *src[ZUC_MAX_BURST]; + uint32_t *dst[ZUC_MAX_BURST]; + uint32_t length_in_bits[ZUC_MAX_BURST]; + uint8_t *iv[ZUC_MAX_BURST]; + const void *hash_keys[ZUC_MAX_BURST]; + struct zuc_session *sess; + struct zuc_qp_data *qp_data = ipsec_mb_get_qp_private_data(qp); + + + for (i = 0; i < num_ops; i++) { + /* Data must be byte aligned */ + if ((ops[i]->sym->auth.data.offset % BYTE_LEN) != 0) { + ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + IPSEC_MB_LOG(ERR, "Offset"); + break; + } + + sess = sessions[i]; + + length_in_bits[i] = ops[i]->sym->auth.data.length; + + src[i] = rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + + (ops[i]->sym->auth.data.offset >> 3); + iv[i] = rte_crypto_op_ctod_offset(ops[i], uint8_t *, + sess->auth_iv_offset); + + hash_keys[i] = sess->pKey_hash; + if (sess->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) + dst[i] = (uint32_t *)qp_data->temp_digest; + else + dst[i] = (uint32_t *)ops[i]->sym->auth.digest.data; + + processed_ops++; + } + + IMB_ZUC_EIA3_N_BUFFER(qp->mb_mgr, (const void **)hash_keys, + (const void * const *)iv, (const void * const *)src, + length_in_bits, dst, processed_ops); + + /* + * If tag needs to be verified, compare generated tag + * with attached tag + */ + for (i = 0; i < processed_ops; i++) + if (sessions[i]->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) + if (memcmp(dst[i], ops[i]->sym->auth.digest.data, + ZUC_DIGEST_LENGTH) != 0) + ops[i]->status = + RTE_CRYPTO_OP_STATUS_AUTH_FAILED; + + return processed_ops; +} + +/** Process a batch of crypto ops which shares the same operation type. */ +static int +process_ops(struct rte_crypto_op **ops, enum ipsec_mb_operation op_type, + struct zuc_session **sessions, + struct ipsec_mb_qp *qp, uint8_t num_ops) +{ + unsigned int i; + unsigned int processed_ops; + + switch (op_type) { + case IPSEC_MB_OP_ENCRYPT_ONLY: + case IPSEC_MB_OP_DECRYPT_ONLY: + processed_ops = process_zuc_cipher_op(qp, ops, + sessions, num_ops); + break; + case IPSEC_MB_OP_HASH_GEN_ONLY: + case IPSEC_MB_OP_HASH_VERIFY_ONLY: + processed_ops = process_zuc_hash_op(qp, ops, sessions, + num_ops); + break; + case IPSEC_MB_OP_ENCRYPT_THEN_HASH_GEN: + processed_ops = process_zuc_cipher_op(qp, ops, sessions, + num_ops); + process_zuc_hash_op(qp, ops, sessions, processed_ops); + break; + case IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT: + processed_ops = process_zuc_hash_op(qp, ops, sessions, + num_ops); + process_zuc_cipher_op(qp, ops, sessions, processed_ops); + break; + default: + /* Operation not supported. */ + processed_ops = 0; + } + + for (i = 0; i < num_ops; i++) { + /* + * If there was no error/authentication failure, + * change status to successful. + */ + if (ops[i]->status == RTE_CRYPTO_OP_STATUS_NOT_PROCESSED) + ops[i]->status = RTE_CRYPTO_OP_STATUS_SUCCESS; + /* Free session if a session-less crypto op. */ + if (ops[i]->sess_type == RTE_CRYPTO_OP_SESSIONLESS) { + memset(sessions[i], 0, sizeof(struct zuc_session)); + memset(ops[i]->sym->session, 0, + rte_cryptodev_sym_get_existing_header_session_size( + ops[i]->sym->session)); + rte_mempool_put(qp->sess_mp_priv, sessions[i]); + rte_mempool_put(qp->sess_mp, ops[i]->sym->session); + ops[i]->sym->session = NULL; + } + } + return processed_ops; +} + +static uint16_t +zuc_pmd_dequeue_burst(void *queue_pair, + struct rte_crypto_op **c_ops, uint16_t nb_ops) +{ + + struct rte_crypto_op *curr_c_op; + + struct zuc_session *curr_sess; + struct zuc_session *sessions[ZUC_MAX_BURST]; + enum ipsec_mb_operation prev_zuc_op = IPSEC_MB_OP_NOT_SUPPORTED; + enum ipsec_mb_operation curr_zuc_op; + struct ipsec_mb_qp *qp = queue_pair; + unsigned int nb_dequeued; + unsigned int i; + uint8_t burst_size = 0; + uint8_t processed_ops; + + nb_dequeued = rte_ring_dequeue_burst(qp->ingress_queue, + (void **)c_ops, nb_ops, NULL); + + + for (i = 0; i < nb_dequeued; i++) { + curr_c_op = c_ops[i]; + + curr_sess = (struct zuc_session *) + ipsec_mb_get_session_private(qp, curr_c_op); + if (unlikely(curr_sess == NULL)) { + curr_c_op->status = + RTE_CRYPTO_OP_STATUS_INVALID_SESSION; + break; + } + + curr_zuc_op = curr_sess->op; + + /* + * Batch ops that share the same operation type + * (cipher only, auth only...). + */ + if (burst_size == 0) { + prev_zuc_op = curr_zuc_op; + c_ops[0] = curr_c_op; + sessions[0] = curr_sess; + burst_size++; + } else if (curr_zuc_op == prev_zuc_op) { + c_ops[burst_size] = curr_c_op; + sessions[burst_size] = curr_sess; + burst_size++; + /* + * When there are enough ops to process in a batch, + * process them, and start a new batch. + */ + if (burst_size == ZUC_MAX_BURST) { + processed_ops = process_ops(c_ops, curr_zuc_op, + sessions, qp, burst_size); + if (processed_ops < burst_size) { + burst_size = 0; + break; + } + + burst_size = 0; + } + } else { + /* + * Different operation type, process the ops + * of the previous type. + */ + processed_ops = process_ops(c_ops, prev_zuc_op, + sessions, qp, burst_size); + if (processed_ops < burst_size) { + burst_size = 0; + break; + } + + burst_size = 0; + prev_zuc_op = curr_zuc_op; + + c_ops[0] = curr_c_op; + sessions[0] = curr_sess; + burst_size++; + } + } + + if (burst_size != 0) { + /* Process the crypto ops of the last operation type. */ + processed_ops = process_ops(c_ops, prev_zuc_op, + sessions, qp, burst_size); + } + + qp->stats.dequeued_count += i; + return i; +} + +struct rte_cryptodev_ops zuc_pmd_ops = { + .dev_configure = ipsec_mb_config, + .dev_start = ipsec_mb_start, + .dev_stop = ipsec_mb_stop, + .dev_close = ipsec_mb_close, + + .stats_get = ipsec_mb_stats_get, + .stats_reset = ipsec_mb_stats_reset, + + .dev_infos_get = ipsec_mb_info_get, + + .queue_pair_setup = ipsec_mb_qp_setup, + .queue_pair_release = ipsec_mb_qp_release, + + .sym_session_get_size = ipsec_mb_sym_session_get_size, + .sym_session_configure = ipsec_mb_sym_session_configure, + .sym_session_clear = ipsec_mb_sym_session_clear +}; + +struct rte_cryptodev_ops *rte_zuc_pmd_ops = &zuc_pmd_ops; + +static int +zuc_probe(struct rte_vdev_device *vdev) +{ + return ipsec_mb_create(vdev, IPSEC_MB_PMD_TYPE_ZUC); +} + +static struct rte_vdev_driver cryptodev_zuc_pmd_drv = { + .probe = zuc_probe, + .remove = ipsec_mb_remove + +}; + +static struct cryptodev_driver zuc_crypto_drv; + +RTE_PMD_REGISTER_VDEV(CRYPTODEV_NAME_ZUC_PMD, cryptodev_zuc_pmd_drv); +RTE_PMD_REGISTER_ALIAS(CRYPTODEV_NAME_ZUC_PMD, cryptodev_zuc_pmd); +RTE_PMD_REGISTER_PARAM_STRING(CRYPTODEV_NAME_ZUC_PMD, + "max_nb_queue_pairs= socket_id="); +RTE_PMD_REGISTER_CRYPTO_DRIVER(zuc_crypto_drv, cryptodev_zuc_pmd_drv.driver, + pmd_driver_id_zuc); + +/* Constructor function to register zuc PMD */ +RTE_INIT(ipsec_mb_register_zuc) +{ + struct ipsec_mb_internals *zuc_data + = &ipsec_mb_pmds[IPSEC_MB_PMD_TYPE_ZUC]; + + zuc_data->caps = zuc_capabilities; + zuc_data->dequeue_burst = zuc_pmd_dequeue_burst; + zuc_data->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO + | RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING + | RTE_CRYPTODEV_FF_NON_BYTE_ALIGNED_DATA + | RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT + | RTE_CRYPTODEV_FF_SYM_SESSIONLESS + | RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT; + zuc_data->internals_priv_size = 0; + zuc_data->ops = &zuc_pmd_ops; + zuc_data->qp_priv_size = sizeof(struct zuc_qp_data); + zuc_data->session_configure = zuc_session_configure; + zuc_data->session_priv_size = sizeof(struct zuc_session); +} diff --git a/drivers/crypto/ipsec_mb/pmd_zuc_priv.h b/drivers/crypto/ipsec_mb/pmd_zuc_priv.h new file mode 100644 index 0000000000..46d5bfae37 --- /dev/null +++ b/drivers/crypto/ipsec_mb/pmd_zuc_priv.h @@ -0,0 +1,85 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2016-2021 Intel Corporation + */ + +#ifndef _PMD_ZUC_PRIV_H_ +#define _PMD_ZUC_PRIV_H_ + +#include "ipsec_mb_private.h" + +#define ZUC_IV_KEY_LENGTH 16 +#define ZUC_DIGEST_LENGTH 4 +#define ZUC_MAX_BURST 16 +#define BYTE_LEN 8 + +uint8_t pmd_driver_id_zuc; + +static const struct rte_cryptodev_capabilities zuc_capabilities[] = { + { /* ZUC (EIA3) */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, + {.auth = { + .algo = RTE_CRYPTO_AUTH_ZUC_EIA3, + .block_size = 16, + .key_size = { + .min = 16, + .max = 16, + .increment = 0 + }, + .digest_size = { + .min = ZUC_DIGEST_LENGTH, + .max = ZUC_DIGEST_LENGTH, + .increment = 0 + }, + .iv_size = { + .min = ZUC_IV_KEY_LENGTH, + .max = ZUC_IV_KEY_LENGTH, + .increment = 0 + } + }, } + }, } + }, + { /* ZUC (EEA3) */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, + {.cipher = { + .algo = RTE_CRYPTO_CIPHER_ZUC_EEA3, + .block_size = 16, + .key_size = { + .min = 16, + .max = 16, + .increment = 0 + }, + .iv_size = { + .min = ZUC_IV_KEY_LENGTH, + .max = ZUC_IV_KEY_LENGTH, + .increment = 0 + }, + }, } + }, } + }, + RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() +}; + +/** ZUC private session structure */ +struct zuc_session { + enum ipsec_mb_operation op; + enum rte_crypto_auth_operation auth_op; + uint8_t pKey_cipher[ZUC_IV_KEY_LENGTH]; + uint8_t pKey_hash[ZUC_IV_KEY_LENGTH]; + uint16_t cipher_iv_offset; + uint16_t auth_iv_offset; +} __rte_cache_aligned; + +struct zuc_qp_data { + + uint8_t temp_digest[ZUC_DIGEST_LENGTH]; + /* *< Buffers used to store the digest generated + * by the driver when verifying a digest provided + * by the user (using authentication verify operation) + */ +}; + +#endif /* _PMD_ZUC_PRIV_H_ */ diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build index 5ee3f68c89..2585471e93 100644 --- a/drivers/crypto/meson.build +++ b/drivers/crypto/meson.build @@ -23,7 +23,6 @@ drivers = [ 'openssl', 'scheduler', 'virtio', - 'zuc', ] std_deps = ['cryptodev'] # cryptodev pulls in all other needed deps diff --git a/drivers/crypto/zuc/meson.build b/drivers/crypto/zuc/meson.build deleted file mode 100644 index a5f77a22d8..0000000000 --- a/drivers/crypto/zuc/meson.build +++ /dev/null @@ -1,24 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2018-2020 Intel Corporation - -IMB_required_ver = '0.53.0' -lib = cc.find_library('IPSec_MB', required: false) -if not lib.found() - build = false - reason = 'missing dependency, "libIPSec_MB"' -else - # version comes with quotes, so we split based on " and take the middle - imb_ver = cc.get_define('IMB_VERSION_STR', - prefix : '#include').split('"')[1] - - if (imb_ver == '') or (imb_ver.version_compare('<' + IMB_required_ver)) - reason = 'IPSec_MB version >= @0@ is required, found version @1@'.format( - IMB_required_ver, imb_ver) - build = false - endif - -endif - -ext_deps += lib -sources = files('rte_zuc_pmd.c', 'rte_zuc_pmd_ops.c') -deps += ['bus_vdev'] diff --git a/drivers/crypto/zuc/rte_zuc_pmd.c b/drivers/crypto/zuc/rte_zuc_pmd.c deleted file mode 100644 index d4b343a7af..0000000000 --- a/drivers/crypto/zuc/rte_zuc_pmd.c +++ /dev/null @@ -1,583 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2018 Intel Corporation - */ - -#include -#include -#include -#include -#include -#include -#include - -#include "zuc_pmd_private.h" -#define ZUC_MAX_BURST 16 -#define BYTE_LEN 8 - -static uint8_t cryptodev_driver_id; - -/** Get xform chain order. */ -static enum zuc_operation -zuc_get_mode(const struct rte_crypto_sym_xform *xform) -{ - if (xform == NULL) - return ZUC_OP_NOT_SUPPORTED; - - if (xform->next) - if (xform->next->next != NULL) - return ZUC_OP_NOT_SUPPORTED; - - if (xform->type == RTE_CRYPTO_SYM_XFORM_AUTH) { - if (xform->next == NULL) - return ZUC_OP_ONLY_AUTH; - else if (xform->next->type == RTE_CRYPTO_SYM_XFORM_CIPHER) - return ZUC_OP_AUTH_CIPHER; - else - return ZUC_OP_NOT_SUPPORTED; - } - - if (xform->type == RTE_CRYPTO_SYM_XFORM_CIPHER) { - if (xform->next == NULL) - return ZUC_OP_ONLY_CIPHER; - else if (xform->next->type == RTE_CRYPTO_SYM_XFORM_AUTH) - return ZUC_OP_CIPHER_AUTH; - else - return ZUC_OP_NOT_SUPPORTED; - } - - return ZUC_OP_NOT_SUPPORTED; -} - - -/** Parse crypto xform chain and set private session parameters. */ -int -zuc_set_session_parameters(struct zuc_session *sess, - const struct rte_crypto_sym_xform *xform) -{ - const struct rte_crypto_sym_xform *auth_xform = NULL; - const struct rte_crypto_sym_xform *cipher_xform = NULL; - enum zuc_operation mode; - - /* Select Crypto operation - hash then cipher / cipher then hash */ - mode = zuc_get_mode(xform); - - switch (mode) { - case ZUC_OP_CIPHER_AUTH: - auth_xform = xform->next; - - /* Fall-through */ - case ZUC_OP_ONLY_CIPHER: - cipher_xform = xform; - break; - case ZUC_OP_AUTH_CIPHER: - cipher_xform = xform->next; - /* Fall-through */ - case ZUC_OP_ONLY_AUTH: - auth_xform = xform; - break; - case ZUC_OP_NOT_SUPPORTED: - default: - ZUC_LOG(ERR, "Unsupported operation chain order parameter"); - return -ENOTSUP; - } - - if (cipher_xform) { - /* Only ZUC EEA3 supported */ - if (cipher_xform->cipher.algo != RTE_CRYPTO_CIPHER_ZUC_EEA3) - return -ENOTSUP; - - if (cipher_xform->cipher.iv.length != ZUC_IV_KEY_LENGTH) { - ZUC_LOG(ERR, "Wrong IV length"); - return -EINVAL; - } - sess->cipher_iv_offset = cipher_xform->cipher.iv.offset; - - /* Copy the key */ - memcpy(sess->pKey_cipher, cipher_xform->cipher.key.data, - ZUC_IV_KEY_LENGTH); - } - - if (auth_xform) { - /* Only ZUC EIA3 supported */ - if (auth_xform->auth.algo != RTE_CRYPTO_AUTH_ZUC_EIA3) - return -ENOTSUP; - - if (auth_xform->auth.digest_length != ZUC_DIGEST_LENGTH) { - ZUC_LOG(ERR, "Wrong digest length"); - return -EINVAL; - } - - sess->auth_op = auth_xform->auth.op; - - if (auth_xform->auth.iv.length != ZUC_IV_KEY_LENGTH) { - ZUC_LOG(ERR, "Wrong IV length"); - return -EINVAL; - } - sess->auth_iv_offset = auth_xform->auth.iv.offset; - - /* Copy the key */ - memcpy(sess->pKey_hash, auth_xform->auth.key.data, - ZUC_IV_KEY_LENGTH); - } - - - sess->op = mode; - - return 0; -} - -/** Get ZUC session. */ -static struct zuc_session * -zuc_get_session(struct zuc_qp *qp, struct rte_crypto_op *op) -{ - struct zuc_session *sess = NULL; - - if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) { - if (likely(op->sym->session != NULL)) - sess = (struct zuc_session *)get_sym_session_private_data( - op->sym->session, - cryptodev_driver_id); - } else { - void *_sess = NULL; - void *_sess_private_data = NULL; - - if (rte_mempool_get(qp->sess_mp, (void **)&_sess)) - return NULL; - - if (rte_mempool_get(qp->sess_mp_priv, - (void **)&_sess_private_data)) - return NULL; - - sess = (struct zuc_session *)_sess_private_data; - - if (unlikely(zuc_set_session_parameters(sess, - op->sym->xform) != 0)) { - rte_mempool_put(qp->sess_mp, _sess); - rte_mempool_put(qp->sess_mp_priv, _sess_private_data); - sess = NULL; - } - op->sym->session = (struct rte_cryptodev_sym_session *)_sess; - set_sym_session_private_data(op->sym->session, - cryptodev_driver_id, _sess_private_data); - } - - if (unlikely(sess == NULL)) - op->status = RTE_CRYPTO_OP_STATUS_INVALID_SESSION; - - - return sess; -} - -/** Encrypt/decrypt mbufs. */ -static uint8_t -process_zuc_cipher_op(struct zuc_qp *qp, struct rte_crypto_op **ops, - struct zuc_session **sessions, - uint8_t num_ops) -{ - unsigned i; - uint8_t processed_ops = 0; - const void *src[ZUC_MAX_BURST]; - void *dst[ZUC_MAX_BURST]; - const void *iv[ZUC_MAX_BURST]; - uint32_t num_bytes[ZUC_MAX_BURST]; - const void *cipher_keys[ZUC_MAX_BURST]; - struct zuc_session *sess; - - for (i = 0; i < num_ops; i++) { - if (((ops[i]->sym->cipher.data.length % BYTE_LEN) != 0) - || ((ops[i]->sym->cipher.data.offset - % BYTE_LEN) != 0)) { - ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; - ZUC_LOG(ERR, "Data Length or offset"); - break; - } - - sess = sessions[i]; - -#ifdef RTE_LIBRTE_PMD_ZUC_DEBUG - if (!rte_pktmbuf_is_contiguous(ops[i]->sym->m_src) || - (ops[i]->sym->m_dst != NULL && - !rte_pktmbuf_is_contiguous( - ops[i]->sym->m_dst))) { - ZUC_LOG(ERR, "PMD supports only contiguous mbufs, " - "op (%p) provides noncontiguous mbuf as " - "source/destination buffer.\n", ops[i]); - ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; - break; - } -#endif - - src[i] = rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + - (ops[i]->sym->cipher.data.offset >> 3); - dst[i] = ops[i]->sym->m_dst ? - rte_pktmbuf_mtod(ops[i]->sym->m_dst, uint8_t *) + - (ops[i]->sym->cipher.data.offset >> 3) : - rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + - (ops[i]->sym->cipher.data.offset >> 3); - iv[i] = rte_crypto_op_ctod_offset(ops[i], uint8_t *, - sess->cipher_iv_offset); - num_bytes[i] = ops[i]->sym->cipher.data.length >> 3; - - cipher_keys[i] = sess->pKey_cipher; - - processed_ops++; - } - - IMB_ZUC_EEA3_N_BUFFER(qp->mb_mgr, (const void **)cipher_keys, - (const void **)iv, (const void **)src, (void **)dst, - num_bytes, processed_ops); - - return processed_ops; -} - -/** Generate/verify hash from mbufs. */ -static int -process_zuc_hash_op(struct zuc_qp *qp, struct rte_crypto_op **ops, - struct zuc_session **sessions, - uint8_t num_ops) -{ - unsigned int i; - uint8_t processed_ops = 0; - uint8_t *src[ZUC_MAX_BURST] = { 0 }; - uint32_t *dst[ZUC_MAX_BURST]; - uint32_t length_in_bits[ZUC_MAX_BURST] = { 0 }; - uint8_t *iv[ZUC_MAX_BURST] = { 0 }; - const void *hash_keys[ZUC_MAX_BURST] = { 0 }; - struct zuc_session *sess; - - for (i = 0; i < num_ops; i++) { - /* Data must be byte aligned */ - if ((ops[i]->sym->auth.data.offset % BYTE_LEN) != 0) { - ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; - ZUC_LOG(ERR, "Offset"); - break; - } - - sess = sessions[i]; - - length_in_bits[i] = ops[i]->sym->auth.data.length; - - src[i] = rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + - (ops[i]->sym->auth.data.offset >> 3); - iv[i] = rte_crypto_op_ctod_offset(ops[i], uint8_t *, - sess->auth_iv_offset); - - hash_keys[i] = sess->pKey_hash; - if (sess->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) - dst[i] = (uint32_t *)qp->temp_digest; - else - dst[i] = (uint32_t *)ops[i]->sym->auth.digest.data; - -#if IMB_VERSION_NUM < IMB_VERSION(0, 53, 3) - IMB_ZUC_EIA3_1_BUFFER(qp->mb_mgr, hash_keys[i], - iv[i], src[i], length_in_bits[i], dst[i]); -#endif - processed_ops++; - } - -#if IMB_VERSION_NUM >= IMB_VERSION(0, 53, 3) - IMB_ZUC_EIA3_N_BUFFER(qp->mb_mgr, (const void **)hash_keys, - (const void * const *)iv, (const void * const *)src, - length_in_bits, dst, processed_ops); -#endif - - /* - * If tag needs to be verified, compare generated tag - * with attached tag - */ - for (i = 0; i < processed_ops; i++) - if (sessions[i]->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) - if (memcmp(dst[i], ops[i]->sym->auth.digest.data, - ZUC_DIGEST_LENGTH) != 0) - ops[i]->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; - - return processed_ops; -} - -/** Process a batch of crypto ops which shares the same operation type. */ -static int -process_ops(struct rte_crypto_op **ops, enum zuc_operation op_type, - struct zuc_session **sessions, - struct zuc_qp *qp, uint8_t num_ops, - uint16_t *accumulated_enqueued_ops) -{ - unsigned i; - unsigned enqueued_ops, processed_ops; - - switch (op_type) { - case ZUC_OP_ONLY_CIPHER: - processed_ops = process_zuc_cipher_op(qp, ops, - sessions, num_ops); - break; - case ZUC_OP_ONLY_AUTH: - processed_ops = process_zuc_hash_op(qp, ops, sessions, - num_ops); - break; - case ZUC_OP_CIPHER_AUTH: - processed_ops = process_zuc_cipher_op(qp, ops, sessions, - num_ops); - process_zuc_hash_op(qp, ops, sessions, processed_ops); - break; - case ZUC_OP_AUTH_CIPHER: - processed_ops = process_zuc_hash_op(qp, ops, sessions, - num_ops); - process_zuc_cipher_op(qp, ops, sessions, processed_ops); - break; - default: - /* Operation not supported. */ - processed_ops = 0; - } - - for (i = 0; i < num_ops; i++) { - /* - * If there was no error/authentication failure, - * change status to successful. - */ - if (ops[i]->status == RTE_CRYPTO_OP_STATUS_NOT_PROCESSED) - ops[i]->status = RTE_CRYPTO_OP_STATUS_SUCCESS; - /* Free session if a session-less crypto op. */ - if (ops[i]->sess_type == RTE_CRYPTO_OP_SESSIONLESS) { - memset(sessions[i], 0, sizeof(struct zuc_session)); - memset(ops[i]->sym->session, 0, - rte_cryptodev_sym_get_existing_header_session_size( - ops[i]->sym->session)); - rte_mempool_put(qp->sess_mp_priv, sessions[i]); - rte_mempool_put(qp->sess_mp, ops[i]->sym->session); - ops[i]->sym->session = NULL; - } - } - - enqueued_ops = rte_ring_enqueue_burst(qp->processed_ops, - (void **)ops, processed_ops, NULL); - qp->qp_stats.enqueued_count += enqueued_ops; - *accumulated_enqueued_ops += enqueued_ops; - - return enqueued_ops; -} - -static uint16_t -zuc_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops, - uint16_t nb_ops) -{ - struct rte_crypto_op *c_ops[ZUC_MAX_BURST]; - struct rte_crypto_op *curr_c_op; - - struct zuc_session *curr_sess; - struct zuc_session *sessions[ZUC_MAX_BURST]; - enum zuc_operation prev_zuc_op = ZUC_OP_NOT_SUPPORTED; - enum zuc_operation curr_zuc_op; - struct zuc_qp *qp = queue_pair; - unsigned i; - uint8_t burst_size = 0; - uint16_t enqueued_ops = 0; - uint8_t processed_ops; - - for (i = 0; i < nb_ops; i++) { - curr_c_op = ops[i]; - - curr_sess = zuc_get_session(qp, curr_c_op); - if (unlikely(curr_sess == NULL)) { - curr_c_op->status = - RTE_CRYPTO_OP_STATUS_INVALID_SESSION; - break; - } - - curr_zuc_op = curr_sess->op; - - /* - * Batch ops that share the same operation type - * (cipher only, auth only...). - */ - if (burst_size == 0) { - prev_zuc_op = curr_zuc_op; - c_ops[0] = curr_c_op; - sessions[0] = curr_sess; - burst_size++; - } else if (curr_zuc_op == prev_zuc_op) { - c_ops[burst_size] = curr_c_op; - sessions[burst_size] = curr_sess; - burst_size++; - /* - * When there are enough ops to process in a batch, - * process them, and start a new batch. - */ - if (burst_size == ZUC_MAX_BURST) { - processed_ops = process_ops(c_ops, curr_zuc_op, - sessions, qp, burst_size, - &enqueued_ops); - if (processed_ops < burst_size) { - burst_size = 0; - break; - } - - burst_size = 0; - } - } else { - /* - * Different operation type, process the ops - * of the previous type. - */ - processed_ops = process_ops(c_ops, prev_zuc_op, - sessions, qp, burst_size, - &enqueued_ops); - if (processed_ops < burst_size) { - burst_size = 0; - break; - } - - burst_size = 0; - prev_zuc_op = curr_zuc_op; - - c_ops[0] = curr_c_op; - sessions[0] = curr_sess; - burst_size++; - } - } - - if (burst_size != 0) { - /* Process the crypto ops of the last operation type. */ - processed_ops = process_ops(c_ops, prev_zuc_op, - sessions, qp, burst_size, - &enqueued_ops); - } - - qp->qp_stats.enqueue_err_count += nb_ops - enqueued_ops; - return enqueued_ops; -} - -static uint16_t -zuc_pmd_dequeue_burst(void *queue_pair, - struct rte_crypto_op **c_ops, uint16_t nb_ops) -{ - struct zuc_qp *qp = queue_pair; - - unsigned nb_dequeued; - - nb_dequeued = rte_ring_dequeue_burst(qp->processed_ops, - (void **)c_ops, nb_ops, NULL); - qp->qp_stats.dequeued_count += nb_dequeued; - - return nb_dequeued; -} - -static int cryptodev_zuc_remove(struct rte_vdev_device *vdev); - -static int -cryptodev_zuc_create(const char *name, - struct rte_vdev_device *vdev, - struct rte_cryptodev_pmd_init_params *init_params) -{ - struct rte_cryptodev *dev; - struct zuc_private *internals; - MB_MGR *mb_mgr; - - dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params); - if (dev == NULL) { - ZUC_LOG(ERR, "failed to create cryptodev vdev"); - goto init_error; - } - - dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | - RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | - RTE_CRYPTODEV_FF_NON_BYTE_ALIGNED_DATA | - RTE_CRYPTODEV_FF_SYM_SESSIONLESS | - RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT; - - mb_mgr = alloc_mb_mgr(0); - if (mb_mgr == NULL) - return -ENOMEM; - - if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F)) { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX512; - init_mb_mgr_avx512(mb_mgr); - } else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2; - init_mb_mgr_avx2(mb_mgr); - } else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX; - init_mb_mgr_avx(mb_mgr); - } else { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE; - init_mb_mgr_sse(mb_mgr); - } - - dev->driver_id = cryptodev_driver_id; - dev->dev_ops = rte_zuc_pmd_ops; - - /* Register RX/TX burst functions for data path. */ - dev->dequeue_burst = zuc_pmd_dequeue_burst; - dev->enqueue_burst = zuc_pmd_enqueue_burst; - - internals = dev->data->dev_private; - internals->mb_mgr = mb_mgr; - - internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs; - - return 0; -init_error: - ZUC_LOG(ERR, "driver %s: failed", - init_params->name); - - cryptodev_zuc_remove(vdev); - return -EFAULT; -} - -static int -cryptodev_zuc_probe(struct rte_vdev_device *vdev) -{ - struct rte_cryptodev_pmd_init_params init_params = { - "", - sizeof(struct zuc_private), - rte_socket_id(), - RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS - }; - const char *name; - const char *input_args; - - name = rte_vdev_device_name(vdev); - if (name == NULL) - return -EINVAL; - input_args = rte_vdev_device_args(vdev); - - rte_cryptodev_pmd_parse_input_args(&init_params, input_args); - - return cryptodev_zuc_create(name, vdev, &init_params); -} - -static int -cryptodev_zuc_remove(struct rte_vdev_device *vdev) -{ - - struct rte_cryptodev *cryptodev; - const char *name; - struct zuc_private *internals; - - name = rte_vdev_device_name(vdev); - if (name == NULL) - return -EINVAL; - - cryptodev = rte_cryptodev_pmd_get_named_dev(name); - if (cryptodev == NULL) - return -ENODEV; - - internals = cryptodev->data->dev_private; - - free_mb_mgr(internals->mb_mgr); - - return rte_cryptodev_pmd_destroy(cryptodev); -} - -static struct rte_vdev_driver cryptodev_zuc_pmd_drv = { - .probe = cryptodev_zuc_probe, - .remove = cryptodev_zuc_remove -}; - -static struct cryptodev_driver zuc_crypto_drv; - -RTE_PMD_REGISTER_VDEV(CRYPTODEV_NAME_ZUC_PMD, cryptodev_zuc_pmd_drv); -RTE_PMD_REGISTER_PARAM_STRING(CRYPTODEV_NAME_ZUC_PMD, - "max_nb_queue_pairs= " - "socket_id="); -RTE_PMD_REGISTER_CRYPTO_DRIVER(zuc_crypto_drv, cryptodev_zuc_pmd_drv.driver, - cryptodev_driver_id); -RTE_LOG_REGISTER_DEFAULT(zuc_logtype_driver, INFO); diff --git a/drivers/crypto/zuc/rte_zuc_pmd_ops.c b/drivers/crypto/zuc/rte_zuc_pmd_ops.c deleted file mode 100644 index 38642d45ab..0000000000 --- a/drivers/crypto/zuc/rte_zuc_pmd_ops.c +++ /dev/null @@ -1,322 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2018 Intel Corporation - */ - -#include - -#include -#include -#include - -#include "zuc_pmd_private.h" - -static const struct rte_cryptodev_capabilities zuc_pmd_capabilities[] = { - { /* ZUC (EIA3) */ - .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, - {.sym = { - .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, - {.auth = { - .algo = RTE_CRYPTO_AUTH_ZUC_EIA3, - .block_size = 16, - .key_size = { - .min = 16, - .max = 16, - .increment = 0 - }, - .digest_size = { - .min = 4, - .max = 4, - .increment = 0 - }, - .iv_size = { - .min = 16, - .max = 16, - .increment = 0 - } - }, } - }, } - }, - { /* ZUC (EEA3) */ - .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, - {.sym = { - .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, - {.cipher = { - .algo = RTE_CRYPTO_CIPHER_ZUC_EEA3, - .block_size = 16, - .key_size = { - .min = 16, - .max = 16, - .increment = 0 - }, - .iv_size = { - .min = 16, - .max = 16, - .increment = 0 - }, - }, } - }, } - }, - RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() -}; - -/** Configure device */ -static int -zuc_pmd_config(__rte_unused struct rte_cryptodev *dev, - __rte_unused struct rte_cryptodev_config *config) -{ - return 0; -} - -/** Start device */ -static int -zuc_pmd_start(__rte_unused struct rte_cryptodev *dev) -{ - return 0; -} - -/** Stop device */ -static void -zuc_pmd_stop(__rte_unused struct rte_cryptodev *dev) -{ -} - -/** Close device */ -static int -zuc_pmd_close(__rte_unused struct rte_cryptodev *dev) -{ - return 0; -} - - -/** Get device statistics */ -static void -zuc_pmd_stats_get(struct rte_cryptodev *dev, - struct rte_cryptodev_stats *stats) -{ - int qp_id; - - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - struct zuc_qp *qp = dev->data->queue_pairs[qp_id]; - - stats->enqueued_count += qp->qp_stats.enqueued_count; - stats->dequeued_count += qp->qp_stats.dequeued_count; - - stats->enqueue_err_count += qp->qp_stats.enqueue_err_count; - stats->dequeue_err_count += qp->qp_stats.dequeue_err_count; - } -} - -/** Reset device statistics */ -static void -zuc_pmd_stats_reset(struct rte_cryptodev *dev) -{ - int qp_id; - - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - struct zuc_qp *qp = dev->data->queue_pairs[qp_id]; - - memset(&qp->qp_stats, 0, sizeof(qp->qp_stats)); - } -} - - -/** Get device info */ -static void -zuc_pmd_info_get(struct rte_cryptodev *dev, - struct rte_cryptodev_info *dev_info) -{ - struct zuc_private *internals = dev->data->dev_private; - - if (dev_info != NULL) { - dev_info->driver_id = dev->driver_id; - dev_info->max_nb_queue_pairs = internals->max_nb_queue_pairs; - /* No limit of number of sessions */ - dev_info->sym.max_nb_sessions = 0; - dev_info->feature_flags = dev->feature_flags; - dev_info->capabilities = zuc_pmd_capabilities; - } -} - -/** Release queue pair */ -static int -zuc_pmd_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) -{ - if (dev->data->queue_pairs[qp_id] != NULL) { - struct zuc_qp *qp = dev->data->queue_pairs[qp_id]; - - if (qp->processed_ops) - rte_ring_free(qp->processed_ops); - - rte_free(dev->data->queue_pairs[qp_id]); - dev->data->queue_pairs[qp_id] = NULL; - } - return 0; -} - -/** set a unique name for the queue pair based on its name, dev_id and qp_id */ -static int -zuc_pmd_qp_set_unique_name(struct rte_cryptodev *dev, - struct zuc_qp *qp) -{ - unsigned n = snprintf(qp->name, sizeof(qp->name), - "zuc_pmd_%u_qp_%u", - dev->data->dev_id, qp->id); - - if (n >= sizeof(qp->name)) - return -1; - - return 0; -} - -/** Create a ring to place processed ops on */ -static struct rte_ring * -zuc_pmd_qp_create_processed_ops_ring(struct zuc_qp *qp, - unsigned ring_size, int socket_id) -{ - struct rte_ring *r; - - r = rte_ring_lookup(qp->name); - if (r) { - if (rte_ring_get_size(r) >= ring_size) { - ZUC_LOG(INFO, "Reusing existing ring %s" - " for processed packets", - qp->name); - return r; - } - - ZUC_LOG(ERR, "Unable to reuse existing ring %s" - " for processed packets", - qp->name); - return NULL; - } - - return rte_ring_create(qp->name, ring_size, socket_id, - RING_F_SP_ENQ | RING_F_SC_DEQ); -} - -/** Setup a queue pair */ -static int -zuc_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, - const struct rte_cryptodev_qp_conf *qp_conf, - int socket_id) -{ - struct zuc_qp *qp = NULL; - struct zuc_private *internals = dev->data->dev_private; - - /* Free memory prior to re-allocation if needed. */ - if (dev->data->queue_pairs[qp_id] != NULL) - zuc_pmd_qp_release(dev, qp_id); - - /* Allocate the queue pair data structure. */ - qp = rte_zmalloc_socket("ZUC PMD Queue Pair", sizeof(*qp), - RTE_CACHE_LINE_SIZE, socket_id); - if (qp == NULL) - return (-ENOMEM); - - qp->id = qp_id; - dev->data->queue_pairs[qp_id] = qp; - - if (zuc_pmd_qp_set_unique_name(dev, qp)) - goto qp_setup_cleanup; - - qp->processed_ops = zuc_pmd_qp_create_processed_ops_ring(qp, - qp_conf->nb_descriptors, socket_id); - if (qp->processed_ops == NULL) - goto qp_setup_cleanup; - - qp->mb_mgr = internals->mb_mgr; - qp->sess_mp = qp_conf->mp_session; - qp->sess_mp_priv = qp_conf->mp_session_private; - - memset(&qp->qp_stats, 0, sizeof(qp->qp_stats)); - - return 0; - -qp_setup_cleanup: - if (qp) - rte_free(qp); - - return -1; -} - -/** Returns the size of the ZUC session structure */ -static unsigned -zuc_pmd_sym_session_get_size(struct rte_cryptodev *dev __rte_unused) -{ - return sizeof(struct zuc_session); -} - -/** Configure a ZUC session from a crypto xform chain */ -static int -zuc_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused, - struct rte_crypto_sym_xform *xform, - struct rte_cryptodev_sym_session *sess, - struct rte_mempool *mempool) -{ - void *sess_private_data; - int ret; - - if (unlikely(sess == NULL)) { - ZUC_LOG(ERR, "invalid session struct"); - return -EINVAL; - } - - if (rte_mempool_get(mempool, &sess_private_data)) { - ZUC_LOG(ERR, - "Couldn't get object from session mempool"); - - return -ENOMEM; - } - - ret = zuc_set_session_parameters(sess_private_data, xform); - if (ret != 0) { - ZUC_LOG(ERR, "failed configure session parameters"); - - /* Return session to mempool */ - rte_mempool_put(mempool, sess_private_data); - return ret; - } - - set_sym_session_private_data(sess, dev->driver_id, - sess_private_data); - - return 0; -} - -/** Clear the memory of session so it doesn't leave key material behind */ -static void -zuc_pmd_sym_session_clear(struct rte_cryptodev *dev, - struct rte_cryptodev_sym_session *sess) -{ - uint8_t index = dev->driver_id; - void *sess_priv = get_sym_session_private_data(sess, index); - - /* Zero out the whole structure */ - if (sess_priv) { - memset(sess_priv, 0, sizeof(struct zuc_session)); - struct rte_mempool *sess_mp = rte_mempool_from_obj(sess_priv); - set_sym_session_private_data(sess, index, NULL); - rte_mempool_put(sess_mp, sess_priv); - } -} - -struct rte_cryptodev_ops zuc_pmd_ops = { - .dev_configure = zuc_pmd_config, - .dev_start = zuc_pmd_start, - .dev_stop = zuc_pmd_stop, - .dev_close = zuc_pmd_close, - - .stats_get = zuc_pmd_stats_get, - .stats_reset = zuc_pmd_stats_reset, - - .dev_infos_get = zuc_pmd_info_get, - - .queue_pair_setup = zuc_pmd_qp_setup, - .queue_pair_release = zuc_pmd_qp_release, - - .sym_session_get_size = zuc_pmd_sym_session_get_size, - .sym_session_configure = zuc_pmd_sym_session_configure, - .sym_session_clear = zuc_pmd_sym_session_clear -}; - -struct rte_cryptodev_ops *rte_zuc_pmd_ops = &zuc_pmd_ops; diff --git a/drivers/crypto/zuc/version.map b/drivers/crypto/zuc/version.map deleted file mode 100644 index c2e0723b4c..0000000000 --- a/drivers/crypto/zuc/version.map +++ /dev/null @@ -1,3 +0,0 @@ -DPDK_22 { - local: *; -}; diff --git a/drivers/crypto/zuc/zuc_pmd_private.h b/drivers/crypto/zuc/zuc_pmd_private.h deleted file mode 100644 index d8684891ee..0000000000 --- a/drivers/crypto/zuc/zuc_pmd_private.h +++ /dev/null @@ -1,83 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2018 Intel Corporation - */ - -#ifndef _ZUC_PMD_PRIVATE_H_ -#define _ZUC_PMD_PRIVATE_H_ - -#include - -#define CRYPTODEV_NAME_ZUC_PMD crypto_zuc -/**< ZUC PMD device name */ - -/** ZUC PMD LOGTYPE DRIVER */ -extern int zuc_logtype_driver; -#define ZUC_LOG(level, fmt, ...) \ - rte_log(RTE_LOG_ ## level, zuc_logtype_driver, \ - "%s()... line %u: " fmt "\n", __func__, __LINE__, \ - ## __VA_ARGS__) - -#define ZUC_IV_KEY_LENGTH 16 -#define ZUC_DIGEST_LENGTH 4 - -/** private data structure for each virtual ZUC device */ -struct zuc_private { - unsigned max_nb_queue_pairs; - /**< Max number of queue pairs supported by device */ - MB_MGR *mb_mgr; - /**< Multi-buffer instance */ -}; - -/** ZUC buffer queue pair */ -struct zuc_qp { - uint16_t id; - /**< Queue Pair Identifier */ - char name[RTE_CRYPTODEV_NAME_MAX_LEN]; - /**< Unique Queue Pair Name */ - struct rte_ring *processed_ops; - /**< Ring for placing processed ops */ - struct rte_mempool *sess_mp; - /**< Session Mempool */ - struct rte_mempool *sess_mp_priv; - /**< Session Private Data Mempool */ - struct rte_cryptodev_stats qp_stats; - /**< Queue pair statistics */ - uint8_t temp_digest[ZUC_DIGEST_LENGTH]; - /**< Buffer used to store the digest generated - * by the driver when verifying a digest provided - * by the user (using authentication verify operation) - */ - MB_MGR *mb_mgr; - /**< Multi-buffer instance */ -} __rte_cache_aligned; - -enum zuc_operation { - ZUC_OP_ONLY_CIPHER, - ZUC_OP_ONLY_AUTH, - ZUC_OP_CIPHER_AUTH, - ZUC_OP_AUTH_CIPHER, - ZUC_OP_NOT_SUPPORTED -}; - -/** ZUC private session structure */ -struct zuc_session { - enum zuc_operation op; - enum rte_crypto_auth_operation auth_op; - uint8_t pKey_cipher[ZUC_IV_KEY_LENGTH]; - uint8_t pKey_hash[ZUC_IV_KEY_LENGTH]; - uint16_t cipher_iv_offset; - uint16_t auth_iv_offset; -} __rte_cache_aligned; - - -extern int -zuc_set_session_parameters(struct zuc_session *sess, - const struct rte_crypto_sym_xform *xform); - - -/** device specific operations function pointer structure */ -extern struct rte_cryptodev_ops *rte_zuc_pmd_ops; - - - -#endif /* _ZUC_PMD_PRIVATE_H_ */ -- 2.25.1