From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C2DCFA0524; Wed, 2 Jun 2021 19:47:05 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AC3AA410DF; Wed, 2 Jun 2021 19:47:05 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 5D77540E50 for ; Wed, 2 Jun 2021 19:47:04 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 152Hkajv006188; Wed, 2 Jun 2021 10:47:03 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=A7FEBxn0Iz4r9NNVVmGETkBAhhsM0R2TqkRnF9u1R7Q=; b=WQ7cp74SHpLxkBMVBw2bYTUUMOYT9vI/+zaD/imjwuL11UYeIkLnTJyEic2yTjG9lcNc T56ncM0GZlkHI7mFISQ0R4I0SvHasA2Jlk2sq9vV4k8CtXbhEdy6op4iSHgkMmDKm3Gx /GP6Nsz4+/S0ynHouiAA553sjqLmoaarOaqTWHVt8rZpSskE0pgaIgYNphs9GhdxwrUK s050uIIXO7kGbmFDxBA7MsvBorxsYOAfqLM/lwppFQ6sfX7CVzbAFLLsxY6uvEE5gCCZ N0kC9EKAujgAbTlgFRjVeGKaDNxYE3HUgKKWWG4DtGcWEH3IqogXkrPHiiSkZseyGx09 BQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com with ESMTP id 38xe7xr4cq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 02 Jun 2021 10:47:02 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 2 Jun 2021 10:47:01 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Wed, 2 Jun 2021 10:47:01 -0700 Received: from HY-LT1002.marvell.com (unknown [10.193.70.1]) by maili.marvell.com (Postfix) with ESMTP id 94F7F3F703F; Wed, 2 Jun 2021 10:46:56 -0700 (PDT) From: Anoob Joseph To: Akhil Goyal , Thomas Monjalon CC: Kiran Kumar K , Jerin Jacob , Ankur Dwivedi , Tejasree Kondoj , Date: Wed, 2 Jun 2021 23:16:33 +0530 Message-ID: <1622655994-24480-3-git-send-email-anoobj@marvell.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1622655994-24480-1-git-send-email-anoobj@marvell.com> References: <1622655994-24480-1-git-send-email-anoobj@marvell.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Proofpoint-ORIG-GUID: 1ZdwApE8LSv43lqgb5kglZ2n8oNGMQoa X-Proofpoint-GUID: 1ZdwApE8LSv43lqgb5kglZ2n8oNGMQoa X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-06-02_09:2021-06-02, 2021-06-02 signatures=0 Subject: [dpdk-dev] [PATCH 2/3] crypto/cnxk: add asymmetric datapath ops X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Kiran Kumar K Adding asymmetric crypto datapath ops. Signed-off-by: Kiran Kumar K --- drivers/crypto/cnxk/cn10k_cryptodev_ops.c | 108 ++++++ drivers/crypto/cnxk/cn9k_cryptodev_ops.c | 112 +++++- drivers/crypto/cnxk/cnxk_ae.h | 615 ++++++++++++++++++++++++++++++ 3 files changed, 833 insertions(+), 2 deletions(-) diff --git a/drivers/crypto/cnxk/cn10k_cryptodev_ops.c b/drivers/crypto/cnxk/cn10k_cryptodev_ops.c index f98028e..68818a6 100644 --- a/drivers/crypto/cnxk/cn10k_cryptodev_ops.c +++ b/drivers/crypto/cnxk/cn10k_cryptodev_ops.c @@ -10,6 +10,7 @@ #include "cn10k_cryptodev_ops.h" #include "cn10k_ipsec_la_ops.h" #include "cn10k_ipsec.h" +#include "cnxk_ae.h" #include "cnxk_cryptodev.h" #include "cnxk_cryptodev_ops.h" #include "cnxk_se.h" @@ -95,12 +96,71 @@ cpt_sym_inst_fill(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op, return ret; } +static __rte_always_inline int32_t __rte_hot +cn10k_ae_enqueue(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op, + struct cpt_inflight_req *infl_req, struct cpt_inst_s *inst, + struct cnxk_ae_sess *sess) +{ + struct cpt_qp_meta_info *minfo = &qp->meta_info; + struct rte_crypto_asym_op *asym_op = op->asym; + struct cnxk_ae_buf_ptr meta_buf; + uint64_t *mop; + void *mdata; + int ret; + + mdata = cnxk_ae_alloc_meta(&meta_buf, minfo->pool, infl_req); + if (mdata == NULL) + return -ENOMEM; + + /* Reserve 8B for RPTR */ + meta_buf.vaddr = PLT_PTR_ADD(mdata, sizeof(uint64_t)); + + switch (sess->xfrm_type) { + case RTE_CRYPTO_ASYM_XFORM_MODEX: + ret = cnxk_ae_modex_prep(op, &meta_buf, &sess->mod_ctx, inst); + if (unlikely(ret)) + goto req_fail; + break; + case RTE_CRYPTO_ASYM_XFORM_RSA: + ret = cnxk_ae_enqueue_rsa_op(op, &meta_buf, sess, inst); + if (unlikely(ret)) + goto req_fail; + break; + case RTE_CRYPTO_ASYM_XFORM_ECDSA: + ret = cnxk_ae_enqueue_ecdsa_op(op, &meta_buf, sess, + sess->cnxk_fpm_iova, inst); + if (unlikely(ret)) + goto req_fail; + break; + case RTE_CRYPTO_ASYM_XFORM_ECPM: + ret = cnxk_ae_ecpm_prep(&asym_op->ecpm, &meta_buf, + sess->ec_ctx.curveid, inst); + if (unlikely(ret)) + goto req_fail; + break; + default: + op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + ret = -EINVAL; + goto req_fail; + } + + mop = mdata; + mop[0] = inst->rptr; + return 0; + +req_fail: + rte_mempool_put(minfo->pool, infl_req->mdata); + return ret; +} + static inline int cn10k_cpt_fill_inst(struct cnxk_cpt_qp *qp, struct rte_crypto_op *ops[], struct cpt_inst_s inst[], struct cpt_inflight_req *infl_req) { struct cn10k_sec_session *sec_sess; + struct rte_crypto_asym_op *asym_op; struct rte_crypto_sym_op *sym_op; + struct cnxk_ae_sess *ae_sess; struct cnxk_se_sess *sess; struct rte_crypto_op *op; uint64_t w7; @@ -148,6 +208,21 @@ cn10k_cpt_fill_inst(struct cnxk_cpt_qp *qp, struct rte_crypto_op *ops[], } w7 = sess->cpt_inst_w7; } + } else if (op->type == RTE_CRYPTO_OP_TYPE_ASYMMETRIC) { + + if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) { + asym_op = op->asym; + ae_sess = get_asym_session_private_data( + asym_op->session, cn10k_cryptodev_driver_id); + ret = cn10k_ae_enqueue(qp, op, infl_req, &inst[0], + ae_sess); + if (unlikely(ret)) + return 0; + w7 = ae_sess->cpt_inst_w7; + } else { + CPT_LOG_DP_ERR("Not supported Asym op without session"); + return 0; + } } else { CPT_LOG_DP_ERR("Unsupported op type"); return 0; @@ -239,6 +314,35 @@ cn10k_cpt_enqueue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) return count + i; } +static void +cnxk_ae_post_process(struct rte_crypto_op *cop, uint8_t *rptr) +{ + struct rte_crypto_asym_op *op = cop->asym; + struct cnxk_ae_sess *sess; + + sess = get_asym_session_private_data(op->session, + cn10k_cryptodev_driver_id); + + switch (sess->xfrm_type) { + case RTE_CRYPTO_ASYM_XFORM_RSA: + cnxk_ae_dequeue_rsa_op(cop, rptr, &sess->rsa_ctx); + break; + case RTE_CRYPTO_ASYM_XFORM_MODEX: + op->modex.result.length = sess->mod_ctx.modulus.length; + memcpy(op->modex.result.data, rptr, op->modex.result.length); + break; + case RTE_CRYPTO_ASYM_XFORM_ECDSA: + cnxk_ae_dequeue_ecdsa_op(&op->ecdsa, rptr, &sess->ec_ctx); + break; + case RTE_CRYPTO_ASYM_XFORM_ECPM: + cnxk_ae_dequeue_ecpm_op(&op->ecpm, rptr, &sess->ec_ctx); + break; + default: + cop->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + break; + } +} + static inline void cn10k_cpt_sec_post_process(struct rte_crypto_op *cop, struct cpt_inflight_req *infl_req) @@ -303,6 +407,10 @@ cn10k_cpt_dequeue_post_process(struct cnxk_cpt_qp *qp, compl_auth_verify(cop, (uint8_t *)rsp[0], rsp[1]); } + } else if (cop->type == RTE_CRYPTO_OP_TYPE_ASYMMETRIC) { + uintptr_t *mdata = infl_req->mdata; + + cnxk_ae_post_process(cop, (uint8_t *)mdata[0]); } } else { cop->status = RTE_CRYPTO_OP_STATUS_ERROR; diff --git a/drivers/crypto/cnxk/cn9k_cryptodev_ops.c b/drivers/crypto/cnxk/cn9k_cryptodev_ops.c index 07be33d..c4b5832 100644 --- a/drivers/crypto/cnxk/cn9k_cryptodev_ops.c +++ b/drivers/crypto/cnxk/cn9k_cryptodev_ops.c @@ -7,10 +7,97 @@ #include "cn9k_cryptodev.h" #include "cn9k_cryptodev_ops.h" +#include "cnxk_ae.h" #include "cnxk_cryptodev.h" #include "cnxk_cryptodev_ops.h" #include "cnxk_se.h" +static __rte_always_inline int32_t __rte_hot +cn9k_ae_enqueue(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op, + struct cpt_inflight_req *infl_req, struct cpt_inst_s *inst, + struct cnxk_ae_sess *sess) +{ + struct cpt_qp_meta_info *minfo = &qp->meta_info; + struct rte_crypto_asym_op *asym_op = op->asym; + struct cnxk_ae_buf_ptr meta_buf; + uint64_t *mop; + void *mdata; + int ret; + + mdata = cnxk_ae_alloc_meta(&meta_buf, minfo->pool, infl_req); + if (mdata == NULL) + return -ENOMEM; + + /* Reserve 8B for RPTR */ + meta_buf.vaddr = PLT_PTR_ADD(mdata, sizeof(uint64_t)); + + switch (sess->xfrm_type) { + case RTE_CRYPTO_ASYM_XFORM_MODEX: + ret = cnxk_ae_modex_prep(op, &meta_buf, &sess->mod_ctx, inst); + if (unlikely(ret)) + goto req_fail; + break; + case RTE_CRYPTO_ASYM_XFORM_RSA: + ret = cnxk_ae_enqueue_rsa_op(op, &meta_buf, sess, inst); + if (unlikely(ret)) + goto req_fail; + break; + case RTE_CRYPTO_ASYM_XFORM_ECDSA: + ret = cnxk_ae_enqueue_ecdsa_op(op, &meta_buf, sess, + sess->cnxk_fpm_iova, inst); + if (unlikely(ret)) + goto req_fail; + break; + case RTE_CRYPTO_ASYM_XFORM_ECPM: + ret = cnxk_ae_ecpm_prep(&asym_op->ecpm, &meta_buf, + sess->ec_ctx.curveid, inst); + if (unlikely(ret)) + goto req_fail; + break; + default: + op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + ret = -EINVAL; + goto req_fail; + } + + mop = mdata; + mop[0] = inst->rptr; + return 0; + +req_fail: + rte_mempool_put(minfo->pool, infl_req->mdata); + return ret; +} + +static void +cnxk_ae_post_process(struct rte_crypto_op *cop, uint8_t *rptr) +{ + struct rte_crypto_asym_op *op = cop->asym; + struct cnxk_ae_sess *sess; + + sess = get_asym_session_private_data(op->session, + cn9k_cryptodev_driver_id); + + switch (sess->xfrm_type) { + case RTE_CRYPTO_ASYM_XFORM_RSA: + cnxk_ae_dequeue_rsa_op(cop, rptr, &sess->rsa_ctx); + break; + case RTE_CRYPTO_ASYM_XFORM_MODEX: + op->modex.result.length = sess->mod_ctx.modulus.length; + memcpy(op->modex.result.data, rptr, op->modex.result.length); + break; + case RTE_CRYPTO_ASYM_XFORM_ECDSA: + cnxk_ae_dequeue_ecdsa_op(&op->ecdsa, rptr, &sess->ec_ctx); + break; + case RTE_CRYPTO_ASYM_XFORM_ECPM: + cnxk_ae_dequeue_ecpm_op(&op->ecpm, rptr, &sess->ec_ctx); + break; + default: + cop->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + break; + } +} + static __rte_always_inline int __rte_hot cn9k_cpt_sym_inst_fill(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op, struct cnxk_se_sess *sess, @@ -65,11 +152,11 @@ static uint16_t cn9k_cpt_enqueue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) { struct cpt_inflight_req *infl_req; + struct rte_crypto_asym_op *asym_op; struct rte_crypto_sym_op *sym_op; uint16_t nb_allowed, count = 0; struct cnxk_cpt_qp *qp = qptr; struct pending_queue *pend_q; - struct cnxk_se_sess *sess; struct rte_crypto_op *op; struct cpt_inst_s inst; uint64_t lmt_status; @@ -95,6 +182,8 @@ cn9k_cpt_enqueue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) infl_req->op_flags = 0; if (op->type == RTE_CRYPTO_OP_TYPE_SYMMETRIC) { + struct cnxk_se_sess *sess; + if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) { sym_op = op->sym; sess = get_sym_session_private_data( @@ -120,6 +209,22 @@ cn9k_cpt_enqueue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) op->sym->session); } } + inst.w7.u64 = sess->cpt_inst_w7; + } else if (op->type == RTE_CRYPTO_OP_TYPE_ASYMMETRIC) { + struct cnxk_ae_sess *sess; + + ret = -EINVAL; + if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) { + asym_op = op->asym; + sess = get_asym_session_private_data( + asym_op->session, + cn9k_cryptodev_driver_id); + ret = cn9k_ae_enqueue(qp, op, infl_req, &inst, + sess); + if (unlikely(ret)) + return 0; + inst.w7.u64 = sess->cpt_inst_w7; + } } else { CPT_LOG_DP_ERR("Unsupported op type"); break; @@ -134,7 +239,6 @@ cn9k_cpt_enqueue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) infl_req->res.cn9k.compcode = CPT_COMP_NOT_DONE; inst.res_addr = (uint64_t)&infl_req->res; - inst.w7.u64 = sess->cpt_inst_w7; do { /* Copy CPT command to LMTLINE */ @@ -189,6 +293,10 @@ cn9k_cpt_dequeue_post_process(struct cnxk_cpt_qp *qp, struct rte_crypto_op *cop, compl_auth_verify(cop, (uint8_t *)rsp[0], rsp[1]); } + } else if (cop->type == RTE_CRYPTO_OP_TYPE_ASYMMETRIC) { + uintptr_t *mdata = infl_req->mdata; + + cnxk_ae_post_process(cop, (uint8_t *)mdata[0]); } } else { cop->status = RTE_CRYPTO_OP_STATUS_ERROR; diff --git a/drivers/crypto/cnxk/cnxk_ae.h b/drivers/crypto/cnxk/cnxk_ae.h index 5659c0d..0724429 100644 --- a/drivers/crypto/cnxk/cnxk_ae.h +++ b/drivers/crypto/cnxk/cnxk_ae.h @@ -12,6 +12,8 @@ #include "roc_ae.h" #include "roc_ae_fpm_tables.h" +#include "cnxk_cryptodev_ops.h" + struct cnxk_ae_sess { enum rte_crypto_asym_xform_type xfrm_type; union { @@ -23,6 +25,11 @@ struct cnxk_ae_sess { uint64_t cpt_inst_w7; }; +/* Buffer pointer */ +struct cnxk_ae_buf_ptr { + void *vaddr; +}; + static __rte_always_inline void cnxk_ae_modex_param_normalize(uint8_t **data, size_t *len) { @@ -207,4 +214,612 @@ cnxk_ae_free_session_parameters(struct cnxk_ae_sess *sess) break; } } + +static const struct roc_ae_ec_group cnxk_ae_ec_grp[ROC_AE_EC_ID_PMAX] = { + { + .prime = {.data = {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFE, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF}, + .length = 24}, + .order = {.data = {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0x99, 0xDE, 0xF8, 0x36, 0x14, 0x6B, + 0xC9, 0xB1, 0xB4, 0xD2, 0x28, 0x31}, + .length = 24}, + }, + { + .prime = {.data = {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01}, + .length = 28}, + .order = {.data = {0XFF, 0XFF, 0XFF, 0XFF, 0XFF, 0XFF, 0XFF, + 0XFF, 0XFF, 0XFF, 0XFF, 0XFF, 0XFF, 0XFF, + 0X16, 0XA2, 0XE0, 0XB8, 0XF0, 0X3E, 0X13, + 0XDD, 0X29, 0X45, 0X5C, 0X5C, 0X2A, 0X3D}, + .length = 28}, + }, + { + .prime = {.data = {0xFF, 0xFF, 0xFF, 0xFF, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF}, + .length = 32}, + .order = {.data = {0xFF, 0xFF, 0xFF, 0xFF, 0x00, 0x00, 0x00, + 0x00, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xBC, 0xE6, 0xFA, 0xAD, 0xA7, + 0x17, 0x9E, 0x84, 0xF3, 0xB9, 0xCA, 0xC2, + 0xFC, 0x63, 0x25, 0x51}, + .length = 32}, + }, + {.prime = {.data = {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFE, + 0xFF, 0xFF, 0xFF, 0xFF, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0xFF, 0xFF, 0xFF, 0xFF}, + .length = 48}, + .order = {.data = {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xC7, 0x63, 0x4D, 0x81, 0xF4, 0x37, 0x2D, 0xDF, + 0x58, 0x1A, 0x0D, 0xB2, 0x48, 0xB0, 0xA7, 0x7A, + 0xEC, 0xEC, 0x19, 0x6A, 0xCC, 0xC5, 0x29, 0x73}, + .length = 48}}, + {.prime = {.data = {0x01, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF}, + .length = 66}, + .order = {.data = {0x01, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFA, 0x51, 0x86, 0x87, 0x83, 0xBF, 0x2F, + 0x96, 0x6B, 0x7F, 0xCC, 0x01, 0x48, 0xF7, 0x09, + 0xA5, 0xD0, 0x3B, 0xB5, 0xC9, 0xB8, 0x89, 0x9C, + 0x47, 0xAE, 0xBB, 0x6F, 0xB7, 0x1E, 0x91, 0x38, + 0x64, 0x09}, + .length = 66}}}; + +static __rte_always_inline int +cnxk_ae_modex_prep(struct rte_crypto_op *op, struct cnxk_ae_buf_ptr *meta_buf, + struct rte_crypto_modex_xform *mod, struct cpt_inst_s *inst) +{ + uint32_t exp_len = mod->exponent.length; + uint32_t mod_len = mod->modulus.length; + struct rte_crypto_mod_op_param mod_op; + uint64_t total_key_len; + union cpt_inst_w4 w4; + uint32_t base_len; + uint32_t dlen; + uint8_t *dptr; + + mod_op = op->asym->modex; + + base_len = mod_op.base.length; + if (unlikely(base_len > mod_len)) { + op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + return -ENOTSUP; + } + + total_key_len = mod_len + exp_len; + + /* Input buffer */ + dptr = meta_buf->vaddr; + inst->dptr = (uintptr_t)dptr; + memcpy(dptr, mod->modulus.data, total_key_len); + dptr += total_key_len; + memcpy(dptr, mod_op.base.data, base_len); + dptr += base_len; + dlen = total_key_len + base_len; + + /* Setup opcodes */ + w4.s.opcode_major = ROC_AE_MAJOR_OP_MODEX; + w4.s.opcode_minor = ROC_AE_MINOR_OP_MODEX; + + w4.s.param1 = mod_len; + w4.s.param2 = exp_len; + w4.s.dlen = dlen; + + inst->w4.u64 = w4.u64; + inst->rptr = (uintptr_t)dptr; + + return 0; +} + +static __rte_always_inline void +cnxk_ae_rsa_prep(struct rte_crypto_op *op, struct cnxk_ae_buf_ptr *meta_buf, + struct rte_crypto_rsa_xform *rsa, + rte_crypto_param *crypto_param, struct cpt_inst_s *inst) +{ + struct rte_crypto_rsa_op_param rsa_op; + uint32_t mod_len = rsa->n.length; + uint32_t exp_len = rsa->e.length; + uint64_t total_key_len; + union cpt_inst_w4 w4; + uint32_t in_size; + uint32_t dlen; + uint8_t *dptr; + + rsa_op = op->asym->rsa; + total_key_len = mod_len + exp_len; + + /* Input buffer */ + dptr = meta_buf->vaddr; + inst->dptr = (uintptr_t)dptr; + memcpy(dptr, rsa->n.data, total_key_len); + dptr += total_key_len; + + in_size = crypto_param->length; + memcpy(dptr, crypto_param->data, in_size); + + dptr += in_size; + dlen = total_key_len + in_size; + + if (rsa_op.pad == RTE_CRYPTO_RSA_PADDING_NONE) { + /* Use mod_exp operation for no_padding type */ + w4.s.opcode_minor = ROC_AE_MINOR_OP_MODEX; + w4.s.param2 = exp_len; + } else { + if (rsa_op.op_type == RTE_CRYPTO_ASYM_OP_ENCRYPT) { + w4.s.opcode_minor = ROC_AE_MINOR_OP_PKCS_ENC; + /* Public key encrypt, use BT2*/ + w4.s.param2 = ROC_AE_CPT_BLOCK_TYPE2 | + ((uint16_t)(exp_len) << 1); + } else if (rsa_op.op_type == RTE_CRYPTO_ASYM_OP_VERIFY) { + w4.s.opcode_minor = ROC_AE_MINOR_OP_PKCS_DEC; + /* Public key decrypt, use BT1 */ + w4.s.param2 = ROC_AE_CPT_BLOCK_TYPE1; + } + } + + w4.s.opcode_major = ROC_AE_MAJOR_OP_MODEX; + + w4.s.param1 = mod_len; + w4.s.dlen = dlen; + + inst->w4.u64 = w4.u64; + inst->rptr = (uintptr_t)dptr; +} + +static __rte_always_inline void +cnxk_ae_rsa_crt_prep(struct rte_crypto_op *op, struct cnxk_ae_buf_ptr *meta_buf, + struct rte_crypto_rsa_xform *rsa, + rte_crypto_param *crypto_param, struct cpt_inst_s *inst) +{ + uint32_t qInv_len = rsa->qt.qInv.length; + struct rte_crypto_rsa_op_param rsa_op; + uint32_t dP_len = rsa->qt.dP.length; + uint32_t dQ_len = rsa->qt.dQ.length; + uint32_t p_len = rsa->qt.p.length; + uint32_t q_len = rsa->qt.q.length; + uint32_t mod_len = rsa->n.length; + uint64_t total_key_len; + union cpt_inst_w4 w4; + uint32_t in_size; + uint32_t dlen; + uint8_t *dptr; + + rsa_op = op->asym->rsa; + total_key_len = p_len + q_len + dP_len + dQ_len + qInv_len; + + /* Input buffer */ + dptr = meta_buf->vaddr; + inst->dptr = (uintptr_t)dptr; + memcpy(dptr, rsa->qt.q.data, total_key_len); + dptr += total_key_len; + + in_size = crypto_param->length; + memcpy(dptr, crypto_param->data, in_size); + + dptr += in_size; + dlen = total_key_len + in_size; + + if (rsa_op.pad == RTE_CRYPTO_RSA_PADDING_NONE) { + /*Use mod_exp operation for no_padding type */ + w4.s.opcode_minor = ROC_AE_MINOR_OP_MODEX_CRT; + } else { + if (rsa_op.op_type == RTE_CRYPTO_ASYM_OP_SIGN) { + w4.s.opcode_minor = ROC_AE_MINOR_OP_PKCS_ENC_CRT; + /* Private encrypt, use BT1 */ + w4.s.param2 = ROC_AE_CPT_BLOCK_TYPE1; + } else if (rsa_op.op_type == RTE_CRYPTO_ASYM_OP_DECRYPT) { + w4.s.opcode_minor = ROC_AE_MINOR_OP_PKCS_DEC_CRT; + /* Private decrypt, use BT2 */ + w4.s.param2 = ROC_AE_CPT_BLOCK_TYPE2; + } + } + + w4.s.opcode_major = ROC_AE_MAJOR_OP_MODEX; + + w4.s.param1 = mod_len; + w4.s.dlen = dlen; + + inst->w4.u64 = w4.u64; + inst->rptr = (uintptr_t)dptr; +} + +static __rte_always_inline int __rte_hot +cnxk_ae_enqueue_rsa_op(struct rte_crypto_op *op, + struct cnxk_ae_buf_ptr *meta_buf, + struct cnxk_ae_sess *sess, struct cpt_inst_s *inst) +{ + struct rte_crypto_rsa_op_param *rsa = &op->asym->rsa; + + switch (rsa->op_type) { + case RTE_CRYPTO_ASYM_OP_VERIFY: + cnxk_ae_rsa_prep(op, meta_buf, &sess->rsa_ctx, &rsa->sign, + inst); + break; + case RTE_CRYPTO_ASYM_OP_ENCRYPT: + cnxk_ae_rsa_prep(op, meta_buf, &sess->rsa_ctx, &rsa->message, + inst); + break; + case RTE_CRYPTO_ASYM_OP_SIGN: + cnxk_ae_rsa_crt_prep(op, meta_buf, &sess->rsa_ctx, + &rsa->message, inst); + break; + case RTE_CRYPTO_ASYM_OP_DECRYPT: + cnxk_ae_rsa_crt_prep(op, meta_buf, &sess->rsa_ctx, &rsa->cipher, + inst); + break; + default: + op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + return -EINVAL; + } + return 0; +} + +static __rte_always_inline void +cnxk_ae_ecdsa_sign_prep(struct rte_crypto_ecdsa_op_param *ecdsa, + struct cnxk_ae_buf_ptr *meta_buf, + uint64_t fpm_table_iova, uint8_t curveid, + struct cpt_inst_s *inst) +{ + uint16_t message_len = ecdsa->message.length; + uint16_t pkey_len = ecdsa->pkey.length; + uint16_t p_align, k_align, m_align; + uint16_t k_len = ecdsa->k.length; + uint16_t order_len, prime_len; + uint16_t o_offset, pk_offset; + union cpt_inst_w4 w4; + uint16_t dlen; + uint8_t *dptr; + + prime_len = cnxk_ae_ec_grp[curveid].prime.length; + order_len = cnxk_ae_ec_grp[curveid].order.length; + + /* Truncate input length to curve prime length */ + if (message_len > prime_len) + message_len = prime_len; + m_align = RTE_ALIGN_CEIL(message_len, 8); + + p_align = RTE_ALIGN_CEIL(prime_len, 8); + k_align = RTE_ALIGN_CEIL(k_len, 8); + + /* Set write offset for order and private key */ + o_offset = prime_len - order_len; + pk_offset = prime_len - pkey_len; + + /* Input buffer */ + dptr = meta_buf->vaddr; + inst->dptr = (uintptr_t)dptr; + + /* + * Set dlen = sum(sizeof(fpm address), ROUNDUP8(scalar len, input len), + * ROUNDUP8(priv key len, prime len, order len)). + * Please note, private key, order cannot exceed prime + * length i.e 3 * p_align. + */ + dlen = sizeof(fpm_table_iova) + k_align + m_align + p_align * 3; + + memset(dptr, 0, dlen); + + *(uint64_t *)dptr = fpm_table_iova; + dptr += sizeof(fpm_table_iova); + + memcpy(dptr, ecdsa->k.data, k_len); + dptr += k_align; + + memcpy(dptr, cnxk_ae_ec_grp[curveid].prime.data, prime_len); + dptr += p_align; + + memcpy(dptr + o_offset, cnxk_ae_ec_grp[curveid].order.data, order_len); + dptr += p_align; + + memcpy(dptr + pk_offset, ecdsa->pkey.data, pkey_len); + dptr += p_align; + + memcpy(dptr, ecdsa->message.data, message_len); + dptr += m_align; + + /* Setup opcodes */ + w4.s.opcode_major = ROC_AE_MAJOR_OP_ECDSA; + w4.s.opcode_minor = ROC_AE_MINOR_OP_ECDSA_SIGN; + + w4.s.param1 = curveid | (message_len << 8); + w4.s.param2 = k_len; + w4.s.dlen = dlen; + + inst->w4.u64 = w4.u64; + inst->rptr = (uintptr_t)dptr; +} + +static __rte_always_inline void +cnxk_ae_ecdsa_verify_prep(struct rte_crypto_ecdsa_op_param *ecdsa, + struct cnxk_ae_buf_ptr *meta_buf, + uint64_t fpm_table_iova, uint8_t curveid, + struct cpt_inst_s *inst) +{ + uint32_t message_len = ecdsa->message.length; + uint16_t o_offset, r_offset, s_offset; + uint16_t qx_len = ecdsa->q.x.length; + uint16_t qy_len = ecdsa->q.y.length; + uint16_t r_len = ecdsa->r.length; + uint16_t s_len = ecdsa->s.length; + uint16_t order_len, prime_len; + uint16_t qx_offset, qy_offset; + uint16_t p_align, m_align; + union cpt_inst_w4 w4; + uint16_t dlen; + uint8_t *dptr; + + prime_len = cnxk_ae_ec_grp[curveid].prime.length; + order_len = cnxk_ae_ec_grp[curveid].order.length; + + /* Truncate input length to curve prime length */ + if (message_len > prime_len) + message_len = prime_len; + + m_align = RTE_ALIGN_CEIL(message_len, 8); + p_align = RTE_ALIGN_CEIL(prime_len, 8); + + /* Set write offset for sign, order and public key coordinates */ + o_offset = prime_len - order_len; + qx_offset = prime_len - qx_len; + qy_offset = prime_len - qy_len; + r_offset = prime_len - r_len; + s_offset = prime_len - s_len; + + /* Input buffer */ + dptr = meta_buf->vaddr; + inst->dptr = (uintptr_t)dptr; + + /* + * Set dlen = sum(sizeof(fpm address), ROUNDUP8(message len), + * ROUNDUP8(sign len(r and s), public key len(x and y coordinates), + * prime len, order len)). + * Please note sign, public key and order can not exceed prime length + * i.e. 6 * p_align + */ + dlen = sizeof(fpm_table_iova) + m_align + (6 * p_align); + + memset(dptr, 0, dlen); + + *(uint64_t *)dptr = fpm_table_iova; + dptr += sizeof(fpm_table_iova); + + memcpy(dptr + r_offset, ecdsa->r.data, r_len); + dptr += p_align; + + memcpy(dptr + s_offset, ecdsa->s.data, s_len); + dptr += p_align; + + memcpy(dptr, ecdsa->message.data, message_len); + dptr += m_align; + + memcpy(dptr + o_offset, cnxk_ae_ec_grp[curveid].order.data, order_len); + dptr += p_align; + + memcpy(dptr, cnxk_ae_ec_grp[curveid].prime.data, prime_len); + dptr += p_align; + + memcpy(dptr + qx_offset, ecdsa->q.x.data, qx_len); + dptr += p_align; + + memcpy(dptr + qy_offset, ecdsa->q.y.data, qy_len); + dptr += p_align; + + /* Setup opcodes */ + w4.s.opcode_major = ROC_AE_MAJOR_OP_ECDSA; + w4.s.opcode_minor = ROC_AE_MINOR_OP_ECDSA_VERIFY; + + w4.s.param1 = curveid | (message_len << 8); + w4.s.param2 = 0; + w4.s.dlen = dlen; + + inst->w4.u64 = w4.u64; + inst->rptr = (uintptr_t)dptr; +} + +static __rte_always_inline int __rte_hot +cnxk_ae_enqueue_ecdsa_op(struct rte_crypto_op *op, + struct cnxk_ae_buf_ptr *meta_buf, + struct cnxk_ae_sess *sess, uint64_t *fpm_iova, + struct cpt_inst_s *inst) +{ + struct rte_crypto_ecdsa_op_param *ecdsa = &op->asym->ecdsa; + uint8_t curveid = sess->ec_ctx.curveid; + + if (ecdsa->op_type == RTE_CRYPTO_ASYM_OP_SIGN) + cnxk_ae_ecdsa_sign_prep(ecdsa, meta_buf, fpm_iova[curveid], + curveid, inst); + else if (ecdsa->op_type == RTE_CRYPTO_ASYM_OP_VERIFY) + cnxk_ae_ecdsa_verify_prep(ecdsa, meta_buf, fpm_iova[curveid], + curveid, inst); + else { + op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + return -EINVAL; + } + return 0; +} + +static __rte_always_inline int +cnxk_ae_ecpm_prep(struct rte_crypto_ecpm_op_param *ecpm, + struct cnxk_ae_buf_ptr *meta_buf, uint8_t curveid, + struct cpt_inst_s *inst) +{ + uint16_t x1_len = ecpm->p.x.length; + uint16_t y1_len = ecpm->p.y.length; + uint16_t scalar_align, p_align; + uint16_t x1_offset, y1_offset; + uint16_t dlen, prime_len; + union cpt_inst_w4 w4; + uint8_t *dptr; + + prime_len = cnxk_ae_ec_grp[curveid].prime.length; + + /* Input buffer */ + dptr = meta_buf->vaddr; + inst->dptr = (uintptr_t)dptr; + + p_align = RTE_ALIGN_CEIL(prime_len, 8); + scalar_align = RTE_ALIGN_CEIL(ecpm->scalar.length, 8); + + /* + * Set dlen = sum(ROUNDUP8(input point(x and y coordinates), prime, + * scalar length), + * Please note point length is equivalent to prime of the curve + */ + dlen = 3 * p_align + scalar_align; + + x1_offset = prime_len - x1_len; + y1_offset = prime_len - y1_len; + + memset(dptr, 0, dlen); + + /* Copy input point, scalar, prime */ + memcpy(dptr + x1_offset, ecpm->p.x.data, x1_len); + dptr += p_align; + memcpy(dptr + y1_offset, ecpm->p.y.data, y1_len); + dptr += p_align; + memcpy(dptr, ecpm->scalar.data, ecpm->scalar.length); + dptr += scalar_align; + memcpy(dptr, cnxk_ae_ec_grp[curveid].prime.data, + cnxk_ae_ec_grp[curveid].prime.length); + dptr += p_align; + + /* Setup opcodes */ + w4.s.opcode_major = ROC_AE_MAJOR_OP_ECC; + w4.s.opcode_minor = ROC_AE_MINOR_OP_ECC_UMP; + + w4.s.param1 = curveid; + w4.s.param2 = ecpm->scalar.length; + w4.s.dlen = dlen; + + inst->w4.u64 = w4.u64; + inst->rptr = (uintptr_t)dptr; + + return 0; +} + +static __rte_always_inline void +cnxk_ae_dequeue_rsa_op(struct rte_crypto_op *cop, uint8_t *rptr, + struct rte_crypto_rsa_xform *rsa_ctx) +{ + struct rte_crypto_rsa_op_param *rsa = &cop->asym->rsa; + + switch (rsa->op_type) { + case RTE_CRYPTO_ASYM_OP_ENCRYPT: + rsa->cipher.length = rsa_ctx->n.length; + memcpy(rsa->cipher.data, rptr, rsa->cipher.length); + break; + case RTE_CRYPTO_ASYM_OP_DECRYPT: + if (rsa->pad == RTE_CRYPTO_RSA_PADDING_NONE) { + rsa->message.length = rsa_ctx->n.length; + memcpy(rsa->message.data, rptr, rsa->message.length); + } else { + /* Get length of decrypted output */ + rsa->message.length = + rte_cpu_to_be_16(*((uint16_t *)rptr)); + /* + * Offset output data pointer by length field + * (2 bytes) and copy decrypted data. + */ + memcpy(rsa->message.data, rptr + 2, + rsa->message.length); + } + break; + case RTE_CRYPTO_ASYM_OP_SIGN: + rsa->sign.length = rsa_ctx->n.length; + memcpy(rsa->sign.data, rptr, rsa->sign.length); + break; + case RTE_CRYPTO_ASYM_OP_VERIFY: + if (rsa->pad == RTE_CRYPTO_RSA_PADDING_NONE) { + rsa->sign.length = rsa_ctx->n.length; + memcpy(rsa->sign.data, rptr, rsa->sign.length); + } else { + /* Get length of signed output */ + rsa->sign.length = + rte_cpu_to_be_16(*((uint16_t *)rptr)); + /* + * Offset output data pointer by length field + * (2 bytes) and copy signed data. + */ + memcpy(rsa->sign.data, rptr + 2, rsa->sign.length); + } + if (memcmp(rsa->sign.data, rsa->message.data, + rsa->message.length)) { + cop->status = RTE_CRYPTO_OP_STATUS_ERROR; + } + break; + default: + cop->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + break; + } +} + +static __rte_always_inline void +cnxk_ae_dequeue_ecdsa_op(struct rte_crypto_ecdsa_op_param *ecdsa, uint8_t *rptr, + struct roc_ae_ec_ctx *ec) +{ + int prime_len = cnxk_ae_ec_grp[ec->curveid].prime.length; + + if (ecdsa->op_type == RTE_CRYPTO_ASYM_OP_VERIFY) + return; + + /* Separate out sign r and s components */ + memcpy(ecdsa->r.data, rptr, prime_len); + memcpy(ecdsa->s.data, rptr + RTE_ALIGN_CEIL(prime_len, 8), prime_len); + ecdsa->r.length = prime_len; + ecdsa->s.length = prime_len; +} + +static __rte_always_inline void +cnxk_ae_dequeue_ecpm_op(struct rte_crypto_ecpm_op_param *ecpm, uint8_t *rptr, + struct roc_ae_ec_ctx *ec) +{ + int prime_len = cnxk_ae_ec_grp[ec->curveid].prime.length; + + memcpy(ecpm->r.x.data, rptr, prime_len); + memcpy(ecpm->r.y.data, rptr + RTE_ALIGN_CEIL(prime_len, 8), prime_len); + ecpm->r.x.length = prime_len; + ecpm->r.y.length = prime_len; +} + +static __rte_always_inline void * +cnxk_ae_alloc_meta(struct cnxk_ae_buf_ptr *buf, + struct rte_mempool *cpt_meta_pool, + struct cpt_inflight_req *infl_req) +{ + uint8_t *mdata; + + if (unlikely(rte_mempool_get(cpt_meta_pool, (void **)&mdata) < 0)) + return NULL; + + buf->vaddr = mdata; + + infl_req->mdata = mdata; + infl_req->op_flags |= CPT_OP_FLAGS_METABUF; + + return mdata; +} #endif /* _CNXK_AE_H_ */ -- 2.7.4