From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7CE81A04BC; Thu, 8 Oct 2020 12:35:50 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D4FDA1BF33; Thu, 8 Oct 2020 12:35:12 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by dpdk.org (Postfix) with ESMTP id 92BBC1BF33 for ; Thu, 8 Oct 2020 12:35:10 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 098AUIwB024748; Thu, 8 Oct 2020 03:35:08 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=8XGkzARvFyxaej/HcV841k5s+11yvPE8dd2Yfx436LM=; b=MupNEumZFlFzhX75LmHdfd9ZufJn0u5T38Sz/OwuFHoo8nmNpsP93uSN2r0hHUDjzTU2 Ge5OA9Ml+mRmhxFxuONANZuBbBgU/UvzTSdJ7GTLN5oWQFqqpWy6q9/l78tXfCuL6Z7R VPOrsdYT0WEoAiqgHgnSWLb3sUupcvPVTLtuQgreSg70XRJuT2wrsQjJEZq88kPOPZx/ zRoGPyffwB18XhTbTSuEgZIywAkhaUfKZYYViWdugTRPiT5vwDYiY7lD0ZbBTfoIontV tOc+q2UMzkK5VsrPViqiNctP9dHGoUzWXtJ4Xi/XYn42ZeWkh/Tru3iEp3F7i1g2Xktr aQ== Received: from sc-exch03.marvell.com ([199.233.58.183]) by mx0a-0016f401.pphosted.com with ESMTP id 33xpnq0duh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 08 Oct 2020 03:35:08 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by SC-EXCH03.marvell.com (10.93.176.83) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 8 Oct 2020 03:35:07 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 8 Oct 2020 03:35:07 -0700 Received: from hyd1349.t110.caveonetworks.com (unknown [10.29.45.13]) by maili.marvell.com (Postfix) with ESMTP id 9E3853F703F; Thu, 8 Oct 2020 03:35:05 -0700 (PDT) From: Ankur Dwivedi To: CC: , , , , Ankur Dwivedi Date: Thu, 8 Oct 2020 16:04:35 +0530 Message-ID: <20201008103435.19187-4-adwivedi@marvell.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201008103435.19187-1-adwivedi@marvell.com> References: <20201008055423.32259-1-adwivedi@marvell.com> <20201008103435.19187-1-adwivedi@marvell.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235, 18.0.687 definitions=2020-10-08_04:2020-10-08, 2020-10-08 signatures=0 Subject: [dpdk-dev] [PATCH v4 3/3] event/octeontx2: add crypto adapter datapath X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" In the op new mode of crypto adapter, the completed crypto operation is submitted to the event device by the OCTEON TX2 crypto PMD. During event device dequeue the result of crypto operation is checked. Signed-off-by: Ankur Dwivedi --- drivers/common/cpt/cpt_common.h | 1 + drivers/crypto/octeontx2/otx2_cryptodev_ops.c | 47 ++++++++++++ .../rte_pmd_octeontx2_crypto_version.map | 1 + .../octeontx2/otx2_evdev_crypto_adptr_dp.h | 75 +++++++++++++++++++ drivers/event/octeontx2/otx2_worker.h | 28 ++++--- drivers/event/octeontx2/otx2_worker_dual.h | 44 ++++++----- 6 files changed, 168 insertions(+), 28 deletions(-) create mode 100644 drivers/event/octeontx2/otx2_evdev_crypto_adptr_dp.h diff --git a/drivers/common/cpt/cpt_common.h b/drivers/common/cpt/cpt_common.h index 0141b2aed..1ce28e90b 100644 --- a/drivers/common/cpt/cpt_common.h +++ b/drivers/common/cpt/cpt_common.h @@ -72,6 +72,7 @@ struct cpt_request_info { uint64_t ei3; } ist; uint8_t *rptr; + const struct otx2_cpt_qp *qp; /** Control path fields */ uint64_t time_out; diff --git a/drivers/crypto/octeontx2/otx2_cryptodev_ops.c b/drivers/crypto/octeontx2/otx2_cryptodev_ops.c index 027fd0d47..9125369d3 100644 --- a/drivers/crypto/octeontx2/otx2_cryptodev_ops.c +++ b/drivers/crypto/octeontx2/otx2_cryptodev_ops.c @@ -417,6 +417,48 @@ sym_session_configure(int driver_id, struct rte_crypto_sym_xform *xform, return -ENOTSUP; } +static __rte_always_inline void __rte_hot +otx2_ca_enqueue_req(const struct otx2_cpt_qp *qp, + struct cpt_request_info *req, + void *lmtline) +{ + union cpt_inst_s inst; + uint64_t lmt_status; + + inst.u[0] = 0; + inst.s9x.res_addr = req->comp_baddr; + inst.u[2] = 0; + inst.u[3] = 0; + + inst.s9x.ei0 = req->ist.ei0; + inst.s9x.ei1 = req->ist.ei1; + inst.s9x.ei2 = req->ist.ei2; + inst.s9x.ei3 = req->ist.ei3; + + inst.s9x.qord = 1; + inst.s9x.grp = qp->ev.queue_id; + inst.s9x.tt = qp->ev.sched_type; + inst.s9x.tag = (RTE_EVENT_TYPE_CRYPTODEV << 28) | + qp->ev.flow_id; + inst.s9x.wq_ptr = (uint64_t)req >> 3; + req->qp = qp; + + do { + /* Copy CPT command to LMTLINE */ + memcpy(lmtline, &inst, sizeof(inst)); + + /* + * Make sure compiler does not reorder memcpy and ldeor. + * LMTST transactions are always flushed from the write + * buffer immediately, a DMB is not required to push out + * LMTSTs. + */ + rte_io_wmb(); + lmt_status = otx2_lmt_submit(qp->lf_nq_reg); + } while (lmt_status == 0); + +} + static __rte_always_inline int32_t __rte_hot otx2_cpt_enqueue_req(const struct otx2_cpt_qp *qp, struct pending_queue *pend_q, @@ -426,6 +468,11 @@ otx2_cpt_enqueue_req(const struct otx2_cpt_qp *qp, union cpt_inst_s inst; uint64_t lmt_status; + if (qp->ca_enable) { + otx2_ca_enqueue_req(qp, req, lmtline); + return 0; + } + if (unlikely(pend_q->pending_count >= OTX2_CPT_DEFAULT_CMD_QLEN)) return -EAGAIN; diff --git a/drivers/crypto/octeontx2/rte_pmd_octeontx2_crypto_version.map b/drivers/crypto/octeontx2/rte_pmd_octeontx2_crypto_version.map index 95ebda255..02684781b 100644 --- a/drivers/crypto/octeontx2/rte_pmd_octeontx2_crypto_version.map +++ b/drivers/crypto/octeontx2/rte_pmd_octeontx2_crypto_version.map @@ -5,6 +5,7 @@ DPDK_21 { INTERNAL { global: + otx2_cryptodev_driver_id; otx2_cpt_af_reg_read; otx2_cpt_af_reg_write; diff --git a/drivers/event/octeontx2/otx2_evdev_crypto_adptr_dp.h b/drivers/event/octeontx2/otx2_evdev_crypto_adptr_dp.h new file mode 100644 index 000000000..70b63933e --- /dev/null +++ b/drivers/event/octeontx2/otx2_evdev_crypto_adptr_dp.h @@ -0,0 +1,75 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (C) 2020 Marvell International Ltd. + */ + +#ifndef _OTX2_EVDEV_CRYPTO_ADPTR_DP_H_ +#define _OTX2_EVDEV_CRYPTO_ADPTR_DP_H_ + +#include +#include +#include + +#include "cpt_pmd_logs.h" +#include "cpt_ucode.h" + +#include "otx2_cryptodev.h" +#include "otx2_cryptodev_hw_access.h" +#include "otx2_cryptodev_ops_helper.h" +#include "otx2_cryptodev_qp.h" + +static inline void +otx2_ca_deq_post_process(const struct otx2_cpt_qp *qp, + struct rte_crypto_op *cop, uintptr_t *rsp, + uint8_t cc) +{ + if (cop->type == RTE_CRYPTO_OP_TYPE_SYMMETRIC) { + if (likely(cc == NO_ERR)) { + /* Verify authentication data if required */ + if (unlikely(rsp[2])) + compl_auth_verify(cop, (uint8_t *)rsp[2], + rsp[3]); + else + cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS; + } else { + if (cc == ERR_GC_ICV_MISCOMPARE) + cop->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; + else + cop->status = RTE_CRYPTO_OP_STATUS_ERROR; + } + + if (unlikely(cop->sess_type == RTE_CRYPTO_OP_SESSIONLESS)) { + sym_session_clear(otx2_cryptodev_driver_id, + cop->sym->session); + memset(cop->sym->session, 0, + rte_cryptodev_sym_get_existing_header_session_size( + cop->sym->session)); + rte_mempool_put(qp->sess_mp, cop->sym->session); + cop->sym->session = NULL; + } + } + +} + +static inline uint64_t +otx2_handle_crypto_event(uint64_t get_work1) +{ + struct cpt_request_info *req; + struct rte_crypto_op *cop; + uintptr_t *rsp; + void *metabuf; + uint8_t cc; + + req = (struct cpt_request_info *)(get_work1); + cc = otx2_cpt_compcode_get(req); + + rsp = req->op; + metabuf = (void *)rsp[0]; + cop = (void *)rsp[1]; + + otx2_ca_deq_post_process(req->qp, cop, rsp, cc); + + rte_mempool_put(req->qp->meta_info.pool, metabuf); + + return (uint64_t)(cop); +} +#endif /* _OTX2_EVDEV_CRYPTO_ADPTR_DP_H_ */ diff --git a/drivers/event/octeontx2/otx2_worker.h b/drivers/event/octeontx2/otx2_worker.h index cde1288d9..3a3ec0fd4 100644 --- a/drivers/event/octeontx2/otx2_worker.h +++ b/drivers/event/octeontx2/otx2_worker.h @@ -10,6 +10,7 @@ #include #include "otx2_evdev.h" +#include "otx2_evdev_crypto_adptr_dp.h" #include "otx2_ethdev_sec_tx.h" /* SSO Operations */ @@ -66,16 +67,23 @@ otx2_ssogws_get_work(struct otx2_ssogws *ws, struct rte_event *ev, ws->cur_tt = event.sched_type; ws->cur_grp = event.queue_id; - if (event.sched_type != SSO_TT_EMPTY && - event.event_type == RTE_EVENT_TYPE_ETHDEV) { - otx2_wqe_to_mbuf(get_work1, mbuf, event.sub_event_type, - (uint32_t) event.get_work0, flags, lookup_mem); - /* Extracting tstamp, if PTP enabled*/ - tstamp_ptr = *(uint64_t *)(((struct nix_wqe_hdr_s *)get_work1) - + OTX2_SSO_WQE_SG_PTR); - otx2_nix_mbuf_to_tstamp((struct rte_mbuf *)mbuf, ws->tstamp, - flags, (uint64_t *)tstamp_ptr); - get_work1 = mbuf; + if (event.sched_type != SSO_TT_EMPTY) { + if ((flags & NIX_RX_OFFLOAD_SECURITY_F) && + (event.event_type == RTE_EVENT_TYPE_CRYPTODEV)) { + get_work1 = otx2_handle_crypto_event(get_work1); + } else if (event.event_type == RTE_EVENT_TYPE_ETHDEV) { + otx2_wqe_to_mbuf(get_work1, mbuf, event.sub_event_type, + (uint32_t) event.get_work0, flags, + lookup_mem); + /* Extracting tstamp, if PTP enabled*/ + tstamp_ptr = *(uint64_t *)(((struct nix_wqe_hdr_s *) + get_work1) + + OTX2_SSO_WQE_SG_PTR); + otx2_nix_mbuf_to_tstamp((struct rte_mbuf *)mbuf, + ws->tstamp, flags, + (uint64_t *)tstamp_ptr); + get_work1 = mbuf; + } } ev->event = event.get_work0; diff --git a/drivers/event/octeontx2/otx2_worker_dual.h b/drivers/event/octeontx2/otx2_worker_dual.h index 60aa14cca..6e6061821 100644 --- a/drivers/event/octeontx2/otx2_worker_dual.h +++ b/drivers/event/octeontx2/otx2_worker_dual.h @@ -10,6 +10,7 @@ #include #include "otx2_evdev.h" +#include "otx2_evdev_crypto_adptr_dp.h" /* SSO Operations */ static __rte_always_inline uint16_t @@ -63,25 +64,32 @@ otx2_ssogws_dual_get_work(struct otx2_ssogws_state *ws, ws->cur_tt = event.sched_type; ws->cur_grp = event.queue_id; - if (event.sched_type != SSO_TT_EMPTY && - event.event_type == RTE_EVENT_TYPE_ETHDEV) { - uint8_t port = event.sub_event_type; + if (event.sched_type != SSO_TT_EMPTY) { + if ((flags & NIX_RX_OFFLOAD_SECURITY_F) && + (event.event_type == RTE_EVENT_TYPE_CRYPTODEV)) { + get_work1 = otx2_handle_crypto_event(get_work1); + } else if (event.event_type == RTE_EVENT_TYPE_ETHDEV) { + uint8_t port = event.sub_event_type; - event.sub_event_type = 0; - otx2_wqe_to_mbuf(get_work1, mbuf, port, - event.flow_id, flags, lookup_mem); - /* Extracting tstamp, if PTP enabled. CGX will prepend the - * timestamp at starting of packet data and it can be derieved - * from WQE 9 dword which corresponds to SG iova. - * rte_pktmbuf_mtod_offset can be used for this purpose but it - * brings down the performance as it reads mbuf->buf_addr which - * is not part of cache in general fast path. - */ - tstamp_ptr = *(uint64_t *)(((struct nix_wqe_hdr_s *)get_work1) - + OTX2_SSO_WQE_SG_PTR); - otx2_nix_mbuf_to_tstamp((struct rte_mbuf *)mbuf, tstamp, flags, - (uint64_t *)tstamp_ptr); - get_work1 = mbuf; + event.sub_event_type = 0; + otx2_wqe_to_mbuf(get_work1, mbuf, port, + event.flow_id, flags, lookup_mem); + /* Extracting tstamp, if PTP enabled. CGX will prepend + * the timestamp at starting of packet data and it can + * be derieved from WQE 9 dword which corresponds to SG + * iova. + * rte_pktmbuf_mtod_offset can be used for this purpose + * but it brings down the performance as it reads + * mbuf->buf_addr which is not part of cache in general + * fast path. + */ + tstamp_ptr = *(uint64_t *)(((struct nix_wqe_hdr_s *) + get_work1) + + OTX2_SSO_WQE_SG_PTR); + otx2_nix_mbuf_to_tstamp((struct rte_mbuf *)mbuf, tstamp, + flags, (uint64_t *)tstamp_ptr); + get_work1 = mbuf; + } } ev->event = event.get_work0; -- 2.28.0