From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id D6288A034C;
Mon, 8 Aug 2022 10:06:25 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id 7041042B83;
Mon, 8 Aug 2022 10:06:15 +0200 (CEST)
Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com
[67.231.156.173])
by mails.dpdk.org (Postfix) with ESMTP id 7365C42B83
for <dev@dpdk.org>; Mon, 8 Aug 2022 10:06:14 +0200 (CEST)
Received: from pps.filterd (m0045851.ppops.net [127.0.0.1])
by mx0b-0016f401.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 277N0NGU005827
for <dev@dpdk.org>; Mon, 8 Aug 2022 01:06:13 -0700
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com;
h=from : to : cc :
subject : date : message-id : in-reply-to : references : mime-version :
content-transfer-encoding : content-type; s=pfpt0220;
bh=2sEp2bU0Gn60Tb6nO1uddLqqiCn1mbPS5Z3qmxYEUY0=;
b=Iqlrv2lhE6FqWkIJ82LWsLwqsrjeBxAtBBa3MBM2fsIVrWE1QMn7zn8aWEQ3uDfQZ4ZX
1+XN1Ia/fpTM5gOnS4SaW2Lp7tLqtYiawdR04aQFGPB0mnKaIlM72veRNAqj2XxjA7Qt
+PdtyEWCuj1JRtvGi3IAgQEdFa2hvBge1IE1yvUNW0R8DxQublAZ2wTjTwZ+P3h54S9Z
t1pwmoYqAUF3/XK9rUOM3XtP3KPm21VxjLEzyrK5B0pi2hEKj+WBGkQsYMHfWNwf1wJ7
pgORtxWu6zwsBOeHVT15fP/UqDSpyOllnQ9OOrPAqFiZMGPst9M978Fg8Bk7IV94fEyv yw==
Received: from dc5-exch01.marvell.com ([199.233.59.181])
by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3hsqtmmxdv-1
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT)
for <dev@dpdk.org>; Mon, 08 Aug 2022 01:06:13 -0700
Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com
(10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2;
Mon, 8 Aug 2022 01:06:11 -0700
Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com
(10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend
Transport; Mon, 8 Aug 2022 01:06:11 -0700
Received: from BG-LT92004.corp.innovium.com (unknown [10.28.160.62])
by maili.marvell.com (Postfix) with ESMTP id 9CAD63F7059;
Mon, 8 Aug 2022 01:06:09 -0700 (PDT)
From: Anoob Joseph <anoobj@marvell.com>
To: Akhil Goyal <gakhil@marvell.com>, Jerin Jacob <jerinj@marvell.com>
CC: Volodymyr Fialko <vfialko@marvell.com>, Archana Muniganti
<marchana@marvell.com>,
Tejasree Kondoj <ktejasree@marvell.com>, <dev@dpdk.org>
Subject: [PATCH 02/18] crypto/cnxk: add burst enqueue for event crypto
Date: Mon, 8 Aug 2022 13:35:50 +0530
Message-ID: <20220808080606.220-3-anoobj@marvell.com>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20220808080606.220-1-anoobj@marvell.com>
References: <20220808080606.220-1-anoobj@marvell.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
X-Proofpoint-GUID: AKGo1LjXxyvSANlSB574oaj5jXFASd7N
X-Proofpoint-ORIG-GUID: AKGo1LjXxyvSANlSB574oaj5jXFASd7N
X-Proofpoint-Virus-Version: vendor=baseguard
engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1
definitions=2022-08-08_05,2022-08-05_01,2022-06-22_01
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
From: Volodymyr Fialko <vfialko@marvell.com>
Added support for burst enqueue for cn10k event crypto adapter.
Instruction will be grouped based on the queue pair and sent in a burst.
Signed-off-by: Volodymyr Fialko <vfialko@marvell.com>
---
drivers/crypto/cnxk/cn10k_cryptodev_ops.c | 141 +++++++++++++++-------
drivers/crypto/cnxk/cn10k_cryptodev_ops.h | 7 +-
drivers/crypto/cnxk/meson.build | 2 +-
drivers/event/cnxk/cn10k_eventdev.c | 2 +-
drivers/event/cnxk/cn10k_worker.c | 10 --
drivers/event/cnxk/cn10k_worker.h | 2 -
6 files changed, 105 insertions(+), 59 deletions(-)
diff --git a/drivers/crypto/cnxk/cn10k_cryptodev_ops.c b/drivers/crypto/cnxk/cn10k_cryptodev_ops.c
index f761ba36e2..bfa6374005 100644
--- a/drivers/crypto/cnxk/cn10k_cryptodev_ops.c
+++ b/drivers/crypto/cnxk/cn10k_cryptodev_ops.c
@@ -9,11 +9,12 @@
#include "cn10k_cryptodev.h"
#include "cn10k_cryptodev_ops.h"
-#include "cn10k_ipsec_la_ops.h"
#include "cn10k_ipsec.h"
+#include "cn10k_ipsec_la_ops.h"
#include "cnxk_ae.h"
#include "cnxk_cryptodev.h"
#include "cnxk_cryptodev_ops.h"
+#include "cnxk_eventdev.h"
#include "cnxk_se.h"
#include "roc_api.h"
@@ -391,79 +392,135 @@ cn10k_ca_meta_info_extract(struct rte_crypto_op *op,
return 0;
}
-uint16_t
-cn10k_cpt_crypto_adapter_enqueue(uintptr_t base, struct rte_crypto_op *op)
+static inline uint16_t
+ca_lmtst_burst_submit(struct cn10k_sso_hws *ws, uint64_t w2[], struct cnxk_cpt_qp *qp,
+ struct rte_crypto_op *op[], uint16_t nb_ops)
{
+ struct cpt_inflight_req *infl_reqs[PKTS_PER_LOOP];
+ uint64_t lmt_base, lmt_arg, io_addr;
+ struct cpt_inst_s *inst, *inst_base;
struct cpt_inflight_req *infl_req;
- uint64_t lmt_base, lmt_arg, w2;
- struct cpt_inst_s *inst;
union cpt_fc_write_s fc;
- struct cnxk_cpt_qp *qp;
uint64_t *fc_addr;
uint16_t lmt_id;
- int ret;
+ int ret, i;
- ret = cn10k_ca_meta_info_extract(op, &qp, &w2);
- if (unlikely(ret)) {
- rte_errno = EINVAL;
- return 0;
- }
+ lmt_base = qp->lmtline.lmt_base;
+ io_addr = qp->lmtline.io_addr;
+ fc_addr = qp->lmtline.fc_addr;
+
+ const uint32_t fc_thresh = qp->lmtline.fc_thresh;
+
+ ROC_LMT_BASE_ID_GET(lmt_base, lmt_id);
+ inst_base = (struct cpt_inst_s *)lmt_base;
if (unlikely(!qp->ca.enabled)) {
rte_errno = EINVAL;
return 0;
}
- if (unlikely(rte_mempool_get(qp->ca.req_mp, (void **)&infl_req))) {
+ if (unlikely(rte_mempool_get_bulk(qp->ca.req_mp, (void **)infl_reqs, nb_ops))) {
rte_errno = ENOMEM;
return 0;
}
- infl_req->op_flags = 0;
-
- lmt_base = qp->lmtline.lmt_base;
- fc_addr = qp->lmtline.fc_addr;
- const uint32_t fc_thresh = qp->lmtline.fc_thresh;
+ for (i = 0; i < nb_ops; i++) {
+ inst = &inst_base[2 * i];
+ infl_req = infl_reqs[i];
+ infl_req->op_flags = 0;
- ROC_LMT_BASE_ID_GET(lmt_base, lmt_id);
- inst = (struct cpt_inst_s *)lmt_base;
+ ret = cn10k_cpt_fill_inst(qp, &op[i], inst, infl_req);
+ if (unlikely(ret != 1)) {
+ plt_dp_err("Could not process op: %p", op[i]);
+ if (i != 0)
+ goto submit;
+ else
+ goto put;
+ }
- ret = cn10k_cpt_fill_inst(qp, &op, inst, infl_req);
- if (unlikely(ret != 1)) {
- plt_dp_err("Could not process op: %p", op);
- rte_mempool_put(qp->ca.req_mp, infl_req);
- return 0;
+ infl_req->res.cn10k.compcode = CPT_COMP_NOT_DONE;
+ infl_req->qp = qp;
+ inst->w0.u64 = 0;
+ inst->res_addr = (uint64_t)&infl_req->res;
+ inst->w2.u64 = w2[i];
+ inst->w3.u64 = CNXK_CPT_INST_W3(1, infl_req);
}
- infl_req->cop = op;
- infl_req->res.cn10k.compcode = CPT_COMP_NOT_DONE;
- infl_req->qp = qp;
- inst->w0.u64 = 0;
- inst->res_addr = (uint64_t)&infl_req->res;
- inst->w2.u64 = w2;
- inst->w3.u64 = CNXK_CPT_INST_W3(1, infl_req);
-
fc.u64[0] = __atomic_load_n(fc_addr, __ATOMIC_RELAXED);
if (unlikely(fc.s.qsize > fc_thresh)) {
- rte_mempool_put(qp->ca.req_mp, infl_req);
rte_errno = EAGAIN;
- return 0;
+ i = 0;
+ goto put;
}
- if (inst->w2.s.tt == RTE_SCHED_TYPE_ORDERED)
- roc_sso_hws_head_wait(base);
+submit:
+ if (CNXK_TT_FROM_TAG(ws->gw_rdata) == SSO_TT_ORDERED)
+ roc_sso_hws_head_wait(ws->base);
- lmt_arg = ROC_CN10K_CPT_LMT_ARG | (uint64_t)lmt_id;
- roc_lmt_submit_steorl(lmt_arg, qp->lmtline.io_addr);
+ if (i > PKTS_PER_STEORL) {
+ lmt_arg = ROC_CN10K_CPT_LMT_ARG | (PKTS_PER_STEORL - 1) << 12 | (uint64_t)lmt_id;
+ roc_lmt_submit_steorl(lmt_arg, io_addr);
+ lmt_arg = ROC_CN10K_CPT_LMT_ARG | (i - PKTS_PER_STEORL - 1) << 12 |
+ (uint64_t)(lmt_id + PKTS_PER_STEORL);
+ roc_lmt_submit_steorl(lmt_arg, io_addr);
+ } else {
+ lmt_arg = ROC_CN10K_CPT_LMT_ARG | (i - 1) << 12 | (uint64_t)lmt_id;
+ roc_lmt_submit_steorl(lmt_arg, io_addr);
+ }
rte_io_wmb();
- return 1;
+put:
+ if (unlikely(i != nb_ops))
+ rte_mempool_put_bulk(qp->ca.req_mp, (void *)&infl_reqs[i], nb_ops - i);
+
+ return i;
+}
+
+uint16_t __rte_hot
+cn10k_cpt_crypto_adapter_enqueue(void *ws, struct rte_event ev[], uint16_t nb_events)
+{
+ struct rte_crypto_op *ops[PKTS_PER_LOOP], *op;
+ struct cnxk_cpt_qp *qp, *curr_qp = NULL;
+ uint64_t w2s[PKTS_PER_LOOP], w2;
+ uint16_t submitted, count = 0;
+ int ret, i, ops_len = 0;
+
+ for (i = 0; i < nb_events; i++) {
+ op = ev[i].event_ptr;
+ ret = cn10k_ca_meta_info_extract(op, &qp, &w2);
+ if (unlikely(ret)) {
+ rte_errno = EINVAL;
+ return count;
+ }
+
+ if (qp != curr_qp) {
+ if (ops_len) {
+ submitted = ca_lmtst_burst_submit(ws, w2s, curr_qp, ops, ops_len);
+ count += submitted;
+ if (unlikely(submitted != ops_len))
+ return count;
+ ops_len = 0;
+ }
+ curr_qp = qp;
+ }
+ w2s[ops_len] = w2;
+ ops[ops_len] = op;
+ if (++ops_len == PKTS_PER_LOOP) {
+ submitted = ca_lmtst_burst_submit(ws, w2s, curr_qp, ops, ops_len);
+ count += submitted;
+ if (unlikely(submitted != ops_len))
+ return count;
+ ops_len = 0;
+ }
+ }
+ if (ops_len)
+ count += ca_lmtst_burst_submit(ws, w2s, curr_qp, ops, ops_len);
+ return count;
}
static inline void
-cn10k_cpt_sec_post_process(struct rte_crypto_op *cop,
- struct cpt_cn10k_res_s *res)
+cn10k_cpt_sec_post_process(struct rte_crypto_op *cop, struct cpt_cn10k_res_s *res)
{
struct rte_mbuf *mbuf = cop->sym->m_src;
const uint16_t m_len = res->rlen;
diff --git a/drivers/crypto/cnxk/cn10k_cryptodev_ops.h b/drivers/crypto/cnxk/cn10k_cryptodev_ops.h
index 1ad4c16873..628d6a567c 100644
--- a/drivers/crypto/cnxk/cn10k_cryptodev_ops.h
+++ b/drivers/crypto/cnxk/cn10k_cryptodev_ops.h
@@ -5,16 +5,17 @@
#ifndef _CN10K_CRYPTODEV_OPS_H_
#define _CN10K_CRYPTODEV_OPS_H_
-#include <rte_cryptodev.h>
#include <cryptodev_pmd.h>
+#include <rte_cryptodev.h>
+#include <rte_eventdev.h>
extern struct rte_cryptodev_ops cn10k_cpt_ops;
void cn10k_cpt_set_enqdeq_fns(struct rte_cryptodev *dev);
__rte_internal
-uint16_t cn10k_cpt_crypto_adapter_enqueue(uintptr_t base,
- struct rte_crypto_op *op);
+uint16_t __rte_hot cn10k_cpt_crypto_adapter_enqueue(void *ws, struct rte_event ev[],
+ uint16_t nb_events);
__rte_internal
uintptr_t cn10k_cpt_crypto_adapter_dequeue(uintptr_t get_work1);
diff --git a/drivers/crypto/cnxk/meson.build b/drivers/crypto/cnxk/meson.build
index 23a1cc3aac..952554ac12 100644
--- a/drivers/crypto/cnxk/meson.build
+++ b/drivers/crypto/cnxk/meson.build
@@ -24,7 +24,7 @@ sources = files(
deps += ['bus_pci', 'common_cnxk', 'security', 'eventdev']
-includes += include_directories('../../../lib/net')
+includes += include_directories('../../../lib/net', '../../event/cnxk')
if get_option('buildtype').contains('debug')
cflags += [ '-DLA_IPSEC_DEBUG' ]
diff --git a/drivers/event/cnxk/cn10k_eventdev.c b/drivers/event/cnxk/cn10k_eventdev.c
index 5a0cab40a9..25c85902d6 100644
--- a/drivers/event/cnxk/cn10k_eventdev.c
+++ b/drivers/event/cnxk/cn10k_eventdev.c
@@ -454,7 +454,7 @@ cn10k_sso_fp_fns_set(struct rte_eventdev *event_dev)
sso_hws_deq_tmo_ca_burst);
}
}
- event_dev->ca_enqueue = cn10k_sso_hws_ca_enq;
+ event_dev->ca_enqueue = cn10k_cpt_crypto_adapter_enqueue;
if (dev->tx_offloads & NIX_TX_MULTI_SEG_F)
CN10K_SET_EVDEV_ENQ_OP(dev, event_dev->txa_enqueue,
diff --git a/drivers/event/cnxk/cn10k_worker.c b/drivers/event/cnxk/cn10k_worker.c
index f953e19dd0..4581c41233 100644
--- a/drivers/event/cnxk/cn10k_worker.c
+++ b/drivers/event/cnxk/cn10k_worker.c
@@ -64,13 +64,3 @@ cn10k_sso_hws_enq_fwd_burst(void *port, const struct rte_event ev[],
return 1;
}
-
-uint16_t __rte_hot
-cn10k_sso_hws_ca_enq(void *port, struct rte_event ev[], uint16_t nb_events)
-{
- struct cn10k_sso_hws *ws = port;
-
- RTE_SET_USED(nb_events);
-
- return cn10k_cpt_crypto_adapter_enqueue(ws->base, ev->event_ptr);
-}
diff --git a/drivers/event/cnxk/cn10k_worker.h b/drivers/event/cnxk/cn10k_worker.h
index 0915f404e0..65bb08c0a1 100644
--- a/drivers/event/cnxk/cn10k_worker.h
+++ b/drivers/event/cnxk/cn10k_worker.h
@@ -353,8 +353,6 @@ uint16_t __rte_hot cn10k_sso_hws_enq_new_burst(void *port,
uint16_t __rte_hot cn10k_sso_hws_enq_fwd_burst(void *port,
const struct rte_event ev[],
uint16_t nb_events);
-uint16_t __rte_hot cn10k_sso_hws_ca_enq(void *port, struct rte_event ev[],
- uint16_t nb_events);
#define R(name, flags) \
uint16_t __rte_hot cn10k_sso_hws_deq_##name( \
--
2.25.1