From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EF326454AB; Thu, 20 Jun 2024 16:59:48 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DD2E842EA8; Thu, 20 Jun 2024 16:59:22 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 5C07542E64 for ; Thu, 20 Jun 2024 16:59:21 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 45KASfuB027294 for ; Thu, 20 Jun 2024 07:59:20 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pfpt0220; bh=T D/g/LdMrfS3RUJXXdRlMLl7zUnLtm1ylDlZkjiPPfg=; b=jeAVXsPldYyOjFyr6 Zua1benzM7wfNl8A5pj8LFNswwtn9wBxPHMNSOHj2eCMol5IunlDv9SbVB09aI1z z311pYC4LEiWBz6phqvpMuocIkFMA+qrZEUE58xhSlfzRmMFVRttSKb1DsIfj1RC lL2vg53ErS9IYtZJGhF4BPEkRGtB1K+CJlPo6Lq7J+j6oYv5ltox4tngbXV+NyaN jOo/OHm+5uC37PkkvVBVUdP3l9Om+7HX83JHiWI6yqLeTeDa7/C/aQvAwOYajStn wfz5kJ+sQF2111X8HwFCGmzmCBZGJTwespB3BeEPMfLuwbiPgmHF5Xbw6beNPXvA be9Bw== Received: from dc5-exch05.marvell.com ([199.233.59.128]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3yvjq0h32v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Jun 2024 07:59:20 -0700 (PDT) Received: from DC5-EXCH05.marvell.com (10.69.176.209) by DC5-EXCH05.marvell.com (10.69.176.209) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Thu, 20 Jun 2024 07:59:19 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH05.marvell.com (10.69.176.209) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Thu, 20 Jun 2024 07:59:19 -0700 Received: from localhost.localdomain (unknown [10.28.36.177]) by maili.marvell.com (Postfix) with ESMTP id 802783F7081; Thu, 20 Jun 2024 07:59:15 -0700 (PDT) From: Aakash Sasidharan To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao , Harman Kalra , Ankur Dwivedi , Anoob Joseph , Tejasree Kondoj CC: , , , , Subject: [PATCH 08/12] crypto/cnxk: add dual submission in Rx inject Date: Thu, 20 Jun 2024 20:28:44 +0530 Message-ID: <20240620145848.3461844-9-asasidharan@marvell.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240620145848.3461844-1-asasidharan@marvell.com> References: <20240620145848.3461844-1-asasidharan@marvell.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Proofpoint-ORIG-GUID: 0r3qIKUSXxcMwjVLgofT65iULGwTkLbr X-Proofpoint-GUID: 0r3qIKUSXxcMwjVLgofT65iULGwTkLbr X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-06-20_07,2024-06-20_04,2024-05-17_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Anoob Joseph Add dual submission to CPT in Rx inject path. Signed-off-by: Anoob Joseph Signed-off-by: Vidya Sagar Velumuri --- drivers/common/cnxk/roc_cpt.h | 43 +++++++++----- drivers/crypto/cnxk/cn10k_cryptodev_ops.c | 70 +++++++++++++++++------ drivers/crypto/cnxk/cnxk_cryptodev_ops.c | 9 +++ 3 files changed, 90 insertions(+), 32 deletions(-) diff --git a/drivers/common/cnxk/roc_cpt.h b/drivers/common/cnxk/roc_cpt.h index 3721fa08c0..8ef9062ae0 100644 --- a/drivers/common/cnxk/roc_cpt.h +++ b/drivers/common/cnxk/roc_cpt.h @@ -30,23 +30,36 @@ /* Vector of sizes in the burst of 16 CPT inst except first in 63:19 of * APT_LMT_ARG_S */ -#define ROC_CN10K_CPT_LMT_ARG \ - (ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 0) | \ - ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 1) | \ - ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 2) | \ - ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 3) | \ - ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 4) | \ - ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 5) | \ - ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 6) | \ - ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 7) | \ - ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 8) | \ - ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 9) | \ - ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 10) | \ - ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 11) | \ - ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 12) | \ - ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 13) | \ +#define ROC_CN10K_CPT_LMT_ARG \ + (ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 0) | ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 1) | \ + ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 2) | ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 3) | \ + ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 4) | ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 5) | \ + ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 6) | ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 7) | \ + ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 8) | ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 9) | \ + ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 10) | ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 11) | \ + ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 12) | ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 13) | \ ROC_CN10K_CPT_INST_DW_M1 << (19 + 3 * 14)) +/* Vector of sizes in the burst of 2 * 16 CPT inst except first in 63:19 of + * APT_LMT_ARG_S + */ +#define ROC_CN10K_DUAL_CPT_LMT_ARG \ + (ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 0) | \ + ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 1) | \ + ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 2) | \ + ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 3) | \ + ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 4) | \ + ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 5) | \ + ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 6) | \ + ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 7) | \ + ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 8) | \ + ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 9) | \ + ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 10) | \ + ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 11) | \ + ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 12) | \ + ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 13) | \ + ROC_CN10K_TWO_CPT_INST_DW_M1 << (19 + 3 * 14)) + /* CPT helper macros */ #define ROC_CPT_AH_HDR_LEN 12 #define ROC_CPT_AES_GCM_IV_LEN 8 diff --git a/drivers/crypto/cnxk/cn10k_cryptodev_ops.c b/drivers/crypto/cnxk/cn10k_cryptodev_ops.c index 1108a8a1da..3fd002d549 100644 --- a/drivers/crypto/cnxk/cn10k_cryptodev_ops.c +++ b/drivers/crypto/cnxk/cn10k_cryptodev_ops.c @@ -55,6 +55,54 @@ struct vec_request { uint64_t w2; }; +static __rte_always_inline void __rte_hot +cn10k_cpt_lmtst_dual_submit(uint64_t *io_addr, const uint16_t lmt_id, int *i) +{ + uint64_t lmt_arg; + + /* Check if the total number of instructions is odd or even. */ + const int flag_odd = *i & 0x1; + + /* Reduce i by 1 when odd number of instructions.*/ + *i -= flag_odd; + + if (*i > 2 * CN10K_PKTS_PER_STEORL) { + lmt_arg = ROC_CN10K_DUAL_CPT_LMT_ARG | (CN10K_PKTS_PER_STEORL - 1) << 12 | + (uint64_t)lmt_id; + roc_lmt_submit_steorl(lmt_arg, *io_addr); + lmt_arg = ROC_CN10K_DUAL_CPT_LMT_ARG | (*i / 2 - CN10K_PKTS_PER_STEORL - 1) << 12 | + (uint64_t)(lmt_id + CN10K_PKTS_PER_STEORL); + roc_lmt_submit_steorl(lmt_arg, *io_addr); + if (flag_odd) { + *io_addr = (*io_addr & ~(uint64_t)(0x7 << 4)) | + (ROC_CN10K_CPT_INST_DW_M1 << 4); + lmt_arg = (uint64_t)(lmt_id + *i / 2); + roc_lmt_submit_steorl(lmt_arg, *io_addr); + *io_addr = (*io_addr & ~(uint64_t)(0x7 << 4)) | + (ROC_CN10K_TWO_CPT_INST_DW_M1 << 4); + *i += 1; + } + } else { + if (*i != 0) { + lmt_arg = + ROC_CN10K_DUAL_CPT_LMT_ARG | (*i / 2 - 1) << 12 | (uint64_t)lmt_id; + roc_lmt_submit_steorl(lmt_arg, *io_addr); + } + + if (flag_odd) { + *io_addr = (*io_addr & ~(uint64_t)(0x7 << 4)) | + (ROC_CN10K_CPT_INST_DW_M1 << 4); + lmt_arg = (uint64_t)(lmt_id + *i / 2); + roc_lmt_submit_steorl(lmt_arg, *io_addr); + *io_addr = (*io_addr & ~(uint64_t)(0x7 << 4)) | + (ROC_CN10K_TWO_CPT_INST_DW_M1 << 4); + *i += 1; + } + } + + rte_io_wmb(); +} + static inline struct cnxk_se_sess * cn10k_cpt_sym_temp_sess_create(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op) { @@ -1396,7 +1444,7 @@ uint16_t __rte_hot cn10k_cryptodev_sec_inb_rx_inject(void *dev, struct rte_mbuf **pkts, struct rte_security_session **sess, uint16_t nb_pkts) { - uint64_t lmt_base, lmt_arg, io_addr, u64_0, u64_1, l2_len, pf_func; + uint64_t lmt_base, io_addr, u64_0, u64_1, l2_len, pf_func; uint64x2_t inst_01, inst_23, inst_45, inst_67; struct cn10k_sec_session *sec_sess; struct rte_cryptodev *cdev = dev; @@ -1431,7 +1479,7 @@ cn10k_cryptodev_sec_inb_rx_inject(void *dev, struct rte_mbuf **pkts, if (unlikely(fc.s.qsize > fc_thresh)) goto exit; - for (; i < RTE_MIN(CN10K_PKTS_PER_LOOP, nb_pkts); i++) { + for (; i < RTE_MIN(2 * CN10K_PKTS_PER_LOOP, nb_pkts); i++) { m = pkts[i]; sec_sess = (struct cn10k_sec_session *)sess[i]; @@ -1484,24 +1532,12 @@ cn10k_cryptodev_sec_inb_rx_inject(void *dev, struct rte_mbuf **pkts, inst_67 = vsetq_lane_u64(u64_1, inst_67, 1); vst1q_u64(&inst->w6.u64, inst_67); - inst += 2; - } - - if (i > CN10K_PKTS_PER_STEORL) { - lmt_arg = ROC_CN10K_CPT_LMT_ARG | (CN10K_PKTS_PER_STEORL - 1) << 12 | - (uint64_t)lmt_id; - roc_lmt_submit_steorl(lmt_arg, io_addr); - lmt_arg = ROC_CN10K_CPT_LMT_ARG | (i - CN10K_PKTS_PER_STEORL - 1) << 12 | - (uint64_t)(lmt_id + CN10K_PKTS_PER_STEORL); - roc_lmt_submit_steorl(lmt_arg, io_addr); - } else { - lmt_arg = ROC_CN10K_CPT_LMT_ARG | (i - 1) << 12 | (uint64_t)lmt_id; - roc_lmt_submit_steorl(lmt_arg, io_addr); + inst++; } - rte_io_wmb(); + cn10k_cpt_lmtst_dual_submit(&io_addr, lmt_id, &i); - if (nb_pkts - i > 0 && i == CN10K_PKTS_PER_LOOP) { + if (nb_pkts - i > 0 && i == 2 * CN10K_PKTS_PER_LOOP) { nb_pkts -= i; pkts += i; count += i; diff --git a/drivers/crypto/cnxk/cnxk_cryptodev_ops.c b/drivers/crypto/cnxk/cnxk_cryptodev_ops.c index 51369309c5..6acaa4413b 100644 --- a/drivers/crypto/cnxk/cnxk_cryptodev_ops.c +++ b/drivers/crypto/cnxk/cnxk_cryptodev_ops.c @@ -431,6 +431,7 @@ cnxk_cpt_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id, struct rte_pci_device *pci_dev; struct cnxk_cpt_qp *qp; uint32_t nb_desc; + uint64_t io_addr; int ret; if (dev->data->queue_pairs[qp_id] != NULL) @@ -485,6 +486,14 @@ cnxk_cpt_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id, vf->rx_inj_sso_pf_func = roc_idev_nix_inl_dev_pffunc_get(); + /* Update IO addr to enable dual submission */ + io_addr = vf->rx_inj_lmtline.io_addr; + io_addr = (io_addr & ~(uint64_t)(0x7 << 4)) | ROC_CN10K_TWO_CPT_INST_DW_M1 << 4; + vf->rx_inj_lmtline.io_addr = io_addr; + + /* Update FC threshold to reflect dual submission */ + vf->rx_inj_lmtline.fc_thresh -= 32; + /* Block the queue for other submissions */ qp->pend_q.pq_mask = 0; } -- 2.25.1