From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0DAFDA00C2; Thu, 10 Feb 2022 05:32:42 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A5CAD41172; Thu, 10 Feb 2022 05:32:13 +0100 (CET) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2044.outbound.protection.outlook.com [40.107.21.44]) by mails.dpdk.org (Postfix) with ESMTP id 8A6AB41181 for ; Thu, 10 Feb 2022 05:32:12 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=g0MQonTEh/jlkQA7jgShSUnSnyVNjeef0y236cV36DJH36Ihcz06CQZfB5E2hemq5kVajBNjQiQss1X7ukbKmQiOUC52GktSKwl0321mj3/4Po8u1LkP60FLd2EcGAfoEMJekX5TNDjKII+ff6kF8Rie1h+Qb5h5DHhSjha/BlFDCNXAumvhS8FCXT8DRZqeCeiNvFV8bj3rKgeYxL3LrDRiXD0utvzQZy42GcdpBsTkARVap8iHUpvz4r9UgtNtkG6kyVlJ8omI0yv8N1dpbWlmeYGC+JqqBrOO2MULk+p1a46s3fLD1IK6uAbYa7xo8hPzwua456qi2WzX7NCpPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=oGoQ0PQlVehe7bCYn5WHaQfCu4T/yq5ByrV57EGJlbE=; b=FnsEEAnd8MP+03GLwxJcJnWY0CYAOW3TSaclVwv1j/qSKYMQ7XfEZHzSZDY6/BvF1W6/J2L00khcIC7mZ6XHnu68xdfd0/GZAF/U7vg4vbcZ5TybQxJCZD6egBn9BGaBK+tZY5gK/3wHltB4uQ4y+60UO5u+981vEDz+oqUEi1X/h8RGHIrz4LJyl1i0qYOJsdXI+yRX3JEQD/+5yb7rz8h/xroEN3/042moYBuQn0sSN5Mc3ZA0opd3JEY3kBvHTzKHC0qKsJqj9uc609uH5ixrtch/DTVSLjEgwgzKNUfpeeTparC90QniW3QpxD1RaO+q0PxZmUAZXQtPTa2k9w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oGoQ0PQlVehe7bCYn5WHaQfCu4T/yq5ByrV57EGJlbE=; b=cm3g1n5hVuN2BNp9XY/m6ljZ1Gvw31xZY/8j/inKF3d2frcISh1jh/nG4kMTGx+mvGuuq4VfTzY4TiGTAGxUF5k0X71/L5B+pokAi5AX9ddQo0EhJunwEbxZ3OWtUPooxkd5Tvf5m+of68ThX/vVr6yAHU8RIrr9KGUBTyH0sKg= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nxp.com; Received: from AS8PR04MB8198.eurprd04.prod.outlook.com (2603:10a6:20b:3b0::14) by PAXPR04MB8526.eurprd04.prod.outlook.com (2603:10a6:102:211::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4951.19; Thu, 10 Feb 2022 04:32:10 +0000 Received: from AS8PR04MB8198.eurprd04.prod.outlook.com ([fe80::4146:965b:5645:9269]) by AS8PR04MB8198.eurprd04.prod.outlook.com ([fe80::4146:965b:5645:9269%4]) with mapi id 15.20.4975.011; Thu, 10 Feb 2022 04:32:10 +0000 From: Gagandeep Singh To: gakhil@marvell.com, dev@dpdk.org Cc: Nipun Gupta , Gagandeep Singh Subject: [PATCH v3 6/7] crypto/dpaa2_sec: ordered queue support Date: Thu, 10 Feb 2022 10:01:35 +0530 Message-Id: <20220210043136.304845-7-g.singh@nxp.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220210043136.304845-1-g.singh@nxp.com> References: <20211228091058.3620177-2-g.singh@nxp.com> <20220210043136.304845-1-g.singh@nxp.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: SG3P274CA0003.SGPP274.PROD.OUTLOOK.COM (2603:1096:4:be::15) To AS8PR04MB8198.eurprd04.prod.outlook.com (2603:10a6:20b:3b0::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ddfc7e98-7ac8-417d-5c3f-08d9ec4e4d98 X-MS-TrafficTypeDiagnostic: PAXPR04MB8526:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7219; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: TZV3VCB8mYSuWR96SSzoBEhKEAkIhs4U/IiJLB8ElWkc/rmmotDhbPLObXs5DYIFK1ozPgWOduPmpAr0eFXClXK1hmODPQsA1Gk/lovKcxb2Nq9bWWVYpfO4WD1O5k8vivr2tTPNf7QgwZhD37Fd/nCrRZCcK5fvguPoR/KqiVRfNVCsdi14U72w60Rh51wVMMAmIlsudhYESxGwfP4VqlY0A8esGFaLiG4CiXVPEWQzxkWIygwKqxGghhZ2eYtwCFZP8vw3arXb7PcNBmItV5w5tO9jHb7RVfF+QVGy2XKMppd7a3Z8MHtF3pw/L7RvCS1OWsTQzyiAsDmUDZZSH6ptXc4wPMYCud7s8LccTfG6JWEmEttsBqlej/hU4QMPpjiV5MjHtgA7ZCYIWADu88EsLQZLzRNnEEJnraEZXPN4DkQOu3UoyBVpkmWxif6saosATNFgDTWpTjnDcOepsTXB8uap447Sph/HYC8H6+zTswWDx8I230qJB6B057WiGw+wlhGKPlOHUgIWvTYZF0iKRxPCYDuPgY0QQLFHimZtJDWKC0FATosRsr/iXJGq9T8xkXz4LaN2GN5mh5Op6+li45RRgxCkznfYwtXiJCGuCahHiz6/0YZ9wnGcarpd7hrNc4Gb3TKrA6LmzHGz5iDU9dh1zZmNCHkgm4t/t7MJfWDRoI1Fxxbt6+Qt5NZKDF6suHbIjwBQ0ZYxx/PKjw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AS8PR04MB8198.eurprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(4636009)(366004)(66946007)(66556008)(30864003)(1076003)(2616005)(38350700002)(38100700002)(8936002)(66476007)(83380400001)(4326008)(316002)(5660300002)(54906003)(2906002)(186003)(86362001)(508600001)(52116002)(55236004)(26005)(8676002)(6486002)(6666004)(6506007)(36756003)(6512007); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?vKmXYT0hxO+4P0X+kaFkAdzNJ5dDqoOeBPobhPYoTMCCYd2XAh/aYavM90EA?= =?us-ascii?Q?hD+w4d0b6DhiB/CQiQ4u3q31TgtC+tENSvegKxwmr3T3Y3+6OSlRzxa8tXLz?= =?us-ascii?Q?vtnJ5UmP5Dw4ABlCc7dgAhOLsR3yzYo1N5KnSp13/CPVofrvFRoL6ViNhbhw?= =?us-ascii?Q?Xn8b4gycc6A64X/FtuHXcWKQOO0n2t2qgELEWjSHApImojxy1Jpjc6lNXWlT?= =?us-ascii?Q?yDRNqUdQeSDJ4rqJQH++P2yOCz/YddYacnhHh0g0xGeArvocKNqlbkCb8HqM?= =?us-ascii?Q?94bNSGA6eAcmkN9u/f4hS1/fXYWGuBrzFFAn4Vg2R6+UvQPDiDq3qFo6PKEs?= =?us-ascii?Q?2m/KG8l0qD9QB5W8PmfwXsjOC5fUNHIFM9ecocuoEqiIk/YY/ss5vVzBFBMR?= =?us-ascii?Q?NNd0vBMPaFt9Ymc8PH+Bet4x71afalLP/oug70OKM1fLjCBb0dEAENvWOH8Y?= =?us-ascii?Q?63q7s9xNBKSsDokPyA0KeYEAQyqxqvH1ieBiUXKv6goPGL6I++V4+uAFB4zL?= =?us-ascii?Q?rvIHwgzteQ18QlviqVEzqau5HyX2aRGcq+b96Z9IHb7Ud6uwlmt5RfNt6DZX?= =?us-ascii?Q?w7nit17nHNBOCEjsBQ3G4ZA0vq2u0MHbmOGZWDgSgqfRQKJRXOhgeGrk4CMk?= =?us-ascii?Q?pJ6MwHsbH1WaCEqRjVsZ1uYWb+opw1ivwHUIJQSICAfoJUzNLTEHOHGIfJSP?= =?us-ascii?Q?AgBuIzO6nZrBbGIlXFRo7xTuDJEKQi27ciWKl/gSldBdpZvRCXYducYJhDsB?= =?us-ascii?Q?XEi8nW3RJZx4uirq6IPnnWxRYvPeAR9/uuJWioZ5sSPvgKV5LiOE0Yip8kkr?= =?us-ascii?Q?ZtwTEK5xpfvdXp4SUi4151rZPt/ncDeLovl4oGzpOCD3KHLC1BHsnxhFelCN?= =?us-ascii?Q?QnMZC9DbybMVwNEaIuYzwFfZnf16JNhBIoiLI81bFUtnssyZ516Xll0rQtXp?= =?us-ascii?Q?B5aWH6MtY3Zr+eRIJ7CYuVaAkIh3tO/ghwKnzj2N43b6hGeUGPyV8QSVvK+O?= =?us-ascii?Q?aB/Z5Foo9gi/H3gHgnXWMtR6jknA4AjayxrvQFrBP+ajg07kJQMhCIWj4OQR?= =?us-ascii?Q?u2foQ1VkhjUT6QMOZ5E36GS1TWVzbnY8dh1ijMg9jZ0I3NgeIGNhNRoJPcNh?= =?us-ascii?Q?UZ+OJc/DznkEsbrgVWUygfswvhliTNpoNU9nZJkYzWm+iPIxKO5yrP7+iCzq?= =?us-ascii?Q?K6z5o4l2z5uLzPxmBj4jqysnjnMOh2lE1osj2APOmFev7DaXsD0iNMM4EWIS?= =?us-ascii?Q?YHaP6z87xEM0/Zvu0Yy43P51u3nHAST+vy9tXWrYa/rVTQ6bqOuee/S58g3i?= =?us-ascii?Q?zYVhcwdL6xXidoXQl+hm4XIwbA1k6fhWG3nQK5ezaYFPb0n00jfu3bvpKhz6?= =?us-ascii?Q?laUsXKx8gpX4gGRxHrTEHSaqyUR9Xfl16AjQwFOSAYSnFcJgMWB2JPeQTSa3?= =?us-ascii?Q?SfcLrpNNzoSfzgq3CkKrpEDyREVM8DeLPu6Ai2QH8IxExPGyiC44QXxc2HXf?= =?us-ascii?Q?0V9aCVerHcAkvYQhhvncDI65WfGGzAJLXrAk6durm4piqjh6AX4XYTUpY0jL?= =?us-ascii?Q?CTaFn9+GjFoggW1Xkps=3D?= X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: ddfc7e98-7ac8-417d-5c3f-08d9ec4e4d98 X-MS-Exchange-CrossTenant-AuthSource: AS8PR04MB8198.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Feb 2022 04:32:10.7132 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Zs8qiluW6TuO/GPv9wDzIBPmYb0f9TCjda8eA68ArPhuDlkSzhqM3EUaa+HuQ37d X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAXPR04MB8526 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Nipun Gupta This patch supports ordered queue for DPAA2 platform. Signed-off-by: Nipun Gupta Signed-off-by: Gagandeep Singh --- doc/guides/cryptodevs/dpaa2_sec.rst | 7 + drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c | 285 ++++++++++++++++++-- drivers/crypto/dpaa2_sec/dpaa2_sec_priv.h | 8 +- drivers/crypto/dpaa2_sec/mc/fsl_dpseci.h | 14 +- 4 files changed, 290 insertions(+), 24 deletions(-) diff --git a/doc/guides/cryptodevs/dpaa2_sec.rst b/doc/guides/cryptodevs/dpaa2_sec.rst index 875d918068..1a590309a0 100644 --- a/doc/guides/cryptodevs/dpaa2_sec.rst +++ b/doc/guides/cryptodevs/dpaa2_sec.rst @@ -185,3 +185,10 @@ on error, mode 1 means dump HW error code and mode 2 means dump HW error code along with other useful debugging information like session, queue, descriptor data. e.g. ``fslmc:dpseci.1,drv_dump_mode=1`` + +Enable strict ordering +---------------------- + +Use dev arg option ``drv_strict_order=1`` to enable strict ordering. +By default, loose ordering is set for ordered schedule type event. +e.g. ``fslmc:dpseci.1,drv_strict_order=1`` diff --git a/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c b/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c index cb8aaf6446..7a0596e44e 100644 --- a/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c +++ b/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c @@ -52,6 +52,7 @@ #define NO_PREFETCH 0 #define DRIVER_DUMP_MODE "drv_dump_mode" +#define DRIVER_STRICT_ORDER "drv_strict_order" /* DPAA2_SEC_DP_DUMP levels */ enum dpaa2_sec_dump_levels { @@ -1477,14 +1478,14 @@ dpaa2_sec_enqueue_burst(void *qp, struct rte_crypto_op **ops, for (loop = 0; loop < frames_to_send; loop++) { if (*dpaa2_seqn((*ops)->sym->m_src)) { - uint8_t dqrr_index = - *dpaa2_seqn((*ops)->sym->m_src) - 1; - - flags[loop] = QBMAN_ENQUEUE_FLAG_DCA | dqrr_index; - DPAA2_PER_LCORE_DQRR_SIZE--; - DPAA2_PER_LCORE_DQRR_HELD &= ~(1 << dqrr_index); - *dpaa2_seqn((*ops)->sym->m_src) = - DPAA2_INVALID_MBUF_SEQN; + if (*dpaa2_seqn((*ops)->sym->m_src) & QBMAN_ENQUEUE_FLAG_DCA) { + DPAA2_PER_LCORE_DQRR_SIZE--; + DPAA2_PER_LCORE_DQRR_HELD &= ~(1 << + *dpaa2_seqn((*ops)->sym->m_src) & + QBMAN_EQCR_DCA_IDXMASK); + } + flags[loop] = *dpaa2_seqn((*ops)->sym->m_src); + *dpaa2_seqn((*ops)->sym->m_src) = DPAA2_INVALID_MBUF_SEQN; } /*Clear the unused FD fields before sending*/ @@ -1709,6 +1710,168 @@ dpaa2_sec_dump(struct rte_crypto_op *op) } +static void +dpaa2_sec_free_eqresp_buf(uint16_t eqresp_ci) +{ + struct dpaa2_dpio_dev *dpio_dev = DPAA2_PER_LCORE_DPIO; + struct rte_crypto_op *op; + struct qbman_fd *fd; + + fd = qbman_result_eqresp_fd(&dpio_dev->eqresp[eqresp_ci]); + op = sec_fd_to_mbuf(fd); + /* Instead of freeing, enqueue it to the sec tx queue (sec->core) + * after setting an error in FD. But this will have performance impact. + */ + rte_pktmbuf_free(op->sym->m_src); +} + +static void +dpaa2_sec_set_enqueue_descriptor(struct dpaa2_queue *dpaa2_q, + struct rte_mbuf *m, + struct qbman_eq_desc *eqdesc) +{ + struct dpaa2_dpio_dev *dpio_dev = DPAA2_PER_LCORE_DPIO; + struct eqresp_metadata *eqresp_meta; + struct dpaa2_sec_dev_private *priv = dpaa2_q->crypto_data->dev_private; + uint16_t orpid, seqnum; + uint8_t dq_idx; + + if (*dpaa2_seqn(m) & DPAA2_ENQUEUE_FLAG_ORP) { + orpid = (*dpaa2_seqn(m) & DPAA2_EQCR_OPRID_MASK) >> + DPAA2_EQCR_OPRID_SHIFT; + seqnum = (*dpaa2_seqn(m) & DPAA2_EQCR_SEQNUM_MASK) >> + DPAA2_EQCR_SEQNUM_SHIFT; + + + if (!priv->en_loose_ordered) { + qbman_eq_desc_set_orp(eqdesc, 1, orpid, seqnum, 0); + qbman_eq_desc_set_response(eqdesc, (uint64_t) + DPAA2_VADDR_TO_IOVA(&dpio_dev->eqresp[ + dpio_dev->eqresp_pi]), 1); + qbman_eq_desc_set_token(eqdesc, 1); + + eqresp_meta = &dpio_dev->eqresp_meta[dpio_dev->eqresp_pi]; + eqresp_meta->dpaa2_q = dpaa2_q; + eqresp_meta->mp = m->pool; + + dpio_dev->eqresp_pi + 1 < MAX_EQ_RESP_ENTRIES ? + dpio_dev->eqresp_pi++ : (dpio_dev->eqresp_pi = 0); + } else { + qbman_eq_desc_set_orp(eqdesc, 0, orpid, seqnum, 0); + } + } else { + dq_idx = *dpaa2_seqn(m) - 1; + qbman_eq_desc_set_dca(eqdesc, 1, dq_idx, 0); + DPAA2_PER_LCORE_DQRR_SIZE--; + DPAA2_PER_LCORE_DQRR_HELD &= ~(1 << dq_idx); + } + *dpaa2_seqn(m) = DPAA2_INVALID_MBUF_SEQN; +} + + +static uint16_t +dpaa2_sec_enqueue_burst_ordered(void *qp, struct rte_crypto_op **ops, + uint16_t nb_ops) +{ + /* Function to transmit the frames to given device and VQ*/ + uint32_t loop; + int32_t ret; + struct qbman_fd fd_arr[MAX_TX_RING_SLOTS]; + uint32_t frames_to_send, num_free_eq_desc, retry_count; + struct qbman_eq_desc eqdesc[MAX_TX_RING_SLOTS]; + struct dpaa2_sec_qp *dpaa2_qp = (struct dpaa2_sec_qp *)qp; + struct qbman_swp *swp; + uint16_t num_tx = 0; + uint16_t bpid; + struct rte_mempool *mb_pool; + struct dpaa2_sec_dev_private *priv = + dpaa2_qp->tx_vq.crypto_data->dev_private; + + if (unlikely(nb_ops == 0)) + return 0; + + if (ops[0]->sess_type == RTE_CRYPTO_OP_SESSIONLESS) { + DPAA2_SEC_ERR("sessionless crypto op not supported"); + return 0; + } + + if (!DPAA2_PER_LCORE_DPIO) { + ret = dpaa2_affine_qbman_swp(); + if (ret) { + DPAA2_SEC_ERR("Failure in affining portal"); + return 0; + } + } + swp = DPAA2_PER_LCORE_PORTAL; + + while (nb_ops) { + frames_to_send = (nb_ops > dpaa2_eqcr_size) ? + dpaa2_eqcr_size : nb_ops; + + if (!priv->en_loose_ordered) { + if (*dpaa2_seqn((*ops)->sym->m_src)) { + num_free_eq_desc = dpaa2_free_eq_descriptors(); + if (num_free_eq_desc < frames_to_send) + frames_to_send = num_free_eq_desc; + } + } + + for (loop = 0; loop < frames_to_send; loop++) { + /*Prepare enqueue descriptor*/ + qbman_eq_desc_clear(&eqdesc[loop]); + qbman_eq_desc_set_fq(&eqdesc[loop], dpaa2_qp->tx_vq.fqid); + + if (*dpaa2_seqn((*ops)->sym->m_src)) + dpaa2_sec_set_enqueue_descriptor( + &dpaa2_qp->tx_vq, + (*ops)->sym->m_src, + &eqdesc[loop]); + else + qbman_eq_desc_set_no_orp(&eqdesc[loop], + DPAA2_EQ_RESP_ERR_FQ); + + /*Clear the unused FD fields before sending*/ + memset(&fd_arr[loop], 0, sizeof(struct qbman_fd)); + mb_pool = (*ops)->sym->m_src->pool; + bpid = mempool_to_bpid(mb_pool); + ret = build_sec_fd(*ops, &fd_arr[loop], bpid); + if (ret) { + DPAA2_SEC_ERR("error: Improper packet contents" + " for crypto operation"); + goto skip_tx; + } + ops++; + } + + loop = 0; + retry_count = 0; + while (loop < frames_to_send) { + ret = qbman_swp_enqueue_multiple_desc(swp, + &eqdesc[loop], &fd_arr[loop], + frames_to_send - loop); + if (unlikely(ret < 0)) { + retry_count++; + if (retry_count > DPAA2_MAX_TX_RETRY_COUNT) { + num_tx += loop; + nb_ops -= loop; + goto skip_tx; + } + } else { + loop += ret; + retry_count = 0; + } + } + + num_tx += loop; + nb_ops -= loop; + } + +skip_tx: + dpaa2_qp->tx_vq.tx_pkts += num_tx; + dpaa2_qp->tx_vq.err_pkts += nb_ops; + return num_tx; +} + static uint16_t dpaa2_sec_dequeue_burst(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops) @@ -3622,6 +3785,10 @@ dpaa2_sec_dev_start(struct rte_cryptodev *dev) PMD_INIT_FUNC_TRACE(); + /* Change the tx burst function if ordered queues are used */ + if (priv->en_ordered) + dev->enqueue_burst = dpaa2_sec_enqueue_burst_ordered; + memset(&attr, 0, sizeof(struct dpseci_attr)); ret = dpseci_enable(dpseci, CMD_PRI_LOW, priv->token); @@ -3834,12 +4001,46 @@ dpaa2_sec_process_atomic_event(struct qbman_swp *swp __rte_unused, ev->event_ptr = sec_fd_to_mbuf(fd); dqrr_index = qbman_get_dqrr_idx(dq); - *dpaa2_seqn(crypto_op->sym->m_src) = dqrr_index + 1; + *dpaa2_seqn(crypto_op->sym->m_src) = QBMAN_ENQUEUE_FLAG_DCA | dqrr_index; DPAA2_PER_LCORE_DQRR_SIZE++; DPAA2_PER_LCORE_DQRR_HELD |= 1 << dqrr_index; DPAA2_PER_LCORE_DQRR_MBUF(dqrr_index) = crypto_op->sym->m_src; } +static void __rte_hot +dpaa2_sec_process_ordered_event(struct qbman_swp *swp, + const struct qbman_fd *fd, + const struct qbman_result *dq, + struct dpaa2_queue *rxq, + struct rte_event *ev) +{ + struct rte_crypto_op *crypto_op = (struct rte_crypto_op *)ev->event_ptr; + + /* Prefetching mbuf */ + rte_prefetch0((void *)(size_t)(DPAA2_GET_FD_ADDR(fd)- + rte_dpaa2_bpid_info[DPAA2_GET_FD_BPID(fd)].meta_data_size)); + + /* Prefetching ipsec crypto_op stored in priv data of mbuf */ + rte_prefetch0((void *)(size_t)(DPAA2_GET_FD_ADDR(fd)-64)); + + ev->flow_id = rxq->ev.flow_id; + ev->sub_event_type = rxq->ev.sub_event_type; + ev->event_type = RTE_EVENT_TYPE_CRYPTODEV; + ev->op = RTE_EVENT_OP_NEW; + ev->sched_type = rxq->ev.sched_type; + ev->queue_id = rxq->ev.queue_id; + ev->priority = rxq->ev.priority; + ev->event_ptr = sec_fd_to_mbuf(fd); + + *dpaa2_seqn(crypto_op->sym->m_src) = DPAA2_ENQUEUE_FLAG_ORP; + *dpaa2_seqn(crypto_op->sym->m_src) |= qbman_result_DQ_odpid(dq) << + DPAA2_EQCR_OPRID_SHIFT; + *dpaa2_seqn(crypto_op->sym->m_src) |= qbman_result_DQ_seqnum(dq) << + DPAA2_EQCR_SEQNUM_SHIFT; + + qbman_swp_dqrr_consume(swp, dq); +} + int dpaa2_sec_eventq_attach(const struct rte_cryptodev *dev, int qp_id, @@ -3857,6 +4058,8 @@ dpaa2_sec_eventq_attach(const struct rte_cryptodev *dev, qp->rx_vq.cb = dpaa2_sec_process_parallel_event; else if (event->sched_type == RTE_SCHED_TYPE_ATOMIC) qp->rx_vq.cb = dpaa2_sec_process_atomic_event; + else if (event->sched_type == RTE_SCHED_TYPE_ORDERED) + qp->rx_vq.cb = dpaa2_sec_process_ordered_event; else return -EINVAL; @@ -3875,6 +4078,37 @@ dpaa2_sec_eventq_attach(const struct rte_cryptodev *dev, cfg.options |= DPSECI_QUEUE_OPT_ORDER_PRESERVATION; cfg.order_preservation_en = 1; } + + if (event->sched_type == RTE_SCHED_TYPE_ORDERED) { + struct opr_cfg ocfg; + + /* Restoration window size = 256 frames */ + ocfg.oprrws = 3; + /* Restoration window size = 512 frames for LX2 */ + if (dpaa2_svr_family == SVR_LX2160A) + ocfg.oprrws = 4; + /* Auto advance NESN window enabled */ + ocfg.oa = 1; + /* Late arrival window size disabled */ + ocfg.olws = 0; + /* ORL resource exhaustaion advance NESN disabled */ + ocfg.oeane = 0; + + if (priv->en_loose_ordered) + ocfg.oloe = 1; + else + ocfg.oloe = 0; + + ret = dpseci_set_opr(dpseci, CMD_PRI_LOW, priv->token, + qp_id, OPR_OPT_CREATE, &ocfg); + if (ret) { + RTE_LOG(ERR, PMD, "Error setting opr: ret: %d\n", ret); + return ret; + } + qp->tx_vq.cb_eqresp_free = dpaa2_sec_free_eqresp_buf; + priv->en_ordered = 1; + } + ret = dpseci_set_rx_queue(dpseci, CMD_PRI_LOW, priv->token, qp_id, &cfg); if (ret) { @@ -3979,24 +4213,34 @@ dpaa2_sec_uninit(const struct rte_cryptodev *dev) } static int -check_devargs_handler(__rte_unused const char *key, const char *value, - __rte_unused void *opaque) +check_devargs_handler(const char *key, const char *value, + void *opaque) { - dpaa2_sec_dp_dump = atoi(value); - if (dpaa2_sec_dp_dump > DPAA2_SEC_DP_FULL_DUMP) { - DPAA2_SEC_WARN("WARN: DPAA2_SEC_DP_DUMP_LEVEL is not " - "supported, changing to FULL error prints\n"); - dpaa2_sec_dp_dump = DPAA2_SEC_DP_FULL_DUMP; + struct rte_cryptodev *dev = (struct rte_cryptodev *)opaque; + struct dpaa2_sec_dev_private *priv = dev->data->dev_private; + + if (!strcmp(key, "drv_strict_order")) { + priv->en_loose_ordered = false; + } else { + dpaa2_sec_dp_dump = atoi(value); + if (dpaa2_sec_dp_dump > DPAA2_SEC_DP_FULL_DUMP) { + DPAA2_SEC_WARN("WARN: DPAA2_SEC_DP_DUMP_LEVEL is not " + "supported, changing to FULL error" + " prints\n"); + dpaa2_sec_dp_dump = DPAA2_SEC_DP_FULL_DUMP; + } } return 0; } static void -dpaa2_sec_get_devargs(struct rte_devargs *devargs, const char *key) +dpaa2_sec_get_devargs(struct rte_cryptodev *cryptodev, const char *key) { struct rte_kvargs *kvlist; + struct rte_devargs *devargs; + devargs = cryptodev->device->devargs; if (!devargs) return; @@ -4010,7 +4254,7 @@ dpaa2_sec_get_devargs(struct rte_devargs *devargs, const char *key) } rte_kvargs_process(kvlist, key, - check_devargs_handler, NULL); + check_devargs_handler, (void *)cryptodev); rte_kvargs_free(kvlist); } @@ -4101,6 +4345,7 @@ dpaa2_sec_dev_init(struct rte_cryptodev *cryptodev) cryptodev->data->nb_queue_pairs = internals->max_nb_queue_pairs; internals->hw = dpseci; internals->token = token; + internals->en_loose_ordered = true; snprintf(str, sizeof(str), "sec_fle_pool_p%d_%d", getpid(), cryptodev->data->dev_id); @@ -4115,7 +4360,8 @@ dpaa2_sec_dev_init(struct rte_cryptodev *cryptodev) goto init_error; } - dpaa2_sec_get_devargs(cryptodev->device->devargs, DRIVER_DUMP_MODE); + dpaa2_sec_get_devargs(cryptodev, DRIVER_DUMP_MODE); + dpaa2_sec_get_devargs(cryptodev, DRIVER_STRICT_ORDER); DPAA2_SEC_INFO("driver %s: created", cryptodev->data->name); return 0; @@ -4215,5 +4461,6 @@ RTE_PMD_REGISTER_DPAA2(CRYPTODEV_NAME_DPAA2_SEC_PMD, rte_dpaa2_sec_driver); RTE_PMD_REGISTER_CRYPTO_DRIVER(dpaa2_sec_crypto_drv, rte_dpaa2_sec_driver.driver, cryptodev_driver_id); RTE_PMD_REGISTER_PARAM_STRING(CRYPTODEV_NAME_DPAA2_SEC_PMD, + DRIVER_STRICT_ORDER "=" DRIVER_DUMP_MODE "="); RTE_LOG_REGISTER(dpaa2_logtype_sec, pmd.crypto.dpaa2, NOTICE); diff --git a/drivers/crypto/dpaa2_sec/dpaa2_sec_priv.h b/drivers/crypto/dpaa2_sec/dpaa2_sec_priv.h index a8f9440632..e4a82114c2 100644 --- a/drivers/crypto/dpaa2_sec/dpaa2_sec_priv.h +++ b/drivers/crypto/dpaa2_sec/dpaa2_sec_priv.h @@ -1,8 +1,6 @@ /* SPDX-License-Identifier: BSD-3-Clause - * - * Copyright (c) 2016 Freescale Semiconductor, Inc. All rights reserved. - * Copyright 2016,2020-2021 NXP - * + * Copyright (c) 2016 Freescale Semiconductor, Inc. All rights reserved. + * Copyright 2016,2019-2021 NXP */ #ifndef _DPAA2_SEC_PMD_PRIVATE_H_ @@ -37,6 +35,8 @@ struct dpaa2_sec_dev_private { uint16_t token; /**< Token required by DPxxx objects */ unsigned int max_nb_queue_pairs; /**< Max number of queue pairs supported by device */ + uint8_t en_ordered; + uint8_t en_loose_ordered; }; struct dpaa2_sec_qp { diff --git a/drivers/crypto/dpaa2_sec/mc/fsl_dpseci.h b/drivers/crypto/dpaa2_sec/mc/fsl_dpseci.h index 279e8f4d4a..c295c04f24 100644 --- a/drivers/crypto/dpaa2_sec/mc/fsl_dpseci.h +++ b/drivers/crypto/dpaa2_sec/mc/fsl_dpseci.h @@ -1,7 +1,7 @@ /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0) * * Copyright 2013-2016 Freescale Semiconductor Inc. - * Copyright 2016-2017 NXP + * Copyright 2016-2020 NXP * */ #ifndef __FSL_DPSECI_H @@ -11,6 +11,8 @@ * Contains initialization APIs and runtime control APIs for DPSECI */ +#include + struct fsl_mc_io; /** @@ -41,6 +43,16 @@ int dpseci_close(struct fsl_mc_io *mc_io, */ #define DPSECI_OPT_HAS_CG 0x000020 +/** + * Enable the Order Restoration support + */ +#define DPSECI_OPT_HAS_OPR 0x000040 + +/** + * Order Point Records are shared for the entire DPSECI + */ +#define DPSECI_OPT_OPR_SHARED 0x000080 + /** * struct dpseci_cfg - Structure representing DPSECI configuration * @options: Any combination of the following options: -- 2.25.1