From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5825144183; Fri, 7 Jun 2024 16:29:37 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8382B42E20; Fri, 7 Jun 2024 16:28:37 +0200 (CEST) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2041.outbound.protection.outlook.com [40.107.92.41]) by mails.dpdk.org (Postfix) with ESMTP id 96E0142E06 for ; Fri, 7 Jun 2024 16:28:34 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WzK0dXRW39ucyDpQyeD57Akmk/pYI/8IpHcNMRTbgmgk1jUW9v4LYdGlUW8D6E8fFQ004eugeCWfWAaoCUGz1Yzy1YuzTZG+cd4mqtt8tNxeN4KTWFQ8KpRyrO/awO7n6FhkRd1SwP20NAHO94Qek4ngKsz70Y+8WVSD8YyQh9cBcmFy2hwpQk4CSOsvFP26gQPlUAL2Ue6EyVSCERrBg9RBlTRnSs2HJa2bqBKKl+ghpILPrZE+9bokgk2wmxXpjQ+ZqqxJLd0vXI91zTnlQHItp4n8mgQcCaKIVmG/uj6hBllO/r6IW90NpsrYGvBMR8/AFKCFbWF9ZAb210jrXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TtN7+IZ1hVklixGhOBNrc4pkGPvv3AKl4yZvMV5LOnM=; b=kOYOpCWSEYoHamS3HCc/OLzOieyNsqGewQ3Hp/LIyrIeOygWqBsz5NSXXAUvw50DPeTnqeaTioATXY0yPz9zv4VC//e87GL7sMYSycdfmT5blrDR071OgHmqXOrlktQi//xxIIDPFKn2w/b7/uw20Oyd7TOhYHvW5zdDZaH7AIE5njAzhOIvg5A7hqt3e2cfQBILeo8d2n2I6dlrH+3McA0PcnGV9516izEhQPGpdtGFcxz9c0GlLPc23+/W6igujaLWYP935Qgt4Ja7Bo+nvvxvLjIUvG86DziGEov5CWiFWrdHTxHCeWqLOndTMZNOruc1bCYcpY8P498C1ErJSg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=dpdk.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TtN7+IZ1hVklixGhOBNrc4pkGPvv3AKl4yZvMV5LOnM=; b=MOyhYEcSoi88vwRGCeX4e+hNWO+C0V1sdftfeWYqmAXrmei+dzxeKPmlnI14Qgux/UxbWmED2Q9rDEF11hxs3dIoXNDusUTHDMeGXUrVvXoSomRqOwb5BbwwK4vSgOeirVG1bZg9Z6A1gsJp0N9F2tRHEfbPnQD7Eg2LWiBFKaI= Received: from CH0P223CA0009.NAMP223.PROD.OUTLOOK.COM (2603:10b6:610:116::35) by BY5PR12MB4275.namprd12.prod.outlook.com (2603:10b6:a03:20a::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7633.34; Fri, 7 Jun 2024 14:28:31 +0000 Received: from DS2PEPF0000343B.namprd02.prod.outlook.com (2603:10b6:610:116:cafe::d6) by CH0P223CA0009.outlook.office365.com (2603:10b6:610:116::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7656.22 via Frontend Transport; Fri, 7 Jun 2024 14:28:28 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS2PEPF0000343B.mail.protection.outlook.com (10.167.18.38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7633.15 via Frontend Transport; Fri, 7 Jun 2024 14:28:28 +0000 Received: from driver-dev1.pensando.io (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 7 Jun 2024 09:28:27 -0500 From: Andrew Boyer To: CC: Andrew Boyer Subject: [PATCH v3 8/9] crypto/ionic: add a watchdog operation Date: Fri, 7 Jun 2024 07:27:39 -0700 Message-ID: <20240607142740.60175-9-andrew.boyer@amd.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240607142740.60175-1-andrew.boyer@amd.com> References: <20240430202144.49899-2-andrew.boyer@amd.com> <20240607142740.60175-1-andrew.boyer@amd.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF0000343B:EE_|BY5PR12MB4275:EE_ X-MS-Office365-Filtering-Correlation-Id: 272e49a8-ea0f-483b-e123-08dc86fe195d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230031|82310400017|1800799015|36860700004|376005; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?dtgLyry35EFwUfHD5FlXyZWOfM8Qfv/Eayf/5rLnLo2+Duq7p1LIkMDlszPa?= =?us-ascii?Q?VrQSHKhCOjkjW0xScIIDpaPhY++bVyilV8xkmakKXXFaocxDyMon4BWvQZBV?= =?us-ascii?Q?Q2RlvpT200JFLh5DHAfukZ0IZMQ3UpQHD7PX+YzAoBISLnfDiX8/ANDoUa9H?= =?us-ascii?Q?PN4fP6psMK1TbJACC0m0kiPgx5teSYBbCWdE2WkFpwyCQRaJbwc9lW909Nah?= =?us-ascii?Q?RMWJsb+ylb8bYp5LOqABeFmKgEzu8Z3oo8YAOCc4QZmzihjeOlveqaOR9dAE?= =?us-ascii?Q?fj456nJwsWTjzCuSO/+pMdU1w4XRa9Aob2YHGpswMIKDqs7oN/drfXRc7tdv?= =?us-ascii?Q?wUa8Q8mX6gzg6DTBek1P51ap5LLL1US6gVkdvEJbnGkhg0ewAUu1aRPbMVJC?= =?us-ascii?Q?DnfnXtyBi3BlzQ5zjzsbI7oN1ZrF8RTs2SF85Z7eWKUir7vmSLm9auw/ldBQ?= =?us-ascii?Q?PUjOQwCJHqiXsDmvIHY+3TkFIPmUuvvXw0Ifuh5oKTk3X/yjY8+AH9X/amKm?= =?us-ascii?Q?xhTPKphURSnBSv0jg2f1N0g8vP1pVlcd8UFEm91uTU/PqVZDSfjvKq+0YWFy?= =?us-ascii?Q?PDrK+5TgjXkPYT9Hj98gUj5ukSj492bftbrR7qtsSGcuO4AVr9qMnDcggxp7?= =?us-ascii?Q?qq2Mf8vaEc+lRMOV9VFeEFt1/ZqtyQSwLeL7uCaz8yGvH8Nld/a4mOLp2U2A?= =?us-ascii?Q?9DPzajdKKv2hX7gz9pP6hshgKSks4FLt+B4ZKuWZW/EELGmuy6om1pHytFAr?= =?us-ascii?Q?8GE5w6chwwnVwJvu7WlAV+Mcgz2yYBD2hG45GVRacvd/LSAM4SSYovCn0xJc?= =?us-ascii?Q?NhByhCUnWjZkx9FqIhsrZGMsukWg+l/y2gJfDFi2y9n0k1kaSxKIOqbDvZSD?= =?us-ascii?Q?VblYt6SB2xv2q0V0P3EszMvff/BX5PFJOu0YqezakPsbqR+J7bgZc707C4PF?= =?us-ascii?Q?InfRLdDI0kNXFyVStC3MTJnJtACHSIOn6vFzAaHPYuL7sRWqvxdoXuhGu9Pl?= =?us-ascii?Q?78oDiI2tyrENwnzTO8MyvaGEuE0jBstuBGmJmO7Ulvglxr0m5NcdgktTdgt7?= =?us-ascii?Q?6gDOBkmJxnaK5DHqTU3b9HyYfr8SC2ZWFign/BO/6Y9YeoNzaDaoWiJ5tHpT?= =?us-ascii?Q?Wq/ySRsNqgd3z3h9QVj2xw/xSCSIEwcBy/v3JeTnTkd9rSUohpiILRvhIjGv?= =?us-ascii?Q?6dVlfE7JOesV7uF1cbOsnnOjfdFVNvKeBuorGC2dwhxAEo09XUCbGSKomugh?= =?us-ascii?Q?AZb0MbUrgS6m4BexmTSMlzU0GGdlcYtyQE+uR3Eh5KytMIB9eIMgtephwJ+G?= =?us-ascii?Q?Hmy0CAtYi1sFak2xIGHUMtuBXi4dG9Sd5PK1eRHAbsBYDefj25+LTM9mVHuK?= =?us-ascii?Q?uqrM7mXefvs8ZJlGp+sVoxQRpJDf3xRQVE01V1sAn9SHfwuWDA=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230031)(82310400017)(1800799015)(36860700004)(376005); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jun 2024 14:28:28.4072 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 272e49a8-ea0f-483b-e123-08dc86fe195d X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF0000343B.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4275 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org If no progress has been made within the timeout, post a dummy operation using session 0. This will restart the queue in the rare case that a doorbell is lost inside the device. Signed-off-by: Andrew Boyer --- drivers/crypto/ionic/ionic_crypto.h | 16 ++++ drivers/crypto/ionic/ionic_crypto_main.c | 31 ++++++++ drivers/crypto/ionic/ionic_crypto_ops.c | 97 +++++++++++++++++++++++- 3 files changed, 142 insertions(+), 2 deletions(-) diff --git a/drivers/crypto/ionic/ionic_crypto.h b/drivers/crypto/ionic/ionic_crypto.h index e05b458926..69c17887fb 100644 --- a/drivers/crypto/ionic/ionic_crypto.h +++ b/drivers/crypto/ionic/ionic_crypto.h @@ -89,6 +89,14 @@ struct iocpt_dev_bars { uint32_t num_bars; }; +/* Queue watchdog */ +#define IOCPT_Q_WDOG_SESS_IDX 0 +#define IOCPT_Q_WDOG_KEY_LEN 16 +#define IOCPT_Q_WDOG_IV_LEN 12 +#define IOCPT_Q_WDOG_PLD_LEN 4 +#define IOCPT_Q_WDOG_TAG_LEN 16 +#define IOCPT_Q_WDOG_OP_TYPE RTE_CRYPTO_OP_TYPE_UNDEFINED + struct iocpt_qtype_info { uint8_t version; uint8_t supported; @@ -162,7 +170,15 @@ struct iocpt_crypto_q { IOCPT_COMMON_FIELDS; /* cacheline2 */ + uint64_t last_wdog_cycles; uint16_t flags; + + /* cacheline3 */ + uint64_t enqueued_wdogs; + uint64_t dequeued_wdogs; + uint8_t wdog_iv[IOCPT_Q_WDOG_IV_LEN]; + uint8_t wdog_pld[IOCPT_Q_WDOG_PLD_LEN]; + uint8_t wdog_tag[IOCPT_Q_WDOG_TAG_LEN]; }; #define IOCPT_S_F_INITED BIT(0) diff --git a/drivers/crypto/ionic/ionic_crypto_main.c b/drivers/crypto/ionic/ionic_crypto_main.c index 179ef740ba..76afe9b3f6 100644 --- a/drivers/crypto/ionic/ionic_crypto_main.c +++ b/drivers/crypto/ionic/ionic_crypto_main.c @@ -172,6 +172,22 @@ iocpt_session_write(struct iocpt_session_priv *priv, return 0; } +static int +iocpt_session_wdog(struct iocpt_dev *dev) +{ + struct iocpt_session_priv priv = { + .dev = dev, + .index = IOCPT_Q_WDOG_SESS_IDX, + .type = IOCPT_SESS_AEAD_AES_GCM, + .key_len = IOCPT_Q_WDOG_KEY_LEN, + }; + + /* Reserve session 0 for queue watchdog */ + rte_bitmap_clear(dev->sess_bm, IOCPT_Q_WDOG_SESS_IDX); + + return iocpt_session_write(&priv, IOCPT_SESS_INIT); +} + int iocpt_session_init(struct iocpt_session_priv *priv) { @@ -491,6 +507,9 @@ iocpt_cryptoq_deinit(struct iocpt_crypto_q *cptq) IOCPT_PRINT(DEBUG, "Deinit queue %u returned %d after %u ms", cptq->q.index, err, sleep_cnt * 100); + IOCPT_PRINT(DEBUG, "Queue %u watchdog: enq %"PRIu64" deq %"PRIu64, + cptq->q.index, cptq->enqueued_wdogs, cptq->dequeued_wdogs); + cptq->flags &= ~IOCPT_Q_F_INITED; } @@ -659,9 +678,21 @@ iocpt_init(struct iocpt_dev *dev) if (err != 0) return err; + /* Write the queue watchdog key */ + err = iocpt_session_wdog(dev); + if (err != 0) { + IOCPT_PRINT(ERR, "Cannot setup watchdog session"); + goto err_out_adminq_deinit; + } + dev->state |= IOCPT_DEV_F_INITED; return 0; + +err_out_adminq_deinit: + iocpt_adminq_deinit(dev); + + return err; } void diff --git a/drivers/crypto/ionic/ionic_crypto_ops.c b/drivers/crypto/ionic/ionic_crypto_ops.c index 28b099dea2..0330fd76ad 100644 --- a/drivers/crypto/ionic/ionic_crypto_ops.c +++ b/drivers/crypto/ionic/ionic_crypto_ops.c @@ -58,7 +58,8 @@ iocpt_op_info_get(struct rte_cryptodev *cdev, struct rte_cryptodev_info *info) info->max_nb_queue_pairs = dev->max_qps; info->feature_flags = dev->features; info->capabilities = iocpt_get_caps(info->feature_flags); - info->sym.max_nb_sessions = dev->max_sessions; + /* Reserve one session for watchdog */ + info->sym.max_nb_sessions = dev->max_sessions - 1; info->driver_id = dev->driver_id; info->min_mbuf_headroom_req = 0; info->min_mbuf_tailroom_req = 0; @@ -380,12 +381,72 @@ iocpt_enqueue_sym(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops) count++; } - if (likely(count > 0)) + if (likely(count > 0)) { iocpt_q_flush(&cptq->q); + /* Restart timer if ops are being enqueued */ + cptq->last_wdog_cycles = rte_get_timer_cycles(); + } + return count; } +static void +iocpt_enqueue_wdog(struct iocpt_crypto_q *cptq) +{ + struct iocpt_queue *q = &cptq->q; + struct iocpt_crypto_desc *desc, *desc_base = q->base; + struct iocpt_crypto_sg_desc *sg_desc, *sg_desc_base = q->sg_base; + struct iocpt_crypto_sg_elem *src; + struct rte_crypto_op *wdog_op; + rte_iova_t iv_addr, pld_addr, tag_addr; + uint8_t nsge_src = 0; + uint16_t avail; + + avail = iocpt_q_space_avail(&cptq->q); + if (avail < 1) + goto out_flush; + + wdog_op = rte_zmalloc_socket("iocpt", sizeof(*wdog_op), + RTE_CACHE_LINE_SIZE, rte_socket_id()); + if (wdog_op == NULL) + goto out_flush; + + wdog_op->type = IOCPT_Q_WDOG_OP_TYPE; + wdog_op->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED; + + desc = &desc_base[q->head_idx]; + sg_desc = &sg_desc_base[q->head_idx]; + src = sg_desc->src_elems; + + /* Fill the first SGE with the IV / Nonce */ + iv_addr = rte_mem_virt2iova(cptq->wdog_iv); + iocpt_fill_sge(src, nsge_src++, iv_addr, IOCPT_Q_WDOG_IV_LEN); + + /* Fill the second SGE with the payload segment */ + pld_addr = rte_mem_virt2iova(cptq->wdog_pld); + iocpt_fill_sge(src, nsge_src++, pld_addr, IOCPT_Q_WDOG_PLD_LEN); + + /* AEAD AES-GCM: digest == authentication tag */ + tag_addr = rte_mem_virt2iova(cptq->wdog_tag); + iocpt_fill_sge(src, nsge_src++, tag_addr, IOCPT_Q_WDOG_TAG_LEN); + + desc->opcode = IOCPT_DESC_OPCODE_GCM_AEAD_ENCRYPT; + desc->flags = 0; + desc->num_src_dst_sgs = iocpt_encode_nsge_src_dst(nsge_src, 0); + desc->session_tag = rte_cpu_to_le_32(IOCPT_Q_WDOG_SESS_IDX); + + q->info[q->head_idx] = wdog_op; + q->head_idx = Q_NEXT_TO_POST(q, 1); + + IOCPT_PRINT(DEBUG, "Queue %u wdog enq %p", + q->index, wdog_op); + cptq->enqueued_wdogs++; + +out_flush: + iocpt_q_flush(q); +} + static uint16_t iocpt_dequeue_sym(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops) { @@ -395,6 +456,7 @@ iocpt_dequeue_sym(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops) struct rte_crypto_op *op; struct iocpt_crypto_comp *cq_desc_base = cq->base; volatile struct iocpt_crypto_comp *cq_desc; + uint64_t then, now, hz, delta; uint16_t count = 0; cq_desc = &cq_desc_base[cq->tail_idx]; @@ -442,6 +504,17 @@ iocpt_dequeue_sym(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops) op->status == RTE_CRYPTO_OP_STATUS_NOT_PROCESSED) break; + /* Handle watchdog operations */ + if (unlikely(op->type == IOCPT_Q_WDOG_OP_TYPE)) { + IOCPT_PRINT(DEBUG, "Queue %u wdog deq %p st %d", + q->index, op, op->status); + q->info[q->tail_idx] = NULL; + q->tail_idx = Q_NEXT_TO_SRVC(q, 1); + cptq->dequeued_wdogs++; + rte_free(op); + continue; + } + ops[count] = op; q->info[q->tail_idx] = NULL; @@ -449,6 +522,26 @@ iocpt_dequeue_sym(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops) count++; } + if (!count) { + /* + * Ring the doorbell again if no work was dequeued and work + * is still pending after the deadline. + */ + if (q->head_idx != q->tail_idx) { + then = cptq->last_wdog_cycles; + now = rte_get_timer_cycles(); + hz = rte_get_timer_hz(); + delta = (now - then) * 1000; + + if (delta >= hz * IONIC_Q_WDOG_MS) { + iocpt_enqueue_wdog(cptq); + cptq->last_wdog_cycles = now; + } + } + } else + /* Restart timer if the queue is making progress */ + cptq->last_wdog_cycles = rte_get_timer_cycles(); + return count; } -- 2.17.1