From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BE34945A68; Mon, 30 Sep 2024 14:16:36 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DB9ED40647; Mon, 30 Sep 2024 14:16:15 +0200 (CEST) Received: from AM0PR83CU005.outbound.protection.outlook.com (mail-westeuropeazon11010015.outbound.protection.outlook.com [52.101.69.15]) by mails.dpdk.org (Postfix) with ESMTP id 60FD6402EF; Mon, 30 Sep 2024 14:16:12 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ZPxS7WJlj+NVYAgq96mBRr/XrZFALSTo8lKTiBhxS9TuZWauMLtihV1UNlch52+FuXOuvvPCXasXLP0AzFfjvDfrCUgVcB5n9tHSkGgr+9EMb7Dnf70JQFoEu2JCybnZHVkFRQCX9e5I28jiiH9CEITEGfKP30/RDOtgtwhCNd/O7rNWNZr8O1kkwyggxLrORjR4d/wUKa5xJdDIErErR9YN07s5WehtSSF2G7+fZq4CbYiSs6xKNBB7RLZqBCADP/KBjSnScQGm7JMS9mqwn0wSlHzGM6548jd4oZrMmDyNBX6LRMWcyz6vh0C8aVqV34/uOapGb6SWpvzllVX+QQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VQTBFf6iffqhdrfBDCMQV7dbsStZ0GSTY+9yhr9iFJw=; b=HyHW8ZJjgiCMQUu9e7yadBLVi+255vZ2aKnKlUsKN+skN0yItKHPzg853kP/sQD39LZvHy6Eb1qbgZgu70AAVCDUFKvmOONbnkeCZpfJJrXhSyZlS5XbkdQTBTX68a5s945efhG3igqTl310R28hG2cqlPelxocIoX6aX0bNUOVrT205qssExMj3B6KRzguSGUlk90BmhuzyWFY+uTavBl63nTJ4t3GD9teErTAptGXZ5sRkrAEJD1z3yYPmZZNpuASBvTZInDGCjtRv0QeVgUgAmuDhRVkMgZJfP0JFeOXCto3dw+TeLbwfTFnihY60JeY9G4aWJ7447a2B1m0Ukg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nxp.com; dmarc=pass action=none header.from=nxp.com; dkim=pass header.d=nxp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VQTBFf6iffqhdrfBDCMQV7dbsStZ0GSTY+9yhr9iFJw=; b=gvFY34cSwYWkF4QICm8RXgb4vOwz35pO2aAzf6SYFopjggRHBKNizFsUOpoiMkNEQUDYrZo3VaYrYXOvgkXt2oy2cVBxRYJPJOwki1iOK3GlxeR1bJxTF4T6w43sIxWKg9FXZiJ0F7DEtFzOq3Dytd8KbNGmlPjhn1LUBGAJNFJc3Q2j+UUli+Aal1BD4Kf5TzTCqOm96Jeumm37ZiNk6Qq/Ip26U1Y8GH4K9+MvS7ix6O9oaCKsoSnt4GAwwyF3ikmIMP9aeU7BzM7uTrRRxMNYdDOVE5LhnFZRPk9iCXckdOHNA/N8SantOerGJCGi1JzAr22H9eixj9TE2lfr0Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nxp.com; Received: from PAXPR04MB9328.eurprd04.prod.outlook.com (2603:10a6:102:2b6::15) by GV1PR04MB10583.eurprd04.prod.outlook.com (2603:10a6:150:203::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8005.26; Mon, 30 Sep 2024 12:16:07 +0000 Received: from PAXPR04MB9328.eurprd04.prod.outlook.com ([fe80::367:d59:375:fb87]) by PAXPR04MB9328.eurprd04.prod.outlook.com ([fe80::367:d59:375:fb87%3]) with mapi id 15.20.8005.026; Mon, 30 Sep 2024 12:16:07 +0000 From: Hemant Agrawal To: dev@dpdk.org Cc: ferruh.yigit@amd.com, Gagandeep Singh , stable@dpdk.org Subject: [PATCH v4 01/18] bus/dpaa: fix PFDRs leaks due to FQRNIs Date: Mon, 30 Sep 2024 17:45:08 +0530 Message-Id: <20240930121525.3452847-2-hemant.agrawal@nxp.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240930121525.3452847-1-hemant.agrawal@nxp.com> References: <20240823073240.3708320-1-hemant.agrawal@nxp.com> <20240930121525.3452847-1-hemant.agrawal@nxp.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: SG2PR01CA0124.apcprd01.prod.exchangelabs.com (2603:1096:4:40::28) To PAXPR04MB9328.eurprd04.prod.outlook.com (2603:10a6:102:2b6::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR04MB9328:EE_|GV1PR04MB10583:EE_ X-MS-Office365-Filtering-Correlation-Id: 14502232-a9ad-47ad-296b-08dce149a98e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|366016|52116014|376014|38350700014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?nm7iSTwJLPxmoyc3K827MfDV9xe2VTyrpMzvFVVd2nO5IrEnOygoohJqp8Nw?= =?us-ascii?Q?0+lyvtTLUJXWB53RbFVq6YARfhOaSJPtj2yWXoHGSVaGLhQgqF3qKdMgolIM?= =?us-ascii?Q?bg7gT6aetmbTaeU5F01PGduYhNToIxCfUXNvHzKDKdT86PLrGHQnaoFaKwv9?= =?us-ascii?Q?PzK6JKHNDdUoW7Pk1cWoCbNUnfWjJ0bEiJUH7JDxrdAbplLrYojald2Z9Hjx?= =?us-ascii?Q?Jlf3wtlaonY0uc+C32YZestmAH8pSoCDOsAp173DZqi2n/m28pI0qdSjg8kA?= =?us-ascii?Q?gwPEP6JrfImtagtqyEffRwaPi0eKWMvF5eUVH3ucX11rbk2G6h/0Y7FsgrWn?= =?us-ascii?Q?OcjM3/5D38a7YsUZ9/LsnJt7UeMBUv/Tc3yEY8sCwAhS1WCgG07y7tAXzlWn?= =?us-ascii?Q?6k69zDHLT7SprfS5XvAoUor+R+5CcBXbMG4tPwHRv8sL4KSrboSvEOmzDRA5?= =?us-ascii?Q?eSiGbYhM8qjEoHs9Ew8E5o2+K+S0c8WF0Mqh63NxIKkYut/L5yTaH3R1op8w?= =?us-ascii?Q?EY3+T2ZNzdm0m92f1k0xYG3Uabu+ow3an1EcHoNan+0BVNd600Tah2VIsb1O?= =?us-ascii?Q?YPcRJtWL16sjmj+wIMcab3mDwCPsDFI8cKKnBGBXBYMxWI1vDZFPk47TRcR3?= =?us-ascii?Q?oQqNgxufNOoUmoMqWRIDCrVfQYhNIZwbyHEqEuhbwUr0+TV71Bq8kQ+OWxWm?= =?us-ascii?Q?QzowLbXwVU6gGo5t0aBilA3TaheANToUYc5N75KXxebSwJkACu76S6QqMOsB?= =?us-ascii?Q?l7BdUne0a3mJYWi6ACEqJgku/HKYJOtnjcljHisv5BpwSyxd+IpLPinceItu?= =?us-ascii?Q?Xj+Otksg2Y5SI1YMlx76ueC0fZL/keYJGkUFTGX291aF+qOy/ksl4G9iAQKd?= =?us-ascii?Q?WM7K6KnsEb8JUprYBOoiUAGtG+kMh0BIKd1Jotwv2f9G9RIK1TnU+VzNW318?= =?us-ascii?Q?6BJyTq5ShiQ8nLA7Nym5OPDUJyt6e7xrHgnkTDeCTmAwO7+Q1/uyIfqo8NgK?= =?us-ascii?Q?rssJJGl1XCy2bIQOoRpNpf3SPgoUEBypbbK41ZyHPkEqw0d3WFUQT+FfN83s?= =?us-ascii?Q?MLw+r0vU46VATS5NMQXflQ8SD/3sZuryvVfgRJWXEJ1hZUcyfKvKo5XZEq5j?= =?us-ascii?Q?ecOGi24JhaR6pdUpHv3tpovmxpbBnhRvrMi0Mm8tet9nTi5Mojm7uYp/M+0E?= =?us-ascii?Q?64cJdWNx3+VHeVKusNYByaGpF/JsbvsHP6zlC7zWXKM8382FVu5o39IcUiJj?= =?us-ascii?Q?z+VeYMHu+qWr3wkZdI18cLQAdzV85nnxy2yhy7lVpvsmATLIsJixVZT6k0lI?= =?us-ascii?Q?ksCGBZWhQAdbRHNgjxlN2KUPjKbXEQ+QOJleB5SoOvyupg=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAXPR04MB9328.eurprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(52116014)(376014)(38350700014); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?JBmwyE5jau61A+qg1kWrpqv1L7tOlEs2haosI3ZkFGdShssBDXVJVwjPJ7uY?= =?us-ascii?Q?mrITaED5IJfu5a+BbC1Ms2o39DRu7q06nhXhrQwdtAajOPT4ZgKYbM7jlKQb?= =?us-ascii?Q?J86sIiSDrwECp/ibRKNIcpNoyDM7DcAAD7p2D9bdWcB4dlXZXblYuJXQNJ5g?= =?us-ascii?Q?FulOf3HJyn3bE6QOg9xjg7BdeiCwyNHLxg3FrPZYyMmrro3cBC0Z4NPSmQt4?= =?us-ascii?Q?ofEjdhiTBl0f5EBFXFGv7fCMokyMmd0nGCMwHB65ykS6HhKpQvBhOHhvx0rU?= =?us-ascii?Q?q01Xw8jiAVOY1HHnOE4/jc9WMn1cxKV4yq1Ehy6yvLcvVOU7mvzFV3zUzB03?= =?us-ascii?Q?frNJvitSh/kmV+lPWetD8zdmdm/7dDsOX0qeoUrhe+VtnjLzzmbVRSrdUngH?= =?us-ascii?Q?KEUuDNuqRVtaYaXw/GVZKjvi6TAlYNFMgG5RbTfuZr7W88qgsGlu0qeUQtUQ?= =?us-ascii?Q?5T4FdGKSOONk+5Hq2PXxdq8eoAHXQqwRpVKH1lLOTYSLnHNPSeGGIdC2Hid3?= =?us-ascii?Q?CTXKdvhTsvQYdg8lvuOtXT6AEHemD6qmtXhe+nzAlntuLAIulo0j2KdmWBE9?= =?us-ascii?Q?YLzDJI3n+aYmnq64MfY72UYs9AZiHFXKLNkxGSbwmxysDRiBznLonEhaLdAw?= =?us-ascii?Q?e0XUcIs/1yMaBBTYGbcINJiYAEjQoFYorEqrjUzKtZvpBtQtKczC/10fHHbu?= =?us-ascii?Q?SN+o+kxFLb4nEQJ8pTdM78ED/D/oG4MaZdiFGpffIhPfWV+cmMmzaE8MDA1U?= =?us-ascii?Q?vSCLv6h1QdjQBtlMlHTV0WEfvnINQMEGFTYHJp9bhDSdpZX/ldK9eu8Dk9cq?= =?us-ascii?Q?B471o1NiNsK60X6Xywu9bgo8LuqvQ8GwSHaF8sF/v+UVxlvqdBlG7ZVww33R?= =?us-ascii?Q?X6wG4bsGQocdWyZF+4j9hNAhMzjkLQSiOP/gkq+tXsXyB5VRgd9hfI/qUZ51?= =?us-ascii?Q?znikgTFGHGxXFcGImMpGAGRfGwvtMXjCy2pCL4ILnkmYIcy+HZeWIWcabnmf?= =?us-ascii?Q?kXm2qUAWE06FEADF6FxNFeuZm8YAQR6Exuen8m2x8MLN/vkusDnX/gfZ2ozC?= =?us-ascii?Q?TfdWAndD+Vgs09vGtOlb9RiRVy8upxPywCDeg+V+8yDoKqx6fETrnjXfCHmF?= =?us-ascii?Q?Gzd1GeQTXa92JgEfLc/+BTNXmpb3thaSpYl7zNMVURYpQZWH3qgmAdjem/4H?= =?us-ascii?Q?Y37vqwXXaDVAJ2JYu9z2RpNhT7Dn3aN/xIRfxctL2hSr4Hdx/CSbFnMlvk82?= =?us-ascii?Q?BuExcwpYFG2zYGGCZkgj9uIork2NvAcja3GJkSUAJbhxzR76xK35JeqMMd21?= =?us-ascii?Q?2kxDrR5GzOzxYQcmwGQo3M6ErXb6s7ZM5XUqU5DmFSt7sIMYsE9rftZzf+sT?= =?us-ascii?Q?tSdKeEcZ+auUuFd91v5p26QgN6cWsBG1itpitMk+Q+2egzWnM4aL7ixBoKjw?= =?us-ascii?Q?AdAuC0bgWbiZ3GoU8InXrKHz1yePNF6YzRm3h4L4u7fpfGhalBEAamx14Kwo?= =?us-ascii?Q?sdEjCUtjV6UmC2a4HkqP/RzJtEQ+wG3EYqvZdj95kv950UWBtn/FQPhTREgY?= =?us-ascii?Q?k9mlo/uNG5J8GjQ1OlojsNp73JmRupO84f9fGzDALM9/ij54GJuI+UVWNrUg?= =?us-ascii?Q?hw=3D=3D?= X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: 14502232-a9ad-47ad-296b-08dce149a98e X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB9328.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Sep 2024 12:16:07.4807 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2E9H5qMcNT87NhkOUHvScDhaP7JGBhWTQsAAkhDJ4x1Z4RYPbt6OTRr26NFaZooNTsV47g04dgerumjbUOj1LA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV1PR04MB10583 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Gagandeep Singh When a Retire FQ command is executed on a FQ in the Tentatively Scheduled or Parked states, in that case FQ is retired immediately and a FQRNI (Frame Queue Retirement Notification Immediate) message is generated. Software must read this message from MR and consume it to free the memory used by it. Although it is not mentioned about which memory to be used by FQRNIs in the RM but through experiments it is proven that it can use PFDRs. So if these messages are allowed to build up indefinitely then PFDR resources can become exhausted and cause enqueues to stall. Therefore software must consume these MR messages on a regular basis to avoid depleting the available PFDR resources. This is the PFDRs leak issue which user can experienace while using the DPDK crypto driver and creating and destroying the sessions multiple times. On a session destroy, DPDK calls the qman_retire_fq() for each FQ used by the session, but it does not handle the FQRNIs generated and allowed them to build up indefinitely in MR. This patch fixes this issue by consuming the FQRNIs received from MR immediately after FQ retire by calling drain_mr_fqrni(). Please note that this drain_mr_fqrni() only look for FQRNI type messages to consume. If there are other type of messages like FQRN, FQRL, FQPN, ERN etc. also coming on MR then those messages need to be handled separately. Fixes: c47ff048b99a ("bus/dpaa: add QMAN driver core routines") Cc: stable@dpdk.org Signed-off-by: Gagandeep Singh --- drivers/bus/dpaa/base/qbman/qman.c | 46 ++++++++++++++++-------------- 1 file changed, 25 insertions(+), 21 deletions(-) diff --git a/drivers/bus/dpaa/base/qbman/qman.c b/drivers/bus/dpaa/base/qbman/qman.c index 301057723e..9c90ee25a6 100644 --- a/drivers/bus/dpaa/base/qbman/qman.c +++ b/drivers/bus/dpaa/base/qbman/qman.c @@ -292,10 +292,32 @@ static inline void qman_stop_dequeues_ex(struct qman_portal *p) qm_dqrr_set_maxfill(&p->p, 0); } +static inline void qm_mr_pvb_update(struct qm_portal *portal) +{ + register struct qm_mr *mr = &portal->mr; + const struct qm_mr_entry *res = qm_cl(mr->ring, mr->pi); + +#ifdef RTE_LIBRTE_DPAA_HWDEBUG + DPAA_ASSERT(mr->pmode == qm_mr_pvb); +#endif + /* when accessing 'verb', use __raw_readb() to ensure that compiler + * inlining doesn't try to optimise out "excess reads". + */ + if ((__raw_readb(&res->ern.verb) & QM_MR_VERB_VBIT) == mr->vbit) { + mr->pi = (mr->pi + 1) & (QM_MR_SIZE - 1); + if (!mr->pi) + mr->vbit ^= QM_MR_VERB_VBIT; + mr->fill++; + res = MR_INC(res); + } + dcbit_ro(res); +} + static int drain_mr_fqrni(struct qm_portal *p) { const struct qm_mr_entry *msg; loop: + qm_mr_pvb_update(p); msg = qm_mr_current(p); if (!msg) { /* @@ -317,6 +339,7 @@ static int drain_mr_fqrni(struct qm_portal *p) do { now = mfatb(); } while ((then + 10000) > now); + qm_mr_pvb_update(p); msg = qm_mr_current(p); if (!msg) return 0; @@ -479,27 +502,6 @@ static inline int qm_mr_init(struct qm_portal *portal, return 0; } -static inline void qm_mr_pvb_update(struct qm_portal *portal) -{ - register struct qm_mr *mr = &portal->mr; - const struct qm_mr_entry *res = qm_cl(mr->ring, mr->pi); - -#ifdef RTE_LIBRTE_DPAA_HWDEBUG - DPAA_ASSERT(mr->pmode == qm_mr_pvb); -#endif - /* when accessing 'verb', use __raw_readb() to ensure that compiler - * inlining doesn't try to optimise out "excess reads". - */ - if ((__raw_readb(&res->ern.verb) & QM_MR_VERB_VBIT) == mr->vbit) { - mr->pi = (mr->pi + 1) & (QM_MR_SIZE - 1); - if (!mr->pi) - mr->vbit ^= QM_MR_VERB_VBIT; - mr->fill++; - res = MR_INC(res); - } - dcbit_ro(res); -} - struct qman_portal * qman_init_portal(struct qman_portal *portal, const struct qm_portal_config *c, @@ -1794,6 +1796,8 @@ int qman_retire_fq(struct qman_fq *fq, u32 *flags) } out: FQUNLOCK(fq); + /* Draining FQRNIs, if any */ + drain_mr_fqrni(&p->p); return rval; } -- 2.25.1