From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D336E45CE3 for ; Mon, 11 Nov 2024 07:37:51 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BCCFF4067A; Mon, 11 Nov 2024 07:37:51 +0100 (CET) Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2079.outbound.protection.outlook.com [40.107.101.79]) by mails.dpdk.org (Postfix) with ESMTP id 8000B4067A for ; Mon, 11 Nov 2024 07:37:50 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=OUqI2Ol1Tw/OAN81EBEZeboWeQg9VIn/TUlETytGrru936AW247DlzsQ+JhauSaEwQoQEIDltyXZluq0ak6ZaDYpIm8Duj+kaKSmoqiJy5jr+xg6f3yHlEUwymllN96DNDhgOhD8krQ9vNJXUXtK+8bC10InioedOK0vbqALH90oQkTsU1MRzmO25XSxWELuipj/R6fdy08T8IR3fS6gw/q4xyBqaOFLUE6mowXUFallrSM8IreX1U+etixVvehKX+WfN06V+NJhtF6ltrKKK62EyqdBE9EagZo7U29Umdc5oZWJleI2OCgNZAfxf1slq5kzWhXcqRRfIm0nAkrPgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qmLQsQZrqFKk8gOiHuBZC7nAHm/atLyOp6Eybar6cI0=; b=WwoL+2HomMCGmQ1yRfxcHX8RMgwEmZX2N5CSSSTP+YhZNk0oFvyKkDz73zkuC6V0iZWzKWzz8/NxMLk8QpNfbmCuVkpiIFO+ATwqfFWK3Yp4JO3WPO1F4fBNzeQLThiiihGenrMUHyqQsFdWB+gxfOsc1GH5jXUg5aAS73o+y3FfMDMa2lHZlmhVIK7SmNkuNdfQCQ9i2aMY72GKklERKIShDQxaffsdDdjJqsu9hPLxFy6OL5lPlkUO2ee5T4lOlUzFPX46XKxHh1Y5Dfg41uq+lq0C+d4dZVyrGbiP1t6ucbr+jdzkdDZMggIFxss3s2WyX3xr2m1suAymLWA0QA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=nxp.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qmLQsQZrqFKk8gOiHuBZC7nAHm/atLyOp6Eybar6cI0=; b=DptSCqvba4e0qh2kTCERui9rPm9iPs5gnPLtMBEahenDuBALKktgjqa9cXNj4wvNQtNqwFOAAgdJ4dzz+/EghjVxGZd3np0drYU/PJshhl5w+sogPb2rB/K48kbiG0VeuOLcAJZd9U/HWy2bfo/on6jCFBGXJOhruBYXokzDuEMrzvVOzn6zxLlPHp68C2N2A297GJPFAux+tebL3jWkY9qmsl9RUr7erXgwgwX5pITiPP4RdZgFD7KloULza1w5dNen+WYZ+ZMDO27LFVeqvOY5JFXdC8IWU5iSfmRoNVNjZg6Vinb8hDpL6PHoYIaSYXd8gWZ3YrpulrqAbWcxQQ== Received: from BN9PR03CA0938.namprd03.prod.outlook.com (2603:10b6:408:108::13) by MW4PR12MB6923.namprd12.prod.outlook.com (2603:10b6:303:208::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8137.26; Mon, 11 Nov 2024 06:37:47 +0000 Received: from BN2PEPF000044A8.namprd04.prod.outlook.com (2603:10b6:408:108:cafe::97) by BN9PR03CA0938.outlook.office365.com (2603:10b6:408:108::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8137.26 via Frontend Transport; Mon, 11 Nov 2024 06:37:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN2PEPF000044A8.mail.protection.outlook.com (10.167.243.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8158.14 via Frontend Transport; Mon, 11 Nov 2024 06:37:46 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 10 Nov 2024 22:37:28 -0800 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 10 Nov 2024 22:37:25 -0800 From: Xueming Li To: Gagandeep Singh CC: , Hemant Agrawal , "dpdk stable" Subject: patch 'bus/dpaa: fix PFDRs leaks due to FQRNIs' has been queued to stable release 23.11.3 Date: Mon, 11 Nov 2024 14:28:09 +0800 Message-ID: <20241111062847.216344-84-xuemingl@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241111062847.216344-1-xuemingl@nvidia.com> References: <20241111062847.216344-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF000044A8:EE_|MW4PR12MB6923:EE_ X-MS-Office365-Filtering-Correlation-Id: f711d85f-13f3-4687-29e4-08dd021b5ae0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|376014|1800799024|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?qfHW24zY+Ov8Ng2Yvp61a/56ZQkloZmFmyLY+SXjlXcXZ58GqtdR70MZjcAK?= =?us-ascii?Q?HL7M5XxBZpwe7oTBYzCZ9kAxjsI8vv7nSFdAORHMAJL8vX5UxmUJv3Vo0PnG?= =?us-ascii?Q?BTP0b2VUZJIZIsfYAgN26Zgb8eFISXD+qO4o5zns4cRKwKOAynDHE0bj7+qw?= =?us-ascii?Q?BGx0hWA7PP8ULIT1H3dHra0juyY4bontCbnxMGhag93D8FquR9NhQVOa9TVA?= =?us-ascii?Q?Kpy2KNAjyGlJP0hBJldRAXFhKOp00n5DncwXfYK4ULe27JTM20qABssFIZOX?= =?us-ascii?Q?+7G/OeXVYFOQuq8DwAM7fyarWGi0VguJOvsyExG/n+EINAOHphJOdPuZbaFq?= =?us-ascii?Q?AGu7pltXkWq33v5zhe2EjteotEE/6D5RQ/DsyxfdwSeOgHJuhXYQSzemuSfW?= =?us-ascii?Q?2gvbPZTnb4dGultTWl4FWgnI17gQGs+pCS2J2n7m6Uz79eGrlegnmqmtFbSM?= =?us-ascii?Q?9h8iOp4r/2CsgzEIbRHusV6/qhsO+mPT+Z9WAfhGPOQtE2worWV3siQbV8Fv?= =?us-ascii?Q?D+3l0WcFT1HVhGzzu1iv6lh7aWy/ZDzwdPc75W8IkklAcGPC3lUOY25k3+te?= =?us-ascii?Q?xYz58seH3criMZ4inlOw2pgjDmb+kVFaFxhws9AJZ1O0yBlvTNc//QZKKWIx?= =?us-ascii?Q?Xt7iPJwW+nBZkFrawbjWFQeZFRA91rU+MQYnszlNeB48hBxGdt0uGZ4ycFIa?= =?us-ascii?Q?iOJLmCzO/EcM1FCRRfoeHYOtLSB7whmzhexVTi390GUeUyUaF5xebc+qS08z?= =?us-ascii?Q?OjMwsUbNLUERMmi0Z7gEgP/pLGe2JA/jVNa7BDskyu8SOLx37B3vdNDLlpzH?= =?us-ascii?Q?UwUPyVAJy/AMB8PiTAYFnoKXGn45JQIlM04N6uR/JbOqzgEp2k8toYsLyUQ9?= =?us-ascii?Q?czpIzlAgYbGZWrxXM3JhhE4BoKbWTbCDNTEEmc7titULhhc1nSA1GCS4JlOM?= =?us-ascii?Q?edwDVq04jLM+3MuKKmtFXfUzIsGKtfxcYSPIn46GcRjtNDrc+KUy+bzoHhNH?= =?us-ascii?Q?150ZLt3wF54UFnfsf1bFnHyGZxjYXayeQ53CvcsRzfyCR+gtzQNxnjkXrMju?= =?us-ascii?Q?Y2p461sb7ViFENVRk9fC3zCZKCexz39u0aJ+hygALaA0Z5OAYkNBW4UiQRQr?= =?us-ascii?Q?NsxZKQWoqJ6ctbdhcoNFNN1QddjR/QUwx8dg3A1ta8QTacOWpJMhbobq9rwD?= =?us-ascii?Q?4CBswImT2zl3Qq/wYvcsxhW9xBU9N2kOYAhgg9PS8EO1VLiFWm7fD6w+26Gq?= =?us-ascii?Q?tD3d6tLcTo8Mcs6cNd0GGEdopLnOEWrOGwpp8q1pEPF4iQK09IyNdUvJayfU?= =?us-ascii?Q?z/a/arP8ZSiRmTw3AysluIoNvhJ53DpU3qSpmh6dxsbLLvKWtF6YQOc89NA2?= =?us-ascii?Q?hyXOiDW3AWAMImmK2W6wYcvm1IYlbczIhF1U97QJIpAaEfE1zw=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(376014)(1800799024)(82310400026)(36860700013); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Nov 2024 06:37:46.6459 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f711d85f-13f3-4687-29e4-08dd021b5ae0 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF000044A8.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB6923 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 23.11.3 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 11/30/24. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://git.dpdk.org/dpdk-stable/log/?h=23.11-staging This queued commit can be viewed at: https://git.dpdk.org/dpdk-stable/commit/?h=23.11-staging&id=508ee4007a756e0eea944d532e3edb31135b31d9 Thanks. Xueming Li --- >From 508ee4007a756e0eea944d532e3edb31135b31d9 Mon Sep 17 00:00:00 2001 From: Gagandeep Singh Date: Tue, 1 Oct 2024 16:33:08 +0530 Subject: [PATCH] bus/dpaa: fix PFDRs leaks due to FQRNIs Cc: Xueming Li [ upstream commit b292acc3c4a8fd5104cfdfa5c6d3d0df95b6543b ] When a Retire FQ command is executed on a FQ in the Tentatively Scheduled or Parked states, in that case FQ is retired immediately and a FQRNI (Frame Queue Retirement Notification Immediate) message is generated. Software must read this message from MR and consume it to free the memory used by it. Although it is not mentioned about which memory to be used by FQRNIs in the RM but through experiments it is proven that it can use PFDRs. So if these messages are allowed to build up indefinitely then PFDR resources can become exhausted and cause enqueues to stall. Therefore software must consume these MR messages on a regular basis to avoid depleting the available PFDR resources. This is the PFDRs leak issue which user can experience while using the DPDK crypto driver and creating and destroying the sessions multiple times. On a session destroy, DPDK calls the qman_retire_fq() for each FQ used by the session, but it does not handle the FQRNIs generated and allowed them to build up indefinitely in MR. This patch fixes this issue by consuming the FQRNIs received from MR immediately after FQ retire by calling drain_mr_fqrni(). Please note that this drain_mr_fqrni() only look for FQRNI type messages to consume. If there are other type of messages like FQRN, FQRL, FQPN, ERN etc. also coming on MR then those messages need to be handled separately. Fixes: c47ff048b99a ("bus/dpaa: add QMAN driver core routines") Signed-off-by: Gagandeep Singh Acked-by: Hemant Agrawal --- drivers/bus/dpaa/base/qbman/qman.c | 46 ++++++++++++++++-------------- 1 file changed, 25 insertions(+), 21 deletions(-) diff --git a/drivers/bus/dpaa/base/qbman/qman.c b/drivers/bus/dpaa/base/qbman/qman.c index 83db0a534e..f06992ca48 100644 --- a/drivers/bus/dpaa/base/qbman/qman.c +++ b/drivers/bus/dpaa/base/qbman/qman.c @@ -294,10 +294,32 @@ static inline void qman_stop_dequeues_ex(struct qman_portal *p) qm_dqrr_set_maxfill(&p->p, 0); } +static inline void qm_mr_pvb_update(struct qm_portal *portal) +{ + register struct qm_mr *mr = &portal->mr; + const struct qm_mr_entry *res = qm_cl(mr->ring, mr->pi); + +#ifdef RTE_LIBRTE_DPAA_HWDEBUG + DPAA_ASSERT(mr->pmode == qm_mr_pvb); +#endif + /* when accessing 'verb', use __raw_readb() to ensure that compiler + * inlining doesn't try to optimise out "excess reads". + */ + if ((__raw_readb(&res->ern.verb) & QM_MR_VERB_VBIT) == mr->vbit) { + mr->pi = (mr->pi + 1) & (QM_MR_SIZE - 1); + if (!mr->pi) + mr->vbit ^= QM_MR_VERB_VBIT; + mr->fill++; + res = MR_INC(res); + } + dcbit_ro(res); +} + static int drain_mr_fqrni(struct qm_portal *p) { const struct qm_mr_entry *msg; loop: + qm_mr_pvb_update(p); msg = qm_mr_current(p); if (!msg) { /* @@ -319,6 +341,7 @@ loop: do { now = mfatb(); } while ((then + 10000) > now); + qm_mr_pvb_update(p); msg = qm_mr_current(p); if (!msg) return 0; @@ -481,27 +504,6 @@ static inline int qm_mr_init(struct qm_portal *portal, return 0; } -static inline void qm_mr_pvb_update(struct qm_portal *portal) -{ - register struct qm_mr *mr = &portal->mr; - const struct qm_mr_entry *res = qm_cl(mr->ring, mr->pi); - -#ifdef RTE_LIBRTE_DPAA_HWDEBUG - DPAA_ASSERT(mr->pmode == qm_mr_pvb); -#endif - /* when accessing 'verb', use __raw_readb() to ensure that compiler - * inlining doesn't try to optimise out "excess reads". - */ - if ((__raw_readb(&res->ern.verb) & QM_MR_VERB_VBIT) == mr->vbit) { - mr->pi = (mr->pi + 1) & (QM_MR_SIZE - 1); - if (!mr->pi) - mr->vbit ^= QM_MR_VERB_VBIT; - mr->fill++; - res = MR_INC(res); - } - dcbit_ro(res); -} - struct qman_portal * qman_init_portal(struct qman_portal *portal, const struct qm_portal_config *c, @@ -1825,6 +1827,8 @@ int qman_retire_fq(struct qman_fq *fq, u32 *flags) } out: FQUNLOCK(fq); + /* Draining FQRNIs, if any */ + drain_mr_fqrni(&p->p); return rval; } -- 2.34.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2024-11-11 14:23:08.964983062 +0800 +++ 0083-bus-dpaa-fix-PFDRs-leaks-due-to-FQRNIs.patch 2024-11-11 14:23:05.212192838 +0800 @@ -1 +1 @@ -From b292acc3c4a8fd5104cfdfa5c6d3d0df95b6543b Mon Sep 17 00:00:00 2001 +From 508ee4007a756e0eea944d532e3edb31135b31d9 Mon Sep 17 00:00:00 2001 @@ -4,0 +5,3 @@ +Cc: Xueming Li + +[ upstream commit b292acc3c4a8fd5104cfdfa5c6d3d0df95b6543b ] @@ -37 +39,0 @@ -Cc: stable@dpdk.org @@ -46 +48 @@ -index 301057723e..9c90ee25a6 100644 +index 83db0a534e..f06992ca48 100644 @@ -49 +51 @@ -@@ -292,10 +292,32 @@ static inline void qman_stop_dequeues_ex(struct qman_portal *p) +@@ -294,10 +294,32 @@ static inline void qman_stop_dequeues_ex(struct qman_portal *p) @@ -82 +84 @@ -@@ -317,6 +339,7 @@ loop: +@@ -319,6 +341,7 @@ loop: @@ -90 +92 @@ -@@ -479,27 +502,6 @@ static inline int qm_mr_init(struct qm_portal *portal, +@@ -481,27 +504,6 @@ static inline int qm_mr_init(struct qm_portal *portal, @@ -118 +120 @@ -@@ -1794,6 +1796,8 @@ int qman_retire_fq(struct qman_fq *fq, u32 *flags) +@@ -1825,6 +1827,8 @@ int qman_retire_fq(struct qman_fq *fq, u32 *flags)