DPDK patches and discussions
 help / color / mirror / Atom feed
From: Hemant Agrawal <hemant.agrawal@nxp.com>
To: dev@dpdk.org
Cc: ferruh.yigit@amd.com, Gagandeep Singh <g.singh@nxp.com>, stable@dpdk.org
Subject: [PATCH v4 01/18] bus/dpaa: fix PFDRs leaks due to FQRNIs
Date: Mon, 30 Sep 2024 17:45:08 +0530	[thread overview]
Message-ID: <20240930121525.3452847-2-hemant.agrawal@nxp.com> (raw)
In-Reply-To: <20240930121525.3452847-1-hemant.agrawal@nxp.com>

From: Gagandeep Singh <g.singh@nxp.com>

When a Retire FQ command is executed on a FQ in the
Tentatively Scheduled or Parked states, in that case FQ
is retired immediately and a FQRNI (Frame Queue Retirement
Notification Immediate) message is generated. Software
must read this message from MR and consume it to free
the memory used by it.

Although it is not mentioned about which memory to be used
by FQRNIs in the RM but through experiments it is proven
that it can use PFDRs. So if these messages are allowed to
build up indefinitely then PFDR resources can become exhausted
and cause enqueues to stall. Therefore software must consume
these MR messages on a regular basis to avoid depleting
the available PFDR resources.

This is the PFDRs leak issue which user can experienace while
using the DPDK crypto driver and creating and destroying the
sessions multiple times. On a session destroy, DPDK calls the
qman_retire_fq() for each FQ used by the session, but it does
not handle the FQRNIs generated and allowed them to build up
indefinitely in MR.

This patch fixes this issue by consuming the FQRNIs received
from MR immediately after FQ retire by calling drain_mr_fqrni().

Please note that this drain_mr_fqrni() only look for
FQRNI type messages to consume. If there are other type of messages
like FQRN, FQRL, FQPN, ERN etc. also coming on MR then those
messages need to be handled separately.

Fixes: c47ff048b99a ("bus/dpaa: add QMAN driver core routines")
Cc: stable@dpdk.org

Signed-off-by: Gagandeep Singh <g.singh@nxp.com>
---
 drivers/bus/dpaa/base/qbman/qman.c | 46 ++++++++++++++++--------------
 1 file changed, 25 insertions(+), 21 deletions(-)

diff --git a/drivers/bus/dpaa/base/qbman/qman.c b/drivers/bus/dpaa/base/qbman/qman.c
index 301057723e..9c90ee25a6 100644
--- a/drivers/bus/dpaa/base/qbman/qman.c
+++ b/drivers/bus/dpaa/base/qbman/qman.c
@@ -292,10 +292,32 @@ static inline void qman_stop_dequeues_ex(struct qman_portal *p)
 		qm_dqrr_set_maxfill(&p->p, 0);
 }
 
+static inline void qm_mr_pvb_update(struct qm_portal *portal)
+{
+	register struct qm_mr *mr = &portal->mr;
+	const struct qm_mr_entry *res = qm_cl(mr->ring, mr->pi);
+
+#ifdef RTE_LIBRTE_DPAA_HWDEBUG
+	DPAA_ASSERT(mr->pmode == qm_mr_pvb);
+#endif
+	/* when accessing 'verb', use __raw_readb() to ensure that compiler
+	 * inlining doesn't try to optimise out "excess reads".
+	 */
+	if ((__raw_readb(&res->ern.verb) & QM_MR_VERB_VBIT) == mr->vbit) {
+		mr->pi = (mr->pi + 1) & (QM_MR_SIZE - 1);
+		if (!mr->pi)
+			mr->vbit ^= QM_MR_VERB_VBIT;
+		mr->fill++;
+		res = MR_INC(res);
+	}
+	dcbit_ro(res);
+}
+
 static int drain_mr_fqrni(struct qm_portal *p)
 {
 	const struct qm_mr_entry *msg;
 loop:
+	qm_mr_pvb_update(p);
 	msg = qm_mr_current(p);
 	if (!msg) {
 		/*
@@ -317,6 +339,7 @@ static int drain_mr_fqrni(struct qm_portal *p)
 		do {
 			now = mfatb();
 		} while ((then + 10000) > now);
+		qm_mr_pvb_update(p);
 		msg = qm_mr_current(p);
 		if (!msg)
 			return 0;
@@ -479,27 +502,6 @@ static inline int qm_mr_init(struct qm_portal *portal,
 	return 0;
 }
 
-static inline void qm_mr_pvb_update(struct qm_portal *portal)
-{
-	register struct qm_mr *mr = &portal->mr;
-	const struct qm_mr_entry *res = qm_cl(mr->ring, mr->pi);
-
-#ifdef RTE_LIBRTE_DPAA_HWDEBUG
-	DPAA_ASSERT(mr->pmode == qm_mr_pvb);
-#endif
-	/* when accessing 'verb', use __raw_readb() to ensure that compiler
-	 * inlining doesn't try to optimise out "excess reads".
-	 */
-	if ((__raw_readb(&res->ern.verb) & QM_MR_VERB_VBIT) == mr->vbit) {
-		mr->pi = (mr->pi + 1) & (QM_MR_SIZE - 1);
-		if (!mr->pi)
-			mr->vbit ^= QM_MR_VERB_VBIT;
-		mr->fill++;
-		res = MR_INC(res);
-	}
-	dcbit_ro(res);
-}
-
 struct qman_portal *
 qman_init_portal(struct qman_portal *portal,
 		   const struct qm_portal_config *c,
@@ -1794,6 +1796,8 @@ int qman_retire_fq(struct qman_fq *fq, u32 *flags)
 	}
 out:
 	FQUNLOCK(fq);
+	/* Draining FQRNIs, if any */
+	drain_mr_fqrni(&p->p);
 	return rval;
 }
 
-- 
2.25.1


  reply	other threads:[~2024-09-30 12:16 UTC|newest]

Thread overview: 129+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-08-01 10:52 [PATCH 00/17] NXP DPAA ETH driver enhancement and fixes Hemant Agrawal
2024-08-01 10:52 ` [PATCH 01/17] bus/dpaa: fix PFDRs leaks due to FQRNIs Hemant Agrawal
2024-08-01 10:52 ` [PATCH 02/17] net/dpaa: fix typecasting ch ID to u32 Hemant Agrawal
2024-08-07 15:37   ` Ferruh Yigit
2024-08-01 10:52 ` [PATCH 03/17] bus/dpaa: fix VSP for 1G fm1-mac9 and 10 Hemant Agrawal
2024-08-07 15:38   ` Ferruh Yigit
2024-08-23  7:33     ` Hemant Agrawal
2024-08-01 10:53 ` [PATCH 04/17] bus/dpaa: add port buffer manager stats Hemant Agrawal
2024-08-07 15:38   ` Ferruh Yigit
2024-08-23  7:33     ` Hemant Agrawal
2024-08-01 10:53 ` [PATCH 05/17] net/dpaa: support Tx confirmation to enable PTP Hemant Agrawal
2024-08-07 15:38   ` Ferruh Yigit
2024-08-23  7:34     ` Hemant Agrawal
2024-08-01 10:53 ` [PATCH 06/17] net/dpaa: add support to separate Tx conf queues Hemant Agrawal
2024-08-01 10:53 ` [PATCH 07/17] net/dpaa: share MAC FMC scheme and CC parse Hemant Agrawal
2024-08-07 15:39   ` Ferruh Yigit
2024-08-23  7:34     ` Hemant Agrawal
2024-08-01 10:53 ` [PATCH 08/17] net/dpaa: support Rx/Tx timestamp read Hemant Agrawal
2024-08-01 10:53 ` [PATCH 09/17] net/dpaa: support IEEE 1588 PTP Hemant Agrawal
2024-08-07 15:39   ` Ferruh Yigit
2024-08-23  7:36     ` Hemant Agrawal
2024-08-01 10:53 ` [PATCH 10/17] net/dpaa: implement detailed packet parsing Hemant Agrawal
2024-08-07 15:39   ` Ferruh Yigit
2024-08-23  7:34     ` Hemant Agrawal
2024-08-01 10:53 ` [PATCH 11/17] net/dpaa: enhance DPAA frame display Hemant Agrawal
2024-08-07 15:39   ` Ferruh Yigit
2024-08-23  7:36     ` Hemant Agrawal
2024-08-01 10:53 ` [PATCH 12/17] net/dpaa: support mempool debug Hemant Agrawal
2024-08-01 10:53 ` [PATCH 13/17] net/dpaa: add Tx rate limiting DPAA PMD API Hemant Agrawal
2024-08-07 15:40   ` Ferruh Yigit
2024-08-23  7:35     ` Hemant Agrawal
2024-08-01 10:53 ` [PATCH 14/17] bus/dpaa: add OH port mode for dpaa eth Hemant Agrawal
2024-08-07 15:40   ` Ferruh Yigit
2024-08-23  7:35     ` Hemant Agrawal
2024-08-01 10:53 ` [PATCH 15/17] bus/dpaa: add ONIC port mode for the DPAA eth Hemant Agrawal
2024-08-01 10:53 ` [PATCH 16/17] net/dpaa: improve the dpaa port cleanup Hemant Agrawal
2024-08-01 10:53 ` [PATCH 17/17] net/dpaa: improve dpaa errata A010022 handling Hemant Agrawal
2024-08-07 15:41   ` Ferruh Yigit
2024-08-23  7:35     ` Hemant Agrawal
2024-08-07 15:42 ` [PATCH 00/17] NXP DPAA ETH driver enhancement and fixes Ferruh Yigit
2024-08-08  8:51   ` Hemant Agrawal
2024-08-23  7:32 ` [PATCH v2 00/18] " Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 01/18] bus/dpaa: fix PFDRs leaks due to FQRNIs Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 02/18] net/dpaa: fix typecasting ch ID to u32 Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 03/18] bus/dpaa: fix VSP for 1G fm1-mac9 and 10 Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 04/18] bus/dpaa: fix the fman details status Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 05/18] bus/dpaa: add port buffer manager stats Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 06/18] net/dpaa: support Tx confirmation to enable PTP Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 07/18] net/dpaa: add support to separate Tx conf queues Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 08/18] net/dpaa: share MAC FMC scheme and CC parse Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 09/18] net/dpaa: support Rx/Tx timestamp read Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 10/18] net/dpaa: support IEEE 1588 PTP Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 11/18] net/dpaa: implement detailed packet parsing Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 12/18] net/dpaa: enhance DPAA frame display Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 13/18] net/dpaa: support mempool debug Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 14/18] net/dpaa: add Tx rate limiting DPAA PMD API Hemant Agrawal
2024-09-22  3:14     ` Ferruh Yigit
2024-09-22  4:40       ` Hemant Agrawal
2024-09-22 13:10         ` Ferruh Yigit
2024-09-22 13:27         ` Ferruh Yigit
2024-09-22 15:23           ` Ferruh Yigit
2024-08-23  7:32   ` [PATCH v2 15/18] bus/dpaa: add OH port mode for dpaa eth Hemant Agrawal
2024-09-22 15:24     ` Ferruh Yigit
2024-08-23  7:32   ` [PATCH v2 16/18] bus/dpaa: add ONIC port mode for the DPAA eth Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 17/18] net/dpaa: improve the dpaa port cleanup Hemant Agrawal
2024-08-23  7:32   ` [PATCH v2 18/18] net/dpaa: improve dpaa errata A010022 handling Hemant Agrawal
2024-09-22  3:12   ` [PATCH v2 00/18] NXP DPAA ETH driver enhancement and fixes Ferruh Yigit
2024-09-22  4:38     ` Hemant Agrawal
2024-09-30 10:29   ` [PATCH v3 " Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 01/18] bus/dpaa: fix PFDRs leaks due to FQRNIs Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 02/18] net/dpaa: fix typecasting ch ID to u32 Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 03/18] bus/dpaa: fix VSP for 1G fm1-mac9 and 10 Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 04/18] bus/dpaa: fix the fman details status Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 05/18] bus/dpaa: add port buffer manager stats Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 06/18] net/dpaa: support Tx confirmation to enable PTP Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 07/18] net/dpaa: add support to separate Tx conf queues Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 08/18] net/dpaa: share MAC FMC scheme and CC parse Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 09/18] net/dpaa: support Rx/Tx timestamp read Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 10/18] net/dpaa: support IEEE 1588 PTP Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 11/18] net/dpaa: implement detailed packet parsing Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 12/18] net/dpaa: enhance DPAA frame display Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 13/18] net/dpaa: support mempool debug Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 14/18] bus/dpaa: add OH port mode for dpaa eth Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 15/18] bus/dpaa: add ONIC port mode for the DPAA eth Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 16/18] net/dpaa: improve the dpaa port cleanup Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 17/18] net/dpaa: improve dpaa errata A010022 handling Hemant Agrawal
2024-09-30 10:29     ` [PATCH v3 18/18] net/dpaa: fix reallocate_mbuf handling Hemant Agrawal
2024-09-30 12:15   ` [PATCH v4 00/18] NXP DPAA ETH driver enhancement and fixes Hemant Agrawal
2024-09-30 12:15     ` Hemant Agrawal [this message]
2024-09-30 12:15     ` [PATCH v4 02/18] net/dpaa: fix typecasting ch ID to u32 Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 03/18] bus/dpaa: fix VSP for 1G fm1-mac9 and 10 Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 04/18] bus/dpaa: fix the fman details status Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 05/18] bus/dpaa: add port buffer manager stats Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 06/18] net/dpaa: support Tx confirmation to enable PTP Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 07/18] net/dpaa: add support to separate Tx conf queues Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 08/18] net/dpaa: share MAC FMC scheme and CC parse Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 09/18] net/dpaa: support Rx/Tx timestamp read Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 10/18] net/dpaa: support IEEE 1588 PTP Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 11/18] net/dpaa: implement detailed packet parsing Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 12/18] net/dpaa: enhance DPAA frame display Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 13/18] net/dpaa: support mempool debug Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 14/18] bus/dpaa: add OH port mode for dpaa eth Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 15/18] bus/dpaa: add ONIC port mode for the DPAA eth Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 16/18] net/dpaa: improve the dpaa port cleanup Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 17/18] net/dpaa: improve dpaa errata A010022 handling Hemant Agrawal
2024-09-30 12:15     ` [PATCH v4 18/18] net/dpaa: fix reallocate_mbuf handling Hemant Agrawal
2024-10-01  8:15     ` [PATCH v4 00/18] NXP DPAA ETH driver enhancement and fixes Ferruh Yigit
2024-10-01 11:03     ` [PATCH v5 " Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 01/18] bus/dpaa: fix PFDRs leaks due to FQRNIs Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 02/18] net/dpaa: fix typecasting ch ID to u32 Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 03/18] bus/dpaa: fix VSP for 1G fm1-mac9 and 10 Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 04/18] bus/dpaa: fix the fman details status Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 05/18] bus/dpaa: add port buffer manager stats Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 06/18] net/dpaa: support Tx confirmation to enable PTP Hemant Agrawal
2024-10-04 14:01         ` David Marchand
2024-10-01 11:03       ` [PATCH v5 07/18] net/dpaa: add support to separate Tx conf queues Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 08/18] net/dpaa: share MAC FMC scheme and CC parse Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 09/18] net/dpaa: support Rx/Tx timestamp read Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 10/18] net/dpaa: support IEEE 1588 PTP Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 11/18] net/dpaa: implement detailed packet parsing Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 12/18] net/dpaa: enhance DPAA frame display Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 13/18] net/dpaa: support mempool debug Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 14/18] bus/dpaa: add OH port mode for dpaa eth Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 15/18] bus/dpaa: add ONIC port mode for the DPAA eth Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 16/18] net/dpaa: improve the dpaa port cleanup Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 17/18] net/dpaa: improve dpaa errata A010022 handling Hemant Agrawal
2024-10-01 11:03       ` [PATCH v5 18/18] net/dpaa: fix reallocate_mbuf handling Hemant Agrawal
2024-10-02  0:41       ` [PATCH v5 00/18] NXP DPAA ETH driver enhancement and fixes Ferruh Yigit
2024-10-04 14:03       ` David Marchand

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240930121525.3452847-2-hemant.agrawal@nxp.com \
    --to=hemant.agrawal@nxp.com \
    --cc=dev@dpdk.org \
    --cc=ferruh.yigit@amd.com \
    --cc=g.singh@nxp.com \
    --cc=stable@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).