From: "Morten Brørup" <mb@smartsharesystems.com>
To: dev@dpdk.org, Stephen Hemminger <stephen@networkplumber.org>,
Wathsala Vithanage <wathsala.vithanage@arm.com>,
Konstantin Ananyev <konstantin.ananyev@huawei.com>,
Chengwen Feng <fengchengwen@huawei.com>,
Bruce Richardson <bruce.richardson@intel.com>,
Thomas Monjalon <thomas@monjalon.net>
Cc: "Morten Brørup" <mb@smartsharesystems.com>
Subject: [PATCH v6] mbuf: optimize segment prefree
Date: Thu, 23 Oct 2025 12:48:58 +0000 [thread overview]
Message-ID: <20251023124858.318148-1-mb@smartsharesystems.com> (raw)
In-Reply-To: <20250827213535.21602-1-mb@smartsharesystems.com>
Refactored rte_pktmbuf_prefree_seg() for both performance and readability.
With the optimized RTE_MBUF_DIRECT() macro, the common likely code path
now fits within one instruction cache line on x86-64 when built with GCC.
Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>
Acked-by: Chengwen Feng <fengchengwen@huawei.com>
Reviewed-by: Bruce Richardson <bruce.richardson@intel.com>
---
v6:
* Fixed Doxygen error in RTE_MBUF_DIRECT() macro description. (Thomas Monjalon)
* Added specific RTE_MBUF_DIRECT() macro implementation for API documentation
purposes, using flag names instead of numerical value.
v5:
* Removed the plain RTE_MBUF_DIRECT() macro, and only use the optimized
variant. (Bruce Richardson)
Further testing on Godbolt confirmed that other compilers benefit from
the optimized macro too.
* Shortened the description of the RTE_MBUF_DIRECT() macro, and only
provide one example of code emitted by a compiler. (Bruce Richardson)
* Consolidated the static_assert() into one, covering both little and big
endian.
This also reduces the amount of endian-conditional source code and
improves readability.
(Bruce Richardson)
* Added comment about MSB meaning "most significant byte".
v4:
* Enabled the optimized RTE_MBUF_DIRECT() macro for GCC on all
architectures.
v3:
* Rewrote the optimized RTE_MBUF_DIRECT() macro for readability; use
numerical value instead of long names. (Bruce Richardson)
* Enabled the optimized RTE_MBUF_DIRECT() macro for Loongarch too.
v2:
* Fixed typo in commit description.
* Fixed indentation.
* Added detailed description to the optimized RTE_MBUF_DIRECT() macro.
(Stephen Hemminger)
* Added static_assert() to verify that the optimized RTE_MBUF_DIRECT()
macro is valid, specifically that the tested bits are in the MSB of the
64-bit field.
---
lib/mbuf/rte_mbuf.h | 51 +++++++++++++++-------------------------
lib/mbuf/rte_mbuf_core.h | 43 +++++++++++++++++++++++++++++++--
2 files changed, 60 insertions(+), 34 deletions(-)
diff --git a/lib/mbuf/rte_mbuf.h b/lib/mbuf/rte_mbuf.h
index 3df22125de..2004391f57 100644
--- a/lib/mbuf/rte_mbuf.h
+++ b/lib/mbuf/rte_mbuf.h
@@ -31,6 +31,7 @@
* http://www.kohala.com/start/tcpipiv2.html
*/
+#include <stdbool.h>
#include <stdint.h>
#include <rte_common.h>
@@ -1458,44 +1459,30 @@ static inline int __rte_pktmbuf_pinned_extbuf_decref(struct rte_mbuf *m)
static __rte_always_inline struct rte_mbuf *
rte_pktmbuf_prefree_seg(struct rte_mbuf *m)
{
- __rte_mbuf_sanity_check(m, 0);
-
- if (likely(rte_mbuf_refcnt_read(m) == 1)) {
-
- if (!RTE_MBUF_DIRECT(m)) {
- rte_pktmbuf_detach(m);
- if (RTE_MBUF_HAS_EXTBUF(m) &&
- RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
- __rte_pktmbuf_pinned_extbuf_decref(m))
- return NULL;
- }
-
- if (m->next != NULL)
- m->next = NULL;
- if (m->nb_segs != 1)
- m->nb_segs = 1;
+ bool refcnt_not_one;
- return m;
+ __rte_mbuf_sanity_check(m, 0);
- } else if (__rte_mbuf_refcnt_update(m, -1) == 0) {
+ refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1);
+ if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0)
+ return NULL;
- if (!RTE_MBUF_DIRECT(m)) {
- rte_pktmbuf_detach(m);
- if (RTE_MBUF_HAS_EXTBUF(m) &&
- RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
- __rte_pktmbuf_pinned_extbuf_decref(m))
- return NULL;
- }
+ if (unlikely(!RTE_MBUF_DIRECT(m))) {
+ rte_pktmbuf_detach(m);
+ if (RTE_MBUF_HAS_EXTBUF(m) &&
+ RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
+ __rte_pktmbuf_pinned_extbuf_decref(m))
+ return NULL;
+ }
- if (m->next != NULL)
- m->next = NULL;
- if (m->nb_segs != 1)
- m->nb_segs = 1;
+ if (refcnt_not_one)
rte_mbuf_refcnt_set(m, 1);
+ if (m->nb_segs != 1)
+ m->nb_segs = 1;
+ if (m->next != NULL)
+ m->next = NULL;
- return m;
- }
- return NULL;
+ return m;
}
/**
diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
index a0df265b5d..fb38eca575 100644
--- a/lib/mbuf/rte_mbuf_core.h
+++ b/lib/mbuf/rte_mbuf_core.h
@@ -711,9 +711,48 @@ struct rte_mbuf_ext_shared_info {
*
* If a mbuf embeds its own data after the rte_mbuf structure, this mbuf
* can be defined as a direct mbuf.
- */
+ *
+ * Note: Macro optimized for code size.
+ *
+ * The plain macro would be:
+ * \code{.c}
+ * #define RTE_MBUF_DIRECT(mb) \
+ * (!((mb)->ol_flags & (RTE_MBUF_F_INDIRECT | RTE_MBUF_F_EXTERNAL)))
+ * \endcode
+ *
+ * The flags RTE_MBUF_F_INDIRECT and RTE_MBUF_F_EXTERNAL are both in the MSB (most significant
+ * byte) of the 64-bit ol_flags field, so we only compare this one byte instead of all 64 bits.
+ *
+ * E.g., GCC version 16.0.0 20251019 (experimental) generates the following code for x86-64.
+ *
+ * With the plain macro, 17 bytes of instructions:
+ * \code
+ * movabs rax,0x6000000000000000 // 10 bytes
+ * and rax,QWORD PTR [rdi+0x18] // 4 bytes
+ * sete al // 3 bytes
+ * \endcode
+ * With this optimized macro, only 7 bytes of instructions:
+ * \code
+ * test BYTE PTR [rdi+0x1f],0x60 // 4 bytes
+ * sete al // 3 bytes
+ * \endcode
+ */
+#ifdef __DOXYGEN__
#define RTE_MBUF_DIRECT(mb) \
- (!((mb)->ol_flags & (RTE_MBUF_F_INDIRECT | RTE_MBUF_F_EXTERNAL)))
+ !(((const char *)(&(mb)->ol_flags))[MSB_OFFSET /* 7 or 0, depending on endianness */] & \
+ (char)((RTE_MBUF_F_INDIRECT | RTE_MBUF_F_EXTERNAL) >> (7 * CHAR_BIT)) /* 0x60 */)
+#else /* !__DOXYGEN__ */
+#if RTE_BYTE_ORDER == RTE_LITTLE_ENDIAN
+/* On little endian architecture, the MSB of a 64-bit integer is at byte offset 7. */
+#define RTE_MBUF_DIRECT(mb) !(((const char *)(&(mb)->ol_flags))[7] & 0x60)
+#elif RTE_BYTE_ORDER == RTE_BIG_ENDIAN
+/* On big endian architecture, the MSB of a 64-bit integer is at byte offset 0. */
+#define RTE_MBUF_DIRECT(mb) !(((const char *)(&(mb)->ol_flags))[0] & 0x60)
+#endif /* RTE_BYTE_ORDER */
+#endif /* !__DOXYGEN__ */
+/* Verify the optimization above. */
+static_assert((RTE_MBUF_F_INDIRECT | RTE_MBUF_F_EXTERNAL) == UINT64_C(0x60) << (7 * CHAR_BIT),
+ "(RTE_MBUF_F_INDIRECT | RTE_MBUF_F_EXTERNAL) is not 0x60 at MSB");
/** Uninitialized or unspecified port. */
#define RTE_MBUF_PORT_INVALID UINT16_MAX
--
2.43.0
next prev parent reply other threads:[~2025-10-23 12:49 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-08-27 21:35 [PATCH] " Morten Brørup
2025-08-27 23:17 ` Stephen Hemminger
2025-10-06 17:46 ` Wathsala Vithanage
2025-10-06 18:26 ` Morten Brørup
2025-10-06 14:49 ` Morten Brørup
2025-10-20 12:02 ` [PATCH v2] " Morten Brørup
2025-10-20 14:24 ` Konstantin Ananyev
2025-10-21 8:38 ` fengchengwen
2025-10-22 9:08 ` Bruce Richardson
2025-10-22 13:53 ` Morten Brørup
2025-10-22 14:12 ` Bruce Richardson
2025-10-22 14:14 ` Morten Brørup
2025-10-22 13:23 ` [PATCH v3] " Morten Brørup
2025-10-22 14:47 ` [PATCH v4] " Morten Brørup
2025-10-22 15:02 ` Bruce Richardson
2025-10-22 18:28 ` Morten Brørup
2025-10-23 7:04 ` Morten Brørup
2025-10-23 8:01 ` [PATCH v5] " Morten Brørup
2025-10-23 8:08 ` Bruce Richardson
2025-10-23 8:51 ` Konstantin Ananyev
2025-10-23 11:17 ` Morten Brørup
2025-10-23 14:04 ` Konstantin Ananyev
2025-10-23 14:48 ` Morten Brørup
2025-10-23 15:27 ` Konstantin Ananyev
2025-10-23 15:46 ` Morten Brørup
2025-10-23 16:03 ` Bruce Richardson
2025-10-23 16:24 ` Morten Brørup
2025-10-23 12:48 ` Morten Brørup [this message]
2025-10-23 17:13 ` [PATCH v6] " Thomas Monjalon
2025-10-23 16:18 ` [PATCH v7] " Morten Brørup
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20251023124858.318148-1-mb@smartsharesystems.com \
--to=mb@smartsharesystems.com \
--cc=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=fengchengwen@huawei.com \
--cc=konstantin.ananyev@huawei.com \
--cc=stephen@networkplumber.org \
--cc=thomas@monjalon.net \
--cc=wathsala.vithanage@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).