From: "Morten Brørup" <mb@smartsharesystems.com>
To: <olivier.matz@6wind.com>,
"Shijith Thotton" <sthotton@marvell.com>, <thomas@monjalon.net>,
<andrew.rybchenko@oktetlabs.ru>, <honnappa.nagarahalli@arm.com>,
<bruce.richardson@intel.com>
Cc: <dev@dpdk.org>
Subject: mbuf performance optimization
Date: Sat, 3 Dec 2022 18:13:00 +0100 [thread overview]
Message-ID: <98CBD80474FA8B44BF855DF32C47DC35D8754C@smartserver.smartshare.dk> (raw)
I have been playing around with the idea to make some changes to avoid using the mbuf's 2nd cache line in many common cases, which would reduce the cache pressure significantly, and thus improve performance. I would like to discuss if it is doable. (And let's just assume that ABI breakage is an acceptable tradeoff.)
Move 'tx_offload' to the 1st cache line
---------------------------------------
Under all circumstances:
We would need to move the 'tx_offload' field to the 1st cache line. This field is set by the application's packet forwarding pipeline stage, and read by the PMD TX function. In most cases, these two stages directly follow each other.
This also means that we must make room for it by moving a 64 bit field from the 1st to the 2nd cache line. It could be the 'next' or the 'pool' field, as discussed below.
The 'next' field - make it conditional
--------------------------------------
Optimization for (1) non-segmented packets:
We could avoid touching the 'next' field by making the 'next' field depend on something in the first cache line. E.g.:
- Use the 'ol_flags' field. Add a RTE_MBUF_F_MORE_SEGS flag, to be set/cleared when setting/clearing the 'next' field.
- Use the 'nb_segs' field. Set the 'nb_segs' field to a value >1 when setting the 'next' field, and set it to 1 when clearing the 'next' field.
The 'pool' field - use it less frequently
-----------------------------------------
Optimizations for (2) single-mempool TX queues and (3) single-mempool applications:
The 'pool' field seems to be only used by when a PMD frees a burst of mbufs that it has finished transmitting. Please correct me if I am wrong here.
We could introduce a sibling to RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE, with the only requirement that the mbufs come from the same mempool. When set, only the first mbuf in a burst gets its 'pool' field read, thus avoiding reading it in the remaining mbufs in the burst.
For single-mempool applications, we could introduce a global 'mbuf_pool' variable, to be used instead of the mbuf's 'pool' field, if set.
Med venlig hilsen / Kind regards,
-Morten Brørup
next reply other threads:[~2022-12-03 17:13 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-12-03 17:13 Morten Brørup [this message]
2022-12-04 10:10 ` [PATCH v2] mbuf perf test, please ignore Morten Brørup
2022-12-04 12:00 ` [PATCH v3] " Morten Brørup
2022-12-04 12:33 ` [PATCH v4] " Morten Brørup
2022-12-04 12:54 ` [PATCH v5] " Morten Brørup
2022-12-06 9:20 ` Morten Brørup
2022-12-06 11:03 ` [PATCH v6] " Morten Brørup
2022-12-06 12:12 ` [PATCH v7] " Morten Brørup
2022-12-06 12:16 ` [PATCH v8] " Morten Brørup
2022-12-06 12:35 ` [PATCH v9] " Morten Brørup
2022-12-06 13:53 ` [PATCH v10] " Morten Brørup
2022-12-07 7:35 ` [PATCH v11] " Morten Brørup
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=98CBD80474FA8B44BF855DF32C47DC35D8754C@smartserver.smartshare.dk \
--to=mb@smartsharesystems.com \
--cc=andrew.rybchenko@oktetlabs.ru \
--cc=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=honnappa.nagarahalli@arm.com \
--cc=olivier.matz@6wind.com \
--cc=sthotton@marvell.com \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).