From: "Morten Brørup" <mb@smartsharesystems.com>
To: "Stephen Hemminger" <stephen@networkplumber.org>,
<bruce.richardson@intel.com>, <thomas@monjalon.net>
Cc: <dev@dpdk.org>, "Tyler Retzlaff" <roretzla@linux.microsoft.com>
Subject: RFC: default burst sizes in rte_config
Date: Thu, 9 Nov 2023 11:25:14 +0100 [thread overview]
Message-ID: <98CBD80474FA8B44BF855DF32C47DC35E9F002@smartserver.smartshare.dk> (raw)
In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35E9EFF7@smartserver.smartshare.dk>
> From: Morten Brørup [mailto:mb@smartsharesystems.com]
> Sent: Wednesday, 8 November 2023 18.49
>
> > From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> > Sent: Wednesday, 8 November 2023 17.52
> >
[...]
> >
> > Would it make sense to have an rte_config.h value for maximum burst
> > size?
>
> I would support that!
It would also be a good place to document the reasoning behind the choice of burst size, so application developers can better understand how to fine tune the values according to available hardware and application specific requirements.
Those build-time configurable values should also be used by DPDK libraries, instead of more or less randomly chosen hardcoded burst sizes.
E.g. when I implemented rte_pktmbuf_free_bulk(), I considered 64 plenty of burst capacity, because it was double the size of the traditional burst size of 32. But it is probably sub-optimal for applications using a default burst size of 128.
> There could be a few burst size defines, e.g.
>
> - SMALL: used for small bursts (I think some drivers use bursts of 8)
The reason for choosing 8 is probably rooted in cache alignment:
Eight 64-bit pointers covers one cache line.
I wonder if those drivers would perform better using bursts of 16 mbufs on 32-bit architectures, or on 64-bit architectures with 128 B cache line size?
> - NORMAL: used for typical bursts
This is usually a balance between latency and throughput:
Using shorter bursts can reduce the latency (if the application is designed with this in mind).
Using larger bursts improves processing performance, and thus increases throughput.
There is also some upper limit:
If the burst is too large, the amount of memory touched by a pipeline stage might not fit into the CPU data cache size, and performance drops like a rock.
E.g. a CPU with 64 B cache line size and 32 KB L1 data cache per lcore can hold 512 cache lines in its L1 data cache, so a burst of 32 mbufs allows touching an average of 512/32 = 16 cache lines per packet.
The mbuf structure itself uses 2 cache lines, so the max theoretical burst would be 512/2 = 256 if no other memory was touched.
However, the array holding the mbuf pointers is also touched, so I would put 128 as the largest good burst size on such a CPU.
> - LARGE: used for large bursts, e.g. mempool cache flush
If kept at 512, like the magnitude of the mempool cache flushes/refills, it should only be used for moving mbuf pointers around, without touching the mbufs themselves, or the CPU's L1 data cache will overflow.
>
> Having these available at build time would also allow more
> optimizations in DPDK libs and drivers for those specific burst sizes.
next prev parent reply other threads:[~2023-11-09 10:25 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-11-07 19:32 RFC acceptable handling of VLAs across toolchains Tyler Retzlaff
2023-11-08 2:31 ` Stephen Hemminger
2023-11-08 3:25 ` Tyler Retzlaff
2023-11-08 8:19 ` Morten Brørup
2023-11-08 16:51 ` Stephen Hemminger
2023-11-08 17:48 ` Morten Brørup
2023-11-09 10:25 ` Morten Brørup [this message]
2023-11-09 20:26 ` Tyler Retzlaff
2024-03-21 0:12 ` Tyler Retzlaff
2024-04-04 17:15 ` [PATCH 0/4] RFC samples converting VLA to alloca Tyler Retzlaff
2024-04-04 17:15 ` [PATCH 1/4] latencystats: use alloca instead of vla trivial Tyler Retzlaff
2024-04-06 15:28 ` Morten Brørup
2024-04-07 9:36 ` Mattias Rönnblom
2024-04-07 17:00 ` Stephen Hemminger
2024-04-04 17:15 ` [PATCH 2/4] hash: " Tyler Retzlaff
2024-04-06 16:01 ` Morten Brørup
2024-04-04 17:15 ` [PATCH 3/4] vhost: use alloca instead of vla sizeof Tyler Retzlaff
2024-04-06 22:30 ` Morten Brørup
2024-04-04 17:15 ` [PATCH 4/4] dispatcher: use alloca instead of vla multi dimensional Tyler Retzlaff
2024-04-06 15:49 ` Morten Brørup
2024-04-07 9:31 ` [PATCH 0/4] RFC samples converting VLA to alloca Mattias Rönnblom
2024-04-07 11:07 ` Morten Brørup
2024-04-07 17:03 ` Stephen Hemminger
2024-04-08 15:27 ` Tyler Retzlaff
2024-04-08 15:53 ` Morten Brørup
2024-04-09 8:28 ` Konstantin Ananyev
2024-04-09 15:08 ` Tyler Retzlaff
2024-04-10 9:58 ` Konstantin Ananyev
2024-04-10 17:03 ` Tyler Retzlaff
2024-04-10 7:32 ` Mattias Rönnblom
2024-04-10 7:52 ` Morten Brørup
2024-04-10 17:04 ` Tyler Retzlaff
2024-04-10 7:27 ` Mattias Rönnblom
2024-04-10 17:10 ` Tyler Retzlaff
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=98CBD80474FA8B44BF855DF32C47DC35E9F002@smartserver.smartshare.dk \
--to=mb@smartsharesystems.com \
--cc=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=roretzla@linux.microsoft.com \
--cc=stephen@networkplumber.org \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).