[dpdk-dev] Weird 2 KB MBUF data room requirement

[dpdk-dev] Weird 2 KB MBUF data room requirement

[Morten Brørup]

Dear Ethernet PMD developers,

According to rte_mbuf_core.h, RTE_MBUF_DEFAULT_DATAROOM is 2048 bytes because some NICs need at least 2 KB buffer to receive standard Ethernet frames without splitting them into multiple segments.

This is a serious waste of memory, considering that standard Ethernet frames are max 1518 bytes.

How wide spread is this limitation... is it common or a rare exception?

Where is it documented which NICs suffer from this limitation?

Do any Intel NICs suffer from this limitation?


NB: We are targeting an MBUF total size (incl. memzone element overhead) of 2^N, and this limitation would increase our MBUF total size to 4 KB.


Med venlig hilsen / kind regards
- Morten Brørup

Re: [dpdk-dev] Weird 2 KB MBUF data room requirement

[Bruce Richardson]

On Fri, Jul 10, 2020 at 10:21:40AM +0200, Morten Brørup wrote:
> Dear Ethernet PMD developers,
> 
> According to rte_mbuf_core.h, RTE_MBUF_DEFAULT_DATAROOM is 2048 bytes because some NICs need at least 2 KB buffer to receive standard Ethernet frames without splitting them into multiple segments.
> 
> This is a serious waste of memory, considering that standard Ethernet frames are max 1518 bytes.
> 
> How wide spread is this limitation... is it common or a rare exception?
> 
> Where is it documented which NICs suffer from this limitation?
> 
> Do any Intel NICs suffer from this limitation?
> 
> 
> NB: We are targeting an MBUF total size (incl. memzone element overhead) of 2^N, and this limitation would increase our MBUF total size to 4 KB.
> 
> 
> Med venlig hilsen / kind regards
> - Morten Brørup
> 

AFAIK: the NICs supported by the ixgbe driver only allow the size to be
specified in KB granularity.

However, it may be safe to have a driver modification whereby anything over
1600 bytes is considered as 2KB if jumbo frame support is disabled. I don't
think anyone has actually looked into doing so though, or if there are
other hidden gotchas about attempting to do so.

/Bruce

Re: [dpdk-dev] Weird 2 KB MBUF data room requirement

[Olivier Matz]

Hi,

On Fri, Jul 10, 2020 at 11:26:09AM +0100, Bruce Richardson wrote:
> On Fri, Jul 10, 2020 at 10:21:40AM +0200, Morten Brørup wrote:
> > Dear Ethernet PMD developers,
> > 
> > According to rte_mbuf_core.h, RTE_MBUF_DEFAULT_DATAROOM is 2048 bytes because some NICs need at least 2 KB buffer to receive standard Ethernet frames without splitting them into multiple segments.
> > 
> > This is a serious waste of memory, considering that standard Ethernet frames are max 1518 bytes.
> > 
> > How wide spread is this limitation... is it common or a rare exception?
> > 
> > Where is it documented which NICs suffer from this limitation?
> > 
> > Do any Intel NICs suffer from this limitation?
> > 
> > 
> > NB: We are targeting an MBUF total size (incl. memzone element overhead) of 2^N, and this limitation would increase our MBUF total size to 4 KB.
> > 
> > 
> > Med venlig hilsen / kind regards
> > - Morten Brørup
> > 
> 
> AFAIK: the NICs supported by the ixgbe driver only allow the size to be
> specified in KB granularity.
> 
> However, it may be safe to have a driver modification whereby anything over
> 1600 bytes is considered as 2KB if jumbo frame support is disabled. I don't
> think anyone has actually looked into doing so though, or if there are
> other hidden gotchas about attempting to do so.

If I remember well, the niantic NICs (and probably some others) can
have their rx size configured with 512, 1024, 2048, ...
This is the size that should be available from the given data pointer,
i.e. it does not include the headroom.

I suppose that if we configure the NIC with 2K but give less than 2K, the NIC
may write after the buffer when receiving a large packet.


Olivier

Re: [dpdk-dev] Weird 2 KB MBUF data room requirement

[Morten Brørup]

> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Olivier Matz
> Sent: Friday, July 10, 2020 2:58 PM
> 
> Hi,
> 
> On Fri, Jul 10, 2020 at 11:26:09AM +0100, Bruce Richardson wrote:
> > On Fri, Jul 10, 2020 at 10:21:40AM +0200, Morten Brørup wrote:
> > > Dear Ethernet PMD developers,
> > >
> > > According to rte_mbuf_core.h, RTE_MBUF_DEFAULT_DATAROOM is 2048
> bytes because some NICs need at least 2 KB buffer to receive standard
> Ethernet frames without splitting them into multiple segments.
> > >
> > > This is a serious waste of memory, considering that standard
> Ethernet frames are max 1518 bytes.
> > >
> > > How wide spread is this limitation... is it common or a rare
> exception?
> > >
> > > Where is it documented which NICs suffer from this limitation?
> > >
> > > Do any Intel NICs suffer from this limitation?
> > >
> > >
> > > NB: We are targeting an MBUF total size (incl. memzone element
> overhead) of 2^N, and this limitation would increase our MBUF total
> size to 4 KB.
> > >
> > >
> > > Med venlig hilsen / kind regards
> > > - Morten Brørup
> > >
> >
> > AFAIK: the NICs supported by the ixgbe driver only allow the size to
> be
> > specified in KB granularity.
> >
> > However, it may be safe to have a driver modification whereby
> anything over
> > 1600 bytes is considered as 2KB if jumbo frame support is disabled. I
> don't
> > think anyone has actually looked into doing so though, or if there
> are
> > other hidden gotchas about attempting to do so.
> 
> If I remember well, the niantic NICs (and probably some others) can
> have their rx size configured with 512, 1024, 2048, ...
> This is the size that should be available from the given data pointer,
> i.e. it does not include the headroom.
> 
> I suppose that if we configure the NIC with 2K but give less than 2K,
> the NIC
> may write after the buffer when receiving a large packet.
> 
> 
> Olivier

Thanks for the quick and detailed replies, Bruce and Olivier.

In conclusion, this NIC limitation seems more likely than unlikely.

And a workaround is risky business.

We will abandon the idea of getting an 2^N total size MBUF. It's not a big deal.