Re: [dpdk-dev] Reshuffling of rte_mbuf structure.

[Stephen Hemminger <stephen@networkplumber.org>]

On Sun, 1 Nov 2015 06:45:31 +0200
Arnon Warshavsky <arnon@qwilt.com> wrote:

> My 2 cents,
> 
> This was brought up in the recent user space summit, and it seems that
> indeed there is no one cache lines arrangement that fits all.
> OTOH multiple compile time options to suffice all flavors, would make it
> unpleasant to read maintain test and debug.
> (I think there was quiet a consensus in favor of reducing compile options
> in general)
> 
> Currently I manage similar deviations via our own source control which I
> admit to be quite a pain.
> I would prefer an option of code manipulation/generation by some script
> during dpdk install,
> which takes the default version of rte_mbuf.h,
> along with an optional user file (json,xml,elvish,whatever) defining the
> structure replacements,
> creating your custom version, and placing it instead of the installed copy
> of rte_mbuf.h.
> Maybe the only facility required from dpdk is just the ability to register
> calls to such user scripts at some install stage(s), providing the mean
> along with responsibility to the user.
> 
> /Arnon
> 
> 
> 
> On Sat, Oct 31, 2015 at 6:44 AM, shesha Sreenivasamurthy (shesha) <
> shesha@cisco.com> wrote:
> 
> > In Cisco, we are using DPDK for a very high speed packet processor
> > application. We don't use NIC TCP offload / RSS hashing. Putting those
> > fields in the first cache-line - and the obligatory mb->next datum in the
> > second cache line - causes significant LSU pressure and performance
> > degradation. If it does not affect other applications, I would like to
> > propose reshuffling of fields so that the obligator "next" field falls in
> > first cache line and RSS hashing goes to next. If this re-shuffling indeed
> > hurts other applications, another idea is to make it compile time
> > configurable. Please provide feedback.
> >
> > --
> > - Thanks
> > char * (*shesha) (uint64_t cache, uint8_t F00D)
> > { return 0x0000C0DE; }
> >

Having different layouts will be a disaster for distro's they have to choose one.
And I hate to introduce more configuration!

But we see the same issue. It would make sense if there were configuration options
for some common optimizations NO_TX_OFFLOAD, NO_MULTISEG, NO_REFCOUNT and then
the mbuf got optimized for those combinations. Seems better than config options
like LAYOUT1, LAYOUT2, ...

In this specific case, I think lots of driver could be check nb_segs == 1 and avoiding
the next field for simple packets.

Long term, I think this will be losing battle. As DPDK grows more features, the current
mbuf structure will grow there is really nothing stopping the bloat of meta data.