Re: [dpdk-dev] Beyond DPDK 2.0

[Jay Rolette <rolette@infiniteio.com>]

On Tue, Apr 28, 2015 at 12:26 PM, Neil Horman <nhorman@tuxdriver.com> wrote:

> On Mon, Apr 27, 2015 at 08:46:01AM -0500, Jay Rolette wrote:
> > On Sat, Apr 25, 2015 at 7:10 AM, Neil Horman <nhorman@tuxdriver.com>
> wrote:
> >
> > > On Fri, Apr 24, 2015 at 02:55:33PM -0500, Jay Rolette wrote:
> > > > On Fri, Apr 24, 2015 at 1:51 PM, Neil Horman <nhorman@tuxdriver.com>
> > > wrote:
> > > >
> > > > > So, I hear your arguments, and its understandable that you might
> not
> > > want
> > > > > a GPL
> > > > > licensed product, given that the DPDK is a library (though I'm not
> sure
> > > > > what the
> > > > > aversion to LGPL would be).  Regardless, I think this conversation
> is a
> > > > > bit more
> > > > > about participation than license choice.  While you are correct, in
> > > that
> > > > > the
> > > > > first step to support (by which I presume you mean participation
> in the
> > > > > community) is use, the goal here is to get people contributing
> patches
> > > and
> > > > > helping increase the usefulness of DPDK.
> > > >
> > > >
> > > > > Given that DPDK is primarily licensed as BSD now, whats preventing
> > > you, or
> > > > > what
> > > > > would encourage you to participate in the community?  I see emails
> from
> > > > > infiniteio addresss in the archives asking questions and making
> > > > > suggestions on
> > > > > occasion, but no patches.  What would get you (or others in a
> simmilar
> > > > > situation) to submit those?
> > > > >
> > > >
> > > > 36 hours in the day? :)
> > > >
> > > > It's not a lot, but we've submitted a couple of small patches. It's
> > > mostly
> > > > a matter of opportunity. We submit patches as we come across DPDK
> bugs or
> > > > find useful optos.
> > > >
> > >
> > <snip examples>
> >
> > >
> > > Understand, I'm not trying to single you out here.  I see that you, and
> > > others
> > > from infiniteio have had some participation in the project, and thats
> > > great, and
> > > appreciated. I'm more focused on why that level of participation is not
> > > higher
> > > (not only from you, but from the larger presumed user base in
> general).  As
> > > noted at the start of this thread, one of the goals of this discussion
> is
> > > to
> > > find ways to maximize participation in the project, and from you I'm
> > > hearing
> > > that:
> > >
> > > 1) you use the dpdk because it lowers maintenence overhead
> > > 2) you don't participate more in the project because your product work
> > > schedule
> > > doesn't allow you to do so.
> > >
> > > Are those both accurate statements?
> > >
> >
> > (1) was just my response to Luke about what would motivate a company to
> > submit patches if the license didn't require it (GPL vs BSD discussion).
> > Maint overhead had little to do with why we decided to use DPDK.
> >
> > (2) is certainly true enough, but not really all that big of a factor in
> > the reasons why our participation isn't at some higher level. I'd say
> there
> > are two primary factors:
> >
> > *New Contributors*
> > Dropping a major patch on a project where we have no history would be
> > foolish and frustrating. Every open source project has a vibe to it and
> its
> > own way of operating. It normally takes some time to figure that out, but
> > for major contributions, it generally involves a level of trust.
> >
> > Major code drops or patches aren't generally well received unless it is
> > from someone that is known and trusted by the community. Building up that
> > trust ("street cred") normally involves participating in smaller, less
> > risky ways. Answering questions where you can, small patches to fix bugs,
> > possibly reviewing code (although this can be iffy as well), etc.
> >
> > This facilitates understanding the process, who the players are and what
> > sort of toes you are likely to step on.
> >
> > It also gives you time to ramp up on the internals of the code and
> > philosophy/goals of the project better. With DPDK, performance is
> obviously
> > more important than portability. Until recently, very little care was
> given
> > to API stability or the impact that has on DPDK apps. Both of those are
> > very different approaches than typical and it affects what you might do
> > when approaching submitting a patch.
> >
> > If you want to build up the number of folks contributing, expect them to
> > start small and make sure it actually GOES somewhere.
> >
> > The first patch I submitted in mid-December had some quick responses and
> > questions back-and-forth, but then stalled on a couple of undocumented
> > minor stylistic things (comment style and use of #ifdefs). I never heard
> > anything back and 4.5 months later, a simple startup opto is still
> sitting
> > there.
> >
> > The second patch I submitted (also mid-December) got no response at all
> for
> > 4 months. I've replied to the feedback that came eventually, but once
> > again, no subsequent answers.
> >
> > Neither of the patches were important, but the process doesn't exactly
> > inspire a strong desire to contribute more.
> >
> > *Different Goals*
> > I see at least two different sets of people on the mailing list. The
> heavy
> > hitters generally have a product reason for their high level of
> involvement
> > with DPDK. For Intel and 6Wind, the reasons are pretty obvious. Same deal
> > with NIC vendors. For large companies, sometimes the reasons are less
> > obvious, but supporting certain open source projects can be strategic for
> > them for several reasons.
> >
> > For this group, improving DPDK itself is important enough to dedicate
> > resources to. It's a goal in and of itself, even if it isn't the main
> goal.
> >
> > The other group are what I'd call "DPDK users". They picked DPDK because
> it
> > fit a need they had in their product/project, just like they might pick
> > Linux or FreeBSD, Apache, gcc and any number of libraries.
> >
> > I'm in that second group. DPDK has been tremendously valuable to me, so
> I'm
> > happy to contribute back where I can, but it isn't my goal. I've got a
> > product to build and DPDK is another piece of the puzzle. The same is
> true
> > of Linux and all the various libraries we use.
> >
> > The nature of the contributions from those two groups are going to be
> > different.
>
> So, thank you for your feedback, I think this is tremendously valuable.
> What I
> hear you saying here is that while you like the DPDK, you don't feel
> motivated
> to make any major contribution to it, because its not really your focus,
> is that
> about right?  Thinking about that, I would say, thats actually great.  Most
> successful projects thrive off of a healthy community of contributors like
> you.
> While we always need people pushing the future of the project forward, we
> just
> as importantly need people making sure all the corner cases are tested via
> regular usage, and any rough edges found during that testing is submitted
> to fix
> it.


Agreed


> To that end, let me ask you, do you think that most bugs that you find in
> the DPDK you fix, or do you just report them?  Or do you otherwise work
> around
> them?  If its the first answer great.  If its either of the other two, is
> there
> in your mind a way we can increase that conversion rate from finding the
> bug, to
> fixing the bug?
>

We either fix them or work-around them. I took a look at our DPDK
repository and, ignoring build integration commits, we've made 12 commits
to it so far. We are running DPDK 1.6.0r2. I break those down like this:

   - 6 commits to back-port fixes and enhancements from later versions of
   DPDK or patches that are pending still
   - 2 bug fixes/optos that we submitted patches for
   - 3 bug fixes that we haven't submitted patches for yet
   - 1 change definitely specific to our app

Of the 3 bug fixes we haven't submitted patches for:

   - fixed a buffer overflow in the IP reassembly code we back-ported from
   1.8. We need to submit a patch for this one.
   - one eliminates a bunch of "irq 0x11" spam in igb_uio when running in
   VMware Fusion. From what I've seen, it doesn't look like DPDK takes patches
   for these.
   - fixed rte_mempool_from_obj() to return non-const to fit with the rest
   of the rte_mempool APIs

So not a lot...

On the work-around side of things, it's mostly around port management and
KNI. There are also parts of the DPDK library that we don't bother using.
Minimally, they aren't suitable for our app. I suspect that is more broadly
true than just our case, but you never know. In those cases, we end up
rolling our own.

Sorry, that's a long way of saying that I think we primarily fix the
problems we run into if someone else hasn't already, but I always like
having data :)

> Can we also assume, for the sake of discussion that you have encountered
> > > problems, or desired additions to the DPDK, for which you have
> implemented
> > > your
> > > own code in the library that is not contributed back to the DPDK?
> > >
> >
> > Yes, there are problems we've hit with DPDK that we haven't fed back the
> > fixes for. We generally do, but sometimes it isn't clear that what we are
> > doing would be considered a general problem. Sometimes we fix things with
> > hacks that work for us, but aren't necessarily a good "real" fix that
> > should go into DPDK.
> >
>
> So this, this is the very thing I would like to address.  If you were ever
> curious, or wondered, let me be clear: If you find a bug, we want to see
> the
> fix you have for it.  Even if its a hack, getting the report and an
> initial fix
> gives us data and a starting point to develop a real fix for the issue.
> Thats
> mututally beneficial.  You get a more robust, maintained fix, and so does
> everyone else.  Plus you get increased recognition as a participant/leader
> in
> the community.  Always, we always want your code, please never be shy about
> sharing it.
>

Thanks. That's very helpful to know. Normally I'm unlikely to submit a
patch unless I know it's rock solid - especially when I'm a relative
newcomer. Might be useful to add something to that effect to the dpdk.org
website.


> > On the "desired additions" bit, it's less clear. Is it product specific
> or
> > something that is general purpose DPDK-wise? As a startup company, we
> don't
> > have the luxury yet of being able to spend the time required to refactor
> > subsystems/modules so it can be shared. Hopefully we'll succeed to the
> > point where those discussions are possible :)
> >
> This is an interesting thing to say.  I come from a background in which the
> conventional wisdom is that open source is the starting point on the path
> to
> success (in that it allows for leveraging of the minor bug fixes and
> maintenence
> from interested parties, much as you describe yourselves), while the core
> developers push larger features forward.  There also exists the equally
> prevalent philosophy you describe in which a company has to be successfull,
> prior to being able to 'afford' to open source a product so to speak.  I
> get the
> impression that your company belongs more to the latter group.


Neither really. I should probably clarify my statement a bit... I didn't
mean to imply that once we were successful that we'd look at open-sourcing
our product. I meant that for pieces and parts that aren't core IP, we
should have more flexibility to look at spending the extra effort it takes
to share larger patches (features and subsystems).


> Do you feel like
> that line of thinking is something that is bound to some immutable nature
> of
> your business, or do you feel like discussion, other other education might
> lead
> your, or other companies like yours to see open source as more of an
> investment?
>

The former is probably a good way to describe it ("immutable nature of your
business"). We are a product company, not a services company. That said,
I'm fairly open to discussions, especially over a cold beverage :)


> > In the meantime, we do provide feedback on RFCs that come across the
> > mailing list on things where we have experience. Most of that seems to
> just
> > go in the bit bucket unfortunately. There's not a lot of actual
> *discussion* /
> > back-and-forth on things here.
> >
> I completely agree here, more conversation the list about proposed new
> features
> would be helpful.
>
> > If that is true, perhaps part of this conversation needs to revolve
> around
> > > the
> > > tangible benefits of contributing that code back to the upstream
> project
> > > (the
> > > aforementioned reduction of overhead) as well as the intangible
> (thought
> > > leadership as the project develops).  Its been my experience, that
> these
> > > situations often arise because management at a company often places a
> > > strong
> > > bias on development of their product over participation in the open
> source
> > > portion of it, treating the latter as if they are a customer of it,
> rather
> > > than
> > > a participant in it, and it would be my desire to see that bias
> adjusted
> > > so as
> > > to allow you greater freedom to participate in the project.
> > >
> >
> > It should be clear from above that yes, we have a strong bias on
> > development of our product. We are a startup with a very finite runway.
> > Just the reality that we live in...
> >
> Fair enough.
>
> > There are no executives or senior management telling us we can't
> contribute
> > more, so that isn't the case for us. However, it certainly could be at
> > other companies.
> >
> Agreed, I've worked for several networking startups who all had the
> mentality
> that open source was a 'freebie', only as good as what you can take from
> it, and
> to which we should never give back, for fear of providing others with
> competitive advantage.  I'm glad thats not you, though I'm a bit concerned
> that
> there might be others lurking here for whoom it is true, and I would love
> to
> find a way to reach them.
>

That's tough to get past, especially with networking equipment vendors. I
think the effort you've personally made to push for API stability has the
potential to help significantly.

DPDK has been notoriously bad about changing interfaces and one side-effect
of that is that companies building apps on top of DPDK end up "stuck" at
whatever version they started with. It's just too expensive to rewrite and
retest everything.

In that sort of environment, it really does remove the incentive to push
patches back upstream. They just stick with whatever ancient version they
have and hack on that because they have been left behind. FWIW, that isn't
just theory-craft. I've seen it in action with DPDK at a former employer.

Outside of that, make it as easy as you can for them to contribute patches.


> > One thing I'd describe differently as a "DPDK user" is the notion of
> > treating it as a customer. Treating it as a customer implies there are
> > expectations of (or support contracts for) things being fixed, features
> > being added, etc.
> >
>
> Right, bad use of language on my part.  By customer all I meant was that
> some
> users see it as a product for which they have paid $0.  When its broken
> they
> would typically contact their vendor, but since they have none, they
> understand
> that they 'got what they paid for' and work around the various issues.  I
> only
> meant to imply that a shift from that mentality, that of an 'owner', or
> some
> other entitiy that had a reason to help improve the DPDK.
>
> > I believe we treat it like we do any other open source project. If we run
> > into problems, we search for answers and if we don't find anything, we
> ask
> > the community in case someone else can help. When others ask for help, we
> > help when we can. Our ability to do so grows over time.
> >
> > Would you agree to that assessment?  If so, how would you suggest that
> we,
> > > as a
> > > community address this, and illustrate the appeal of contribution and
> > > participation to your company and the benefits thereof?
> > >
> >
> > Hopefully my comments above address your questions and what I see as the
> > reasons.
> >
> They have, thank you for your thoughts!
> Neil
>

Cheers,
Jay