Re: [dpdk-dev] Beyond DPDK 2.0

[Neil Horman <nhorman@tuxdriver.com>]

On Sat, Apr 25, 2015 at 04:08:23PM +0000, Wiles, Keith wrote:
> 
> 
> On 4/25/15, 8:30 AM, "Marc Sune" <marc.sune@bisdn.de> wrote:
> 
> >
> >
> >On 24/04/15 19:51, Matthew Hall wrote:
> >> On Fri, Apr 24, 2015 at 12:39:47PM -0500, Jay Rolette wrote:
> >>> I can tell you that if DPDK were GPL-based, my company wouldn't be
> >>>using
> >>> it. I suspect we wouldn't be the only ones...
> >>>
> >>> Jay
> >> I could second this, from the past employer where I used it. Right now
> >>I am
> >> using it in an open source app, I have a bit of GPL here and there but
> >>I'm
> >> trying to get rid of it or confine it to separate address spaces, where
> >>it
> >> won't impact the core code written around DPDK, as I don't want to cause
> >> headaches for any downstream users I attract someday.
> >>
> >> Hard-core GPL would not be possible for most. LGPL could be possible,
> >>but I
> >> don't think it could be worth the relicensing headache for that small
> >>change.
> >>
> >> Instead we should make the patch process as easy as humanly possible so
> >>people
> >> are encouraged to send us the fixes and not cart them around their
> >>companies
> >> constantly.
> 
> +1 and besides the GPL or LGPL ship has sailed IMHO and we can not go back.
Actually, IANAL, but I think we can.  The BSD license allows us to fork and
relicense the code I think, under GPL or any other license.  I'm not advocating
for that mind you, just suggesting that its possible should it ever become
needed.

> >
> >I agree. My feeling is that as the number of patches in the mailing list
> >grows, keeping track of them gets more and more complicated. Patchwork
> >website was a way to try to address this issue. I think it was an
> >improvement, but to be honest, patchwork lacks a lot of functionality,
> >such as properly tracking multiple versions of the patch (superseding
> >them automatically), and it lacks some filtering capabilities e.g. per
> >user, per tag/label or library, automatically track if it has been
> >merged, give an overall status of the pending vs merged patches, set
> >milestones... Is there any alternative tool or improved version for that?
> 
Agreed, this has come up before, off list unfortunately.  The volume of patches
seems to be increasing at such a rate that a single maintainer has difficulty
keeping up.  I proposed that the workload be split out to multiple subtrees,
with prefixes being added to patch subjects on the list for local filtering to
stem the tide.  Specifically I had proposed that the PMD's be split into a
separate subtree, but that received pushback in favor of having each library
having its own separate subtree, with a pilot program being made out of the I40e
driver (which you might note sends pull requests to the list now).  I'd still
like to see all PMD's come under a single subtree, but thats likely an argument
for later.

That said, Do you think that this patch latency is really a contributor to low
project participation?  It definately a problem, but it seems to me that this
sort of issue would lead to people trying to parcitipate, then giving up (i.e.
we would see 1-2 emails from an individual, then not see them again).  I'd need
to look through the mailing list for such a pattern, but anecdotally I've not
seen that happen.  The problem you describe above is definately a problem, but
its one for those individuals who are participating, not for those who are
simply choosing not to.  And I think we need to address both.

> I agree patchwork has some limitation, but I think the biggest issue is
> keeping up with the patches. Getting patches introduced into the main line
> is very slow. A patch submitted today may not get applied for weeks or
> months, then when another person submits a patch he is starting to run a
> very high risk of having to redo that patch, because a pervious patch
> makes his fail weeks/months later. I would love to see a better tool then
> patchwork, but the biggest issue is we have a huge backlog of patches.
> Personally I am not sure how Thomas or any is able to keep up with the
> patches.
> 
This is absolutely a problem.  I'd like to think, more than a tool like
patchwork, a subtree organization to allow some modicum of parallel review and
integration would really be a benefit here.

> The other problem I see is how patches are agreed on to be included in the
> mainline. Today it is just an ACK or a NAK on the mailing list. Then I see
> what I think to be only a few people ACKing or NAKing patches. This
> process has a lot of problems from a patch being ignore for some reason or
> someone having negative feed back on very minor detail or no way to push a
> patch forward a single NAK or comment.
> 

So, this is an interesting issue in ideal meritocracies.  Currently is/should be
looking for ACKs/NAK/s from the individuals listed in the MAINTAINER files, and
those people should be the definitive subject matter experts on the code they
cover.  As such, I would agrue that they should be entitled to a modicum of
stylistic/trivial leeway.  That is to say, if they choose to block a patch
around a very minor detail, then between them changing their position, and the
patch author changing the code, the latter is likely the easier course of
action, especially if the author can't make an argument for their position.
That said, if such patch blockage becomes so egregious that individuals stop
contributing, that needs to be known as well.  If you as a patch author:

1) Have tried to submit patches 
2) Had them blocked for what you consider trivial reasons
3) Plan to not contribute further because of this
4) Still rely on the DPDK for your product

Please, say something.  People in charge need to know when they're pushing
contributors away.

FWIW, I've tried to do some correlation between the git history and the mailing
list.  I need to do more searches, but I have a feeling that early on, the
majority of people who stopped contributing, did so because their patches
weren't expressely blocked, but rather because they were simply ignored.  No one
working on DPDK bothered to review those patches, and so they never got merged.
Hopefully that problem has been addressed somewhat now.

> I would like to see some type of layering process to allow patches to be
> applied in a timely manner a few weeks not months or completely ignored.
> Maybe some type of voting is reasonable, but we need to do something to
> turn around the patches in clean reasonable manner.
> 
> Think we need some type of group meeting every week to look at the patches
> and determining which ones get applied, this gives quick feedback to the
> submitter as to the status of the patch.
> 
I think a group meeting is going to be way too much overhead to manage properly.
You'll get different people every week with agenda that may not line up with
code quality, which is really what the review is meant to provide.  I think
perhaps a better approach would be to require that that code owners from the
maintainer file provide and ACK/NAK on their patches within 3-4 days, and
require a corresponding tree maintainer to apply the patch within 7 or so.  That
would cap our patch latency.  Likewise, if a patch slips in creating a
regression, the author needs to be alerted and given a time window in which to
fix the problem before the offending patch is reverted during the QE cycle.


> >
> >On the other side, since user questions, community discussions and
> >development happens in the same mailing list, things get really
> >complicated, specially for users seeking for help. Even though I think
> >the average skills of the users of DPDK is generally higher than in
> >other software projects, if DPDK wants to attract more users, having a
> >better user support is key, IMHO.
> >
> >So I would see with good eyes a separation between, at least, dpdk-user
> >and dpdk-dev.
> 
I wouldn't argue with this separation, seems like a reasonable approach.

> I do not remember seeing too many users on the list and making a list just
> for then is OK if everyone is fine with a list that has very few emails.
> >
> >If the number of patches keeps growing, splitting the "dev" mailing
> >lists into different categories (eal and common, pmds, higher level
> >abstractions...) could be an option. However, this last point opens a
> >lot of questions on how to minimize interference between the different
> >parts and API/ABI compatibility during the development.
> 
> I believe if we just make sure we use tags in the subject line then we can
> have our email clients do the splitting of the emails instead of adding
> more emails lists.
> 
Agreed

> >
> >>
> >> Perhaps it means having some ReviewBoard type of tools, a clone in
> >>Github or
> >> Bitbucket where the less hardcore kernel-workflow types could send back
> >>their
> >> small bug fixes a bit more easily, this kind of stuff. Google has been
> >>getting
> >> good uptake since they moved most of their open source across to Github,
> >> because the contribution workflow was more convenient than Google Code
> >>was.
> 
> I like GitHub it is a much better designed tool then patchwork, plus it
> could get more eyes as it is very well know to the developer community in
> general. I feel GitHub has many advantages over the current systems in
> place but, it does not solve the all patch issues.
> 
Github is actually a bit irritating for this sort of thing, as it presumes a web
based interface for discussion.  They have some modicum of email forwarding
enabled, but it has never quite worked right, or integrated properly.

> The only way we can get patch issues resolved is to put a bit more process
> in place.
> >
> >Although I agree, we have to be careful on how github or bitbucket is
> >used. Having issues or even (e.g. github) pull requests *in addition* to
> >the normal contribution workflow can be a nightmare to deal with, in
> >terms of synchronization and preventing double work. So I guess setting
> >up an official github or bitbucket mirror would be fine, via some simple
> >cronjob, but I guess it would end-up not using PRs or issues in github
> >like the Linux kernel does.
> 
100% agree, we can't be split about this.  Allowing contributions from n
channels just means most developers will only see/reviews 1/nth of the patches
of interest to them.

> From what I can tell GitHub seems to be a better solution for a free open
> environment. Bitbucket I have never used and GitHub seems more popular
> from one article I read.
> 
> https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#
> q=bitbucket%20vs%20github
> 
> 
> >Btw, is this github organization already registered by Intel or some
> >other company of the community?
> >
> >https://github.com/dpdk
> >
> >Marc
> 
> If we can used the above that would be great, but a name like
> Œdpdk-community¹ or something could work too.
> 
> We can host the web site here and have many sub-projects like Pktgen-DPDK
> :-) under the same page. Not to say anything bad about our current web
> pages as I find it difficult to use sometimes and find things like
> patchwork link. Maintaining a web site is a full time job and GitHub does
> maintain the site, plus we can collaborate on host web page on the GitHub
> site easier.
> 
> Moving to the Linux Foundation is an option as well as it is very well
> know and has some nice ways to get your project promoted. It does have a
> few drawbacks in process handling and cost to state a few. The process
> model is all ready defined, which is good and bad it just depends on your
> needs IMO.
> 
> Regards,
> ++Keith
> 
> >
> >>
> >> Matthew.
> >
> 
>