Re: [dpdk-dev] Beyond DPDK 2.0

["Wiles, Keith" <keith.wiles@intel.com>]

On 4/27/15, 5:29 AM, "Neil Horman" <nhorman@tuxdriver.com> wrote:

>On Mon, Apr 27, 2015 at 01:41:11AM +0000, Wiles, Keith wrote:
>> 
>> 
>> On 4/26/15, 4:56 PM, "Neil Horman" <nhorman@tuxdriver.com> wrote:
>> 
>> >On Sat, Apr 25, 2015 at 04:08:23PM +0000, Wiles, Keith wrote:
>> >> 
>> >> 
>> >> On 4/25/15, 8:30 AM, "Marc Sune" <marc.sune@bisdn.de> wrote:
>> >> 
>> >> >
>> >> >
>> >> >On 24/04/15 19:51, Matthew Hall wrote:
>> >> >> On Fri, Apr 24, 2015 at 12:39:47PM -0500, Jay Rolette wrote:
>> >> >>> I can tell you that if DPDK were GPL-based, my company wouldn't
>>be
>> >> >>>using
>> >> >>> it. I suspect we wouldn't be the only ones...
>> >> >>>
>> >> >>> Jay
>> >> >> I could second this, from the past employer where I used it. Right
>> >>now
>> >> >>I am
>> >> >> using it in an open source app, I have a bit of GPL here and there
>> >>but
>> >> >>I'm
>> >> >> trying to get rid of it or confine it to separate address spaces,
>> >>where
>> >> >>it
>> >> >> won't impact the core code written around DPDK, as I don't want to
>> >>cause
>> >> >> headaches for any downstream users I attract someday.
>> >> >>
>> >> >> Hard-core GPL would not be possible for most. LGPL could be
>>possible,
>> >> >>but I
>> >> >> don't think it could be worth the relicensing headache for that
>>small
>> >> >>change.
>> >> >>
>> >> >> Instead we should make the patch process as easy as humanly
>>possible
>> >>so
>> >> >>people
>> >> >> are encouraged to send us the fixes and not cart them around their
>> >> >>companies
>> >> >> constantly.
>> >> 
>> >> +1 and besides the GPL or LGPL ship has sailed IMHO and we can not go
>> >>back.
>> >Actually, IANAL, but I think we can.  The BSD license allows us to fork
>> >and
>> >relicense the code I think, under GPL or any other license.  I'm not
>> >advocating
>> >for that mind you, just suggesting that its possible should it ever
>>become
>> >needed.
>> >
>> >> >
>> >> >I agree. My feeling is that as the number of patches in the mailing
>> >>list
>> >> >grows, keeping track of them gets more and more complicated.
>>Patchwork
>> >> >website was a way to try to address this issue. I think it was an
>> >> >improvement, but to be honest, patchwork lacks a lot of
>>functionality,
>> >> >such as properly tracking multiple versions of the patch
>>(superseding
>> >> >them automatically), and it lacks some filtering capabilities e.g.
>>per
>> >> >user, per tag/label or library, automatically track if it has been
>> >> >merged, give an overall status of the pending vs merged patches, set
>> >> >milestones... Is there any alternative tool or improved version for
>> >>that?
>> >> 
>> >Agreed, this has come up before, off list unfortunately.  The volume of
>> >patches
>> >seems to be increasing at such a rate that a single maintainer has
>> >difficulty
>> >keeping up.  I proposed that the workload be split out to multiple
>> >subtrees,
>> >with prefixes being added to patch subjects on the list for local
>> >filtering to
>> >stem the tide.  Specifically I had proposed that the PMD's be split
>>into a
>> >separate subtree, but that received pushback in favor of having each
>> >library
>> >having its own separate subtree, with a pilot program being made out of
>> >the I40e
>> >driver (which you might note sends pull requests to the list now).  I'd
>> >still
>> >like to see all PMD's come under a single subtree, but thats likely an
>> >argument
>> >for later.
>> >
>> >That said, Do you think that this patch latency is really a contributor
>> >to low
>> >project participation?  It definately a problem, but it seems to me
>>that
>> >this
>> >sort of issue would lead to people trying to parcitipate, then giving
>>up
>> >(i.e.
>> >we would see 1-2 emails from an individual, then not see them again).
>> >I'd need
>> >to look through the mailing list for such a pattern, but anecdotally
>>I've
>> >not
>> >seen that happen.  The problem you describe above is definately a
>> >problem, but
>> >its one for those individuals who are participating, not for those who
>>are
>> >simply choosing not to.  And I think we need to address both.
>> >
>> >> I agree patchwork has some limitation, but I think the biggest issue
>>is
>> >> keeping up with the patches. Getting patches introduced into the main
>> >>line
>> >> is very slow. A patch submitted today may not get applied for weeks
>>or
>> >> months, then when another person submits a patch he is starting to
>>run a
>> >> very high risk of having to redo that patch, because a pervious patch
>> >> makes his fail weeks/months later. I would love to see a better tool
>> >>then
>> >> patchwork, but the biggest issue is we have a huge backlog of
>>patches.
>> >> Personally I am not sure how Thomas or any is able to keep up with
>>the
>> >> patches.
>> >> 
>> >This is absolutely a problem.  I'd like to think, more than a tool like
>> >patchwork, a subtree organization to allow some modicum of parallel
>> >review and
>> >integration would really be a benefit here.
>> Subtrees could work, but the real problem I think is the number of
>> committers must be higher then one. Something like GitHub (and I assume
>> Linux Foundation) have a method to add committers to a project. In the
>> case of GitHub they just have to have a free GitHub account and they can
>> become committers of the project buying the owner of the project enables
>> them.
>> 
>> On GitHub they have personal accounts and organization accounts I know
>> only about the personal accounts, but they allow for 5 private repos and
>> any number of public repos. The organization account has a lot of extra
>> features that seem better for a DPDK community IMO and should be the one
>> we use if we decide it is the right direction. We can always give it a
>> shot for while and keep the dpdk.org and use dev@dpdk.org and its repo
>> mirrored from GitHub as a transition phase. This way we can fall back to
>> dpdk.org or move one to something else if we like.
>> 
>> https://help.github.com/categories/organizations/
>> 
>> The developers could still send patches via email list, but creating a
>> repo and forking dpdk is easy, then send a pull request.
>> 
>I'm not opposed to github per-se, but nothing described above is unique to
>github. Theres no reason we can't allow multiple comitters to the current
>tree
>as hosted on the current server, we just have to configure it as such.
>
>And FWIW, the assumption is that, with multiple subtrees, you implicitly
>have
>multiple comitters, assuming that pull requests from those subtree
>maintainers
>are trusted by the top level tree maintainer.
>
>In fact I feel somewhat better about that model as it provides a nice
>stairstep
>integration path for new features.
>
>Not explicitly opposed to a movement to github, I just feel like it may
>not
>address the problem at hand.

As I see your concerns is creating multiple repos or splitting up the
current repo, which can be done in a single GitHub org account and they
all appear on the page. This way we can move the current other repos like
Pktgen to this location and everyone sees all of the repos in a much
easier way IMO. The org account at GitHub gives you the multiple
committers and even teams. I see we only need one team today for DPDK repo
and then we have something like Pktgen as a different team and so on.
>
>> 
>> >
>> >> The other problem I see is how patches are agreed on to be included
>>in
>> >>the
>> >> mainline. Today it is just an ACK or a NAK on the mailing list. Then
>>I
>> >>see
>> >> what I think to be only a few people ACKing or NAKing patches. This
>> >> process has a lot of problems from a patch being ignore for some
>>reason
>> >>or
>> >> someone having negative feed back on very minor detail or no way to
>> >>push a
>> >> patch forward a single NAK or comment.
>> >> 
>> >
>> >So, this is an interesting issue in ideal meritocracies.  Currently
>> >is/should be
>> >looking for ACKs/NAK/s from the individuals listed in the MAINTAINER
>> >files, and
>> >those people should be the definitive subject matter experts on the
>>code
>> >they
>> >cover.  As such, I would agrue that they should be entitled to a
>>modicum
>> >of
>> >stylistic/trivial leeway.  That is to say, if they choose to block a
>>patch
>> >around a very minor detail, then between them changing their position,
>> >and the
>> >patch author changing the code, the latter is likely the easier course
>>of
>> >action, especially if the author can't make an argument for their
>> >position.
>> >That said, if such patch blockage becomes so egregious that individuals
>> >stop
>> >contributing, that needs to be known as well.  If you as a patch
>>author:
>> >
>> >1) Have tried to submit patches
>> >2) Had them blocked for what you consider trivial reasons
>> >3) Plan to not contribute further because of this
>> >4) Still rely on the DPDK for your product
>> >
>> >Please, say something.  People in charge need to know when they're
>>pushing
>> >contributors away.
>> >
>> >FWIW, I've tried to do some correlation between the git history and the
>> >mailing
>> >list.  I need to do more searches, but I have a feeling that early on,
>>the
>> >majority of people who stopped contributing, did so because their
>>patches
>> >weren't expressely blocked, but rather because they were simply
>>ignored.
>> >No one
>> >working on DPDK bothered to review those patches, and so they never got
>> >merged.
>> >Hopefully that problem has been addressed somewhat now.
>> >
>> >> I would like to see some type of layering process to allow patches
>>to be
>> >> applied in a timely manner a few weeks not months or completely
>>ignored.
>> >> Maybe some type of voting is reasonable, but we need to do something
>>to
>> >> turn around the patches in clean reasonable manner.
>> >> 
>> >> Think we need some type of group meeting every week to look at the
>> >>patches
>> >> and determining which ones get applied, this gives quick feedback to
>>the
>> >> submitter as to the status of the patch.
>> >> 
>> >I think a group meeting is going to be way too much overhead to manage
>> >properly.
>> >You'll get different people every week with agenda that may not line up
>> >with
>> >code quality, which is really what the review is meant to provide.  I
>> >think
>> 
>> I was only suggesting the maintainers attend the meeting. Of course they
>> have to attend or have someone attend for them, just to get the voting
>> done. If you do not attend then you do not get to vote or something like
>> that is reasonable. Not that we should try and define the process here.
>> 
>If you use multiple subtrees, theres no need for a meeting, or any sort of
>defiend process for voting, theres only an implicitly defined heirarchy of
>acceptance in bundled changesets.  If a desire of the community is to see
>more
>efficient review and lower changeset acceptance latency, it seems to be
>that a
>weekly meeting of any sort is somewhat anathema to that.

That is fine if you do not want a meeting, but just trying to figure out
the best solution to the problems.
>
>> >perhaps a better approach would be to require that that code owners
>>from
>> >the
>> >maintainer file provide and ACK/NAK on their patches within 3-4 days,
>>and
>> >require a corresponding tree maintainer to apply the patch within 7 or
>> >so.  That
>> >would cap our patch latency.  Likewise, if a patch slips in creating a
>> >regression, the author needs to be alerted and given a time window in
>> >which to
>> >fix the problem before the offending patch is reverted during the QE
>> >cycle.
>> >
>> >
>> >> >
>> >> >On the other side, since user questions, community discussions and
>> >> >development happens in the same mailing list, things get really
>> >> >complicated, specially for users seeking for help. Even though I
>>think
>> >> >the average skills of the users of DPDK is generally higher than in
>> >> >other software projects, if DPDK wants to attract more users,
>>having a
>> >> >better user support is key, IMHO.
>> >> >
>> >> >So I would see with good eyes a separation between, at least,
>>dpdk-user
>> >> >and dpdk-dev.
>> >> 
>> >I wouldn't argue with this separation, seems like a reasonable
>>approach.
>> >
>> >> I do not remember seeing too many users on the list and making a list
>> >>just
>> >> for then is OK if everyone is fine with a list that has very few
>>emails.
>> >> >
>> >> >If the number of patches keeps growing, splitting the "dev" mailing
>> >> >lists into different categories (eal and common, pmds, higher level
>> >> >abstractions...) could be an option. However, this last point opens
>>a
>> >> >lot of questions on how to minimize interference between the
>>different
>> >> >parts and API/ABI compatibility during the development.
>> >> 
>> >> I believe if we just make sure we use tags in the subject line then
>>we
>> >>can
>> >> have our email clients do the splitting of the emails instead of
>>adding
>> >> more emails lists.
>> >> 
>> >Agreed
>> >
>> >> >
>> >> >>
>> >> >> Perhaps it means having some ReviewBoard type of tools, a clone in
>> >> >>Github or
>> >> >> Bitbucket where the less hardcore kernel-workflow types could send
>> >>back
>> >> >>their
>> >> >> small bug fixes a bit more easily, this kind of stuff. Google has
>> >>been
>> >> >>getting
>> >> >> good uptake since they moved most of their open source across to
>> >>Github,
>> >> >> because the contribution workflow was more convenient than Google
>> >>Code
>> >> >>was.
>> >> 
>> >> I like GitHub it is a much better designed tool then patchwork, plus
>>it
>> >> could get more eyes as it is very well know to the developer
>>community
>> >>in
>> >> general. I feel GitHub has many advantages over the current systems
>>in
>> >> place but, it does not solve the all patch issues.
>> >> 
>> >Github is actually a bit irritating for this sort of thing, as it
>> >presumes a web
>> >based interface for discussion.  They have some modicum of email
>> >forwarding
>> >enabled, but it has never quite worked right, or integrated properly.
>> 
>> Email forwarding has seemed to work for me and in one case it took a bit
>> to have GitHub stop sending me emails on a repo I did not want anymore
>>:-)
>
>Forwarding works fine, its responding that doesn't usually work well.
>Emails
>from pull requests and issues are forwarded from a 'do not reply' address
>and
>responding requires that you visit the github page in a browser.  Thats
>especially trying with patch review, as there is no real concept of
>patches on a
>list, only pull requests which require that you pull a new branch in and
>review
>it.  Once you have to leave your MUA, your efficiency quickly goes down,
>which
>is what we're trying to avoid here.

OK, I see your point and yes that could be an issue, but not a huge impact
in efficiency IMO. Not everyone and not all of the time will you need to
go to the web site.
>
>
>> >
>> >> The only way we can get patch issues resolved is to put a bit more
>> >>process
>> >> in place.
>> >> >
>> >> >Although I agree, we have to be careful on how github or bitbucket
>>is
>> >> >used. Having issues or even (e.g. github) pull requests *in
>>addition*
>> >>to
>> >> >the normal contribution workflow can be a nightmare to deal with, in
>> >> >terms of synchronization and preventing double work. So I guess
>>setting
>> >> >up an official github or bitbucket mirror would be fine, via some
>> >>simple
>> >> >cronjob, but I guess it would end-up not using PRs or issues in
>>github
>> >> >like the Linux kernel does.
>> >> 
>> >100% agree, we can't be split about this.  Allowing contributions from
>>n
>> >channels just means most developers will only see/reviews 1/nth of the
>> >patches
>> >of interest to them.
>> 
>> If we setup a GitHub or some other site, we would need to make Github
>>the
>> primary site to remove this type of problem IMO.
>> >
>> >> From what I can tell GitHub seems to be a better solution for a free
>> >>open
>> >> environment. Bitbucket I have never used and GitHub seems more
>>popular
>> >> from one article I read.
>> >> 
>> >> 
>> 
>>>>https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UT
>>>>F-
>> >>8#
>> >> q=bitbucket%20vs%20github
>> >> 
>> >> 
>> >> >Btw, is this github organization already registered by Intel or some
>> >> >other company of the community?
>> >> >
>> >> >https://github.com/dpdk
>> >> >
>> 
>> I was hoping someone would own up to the GitHub dpdk site.
>> 
>Hmm, looks almost defunct.  If no one steps up, perhaps reporting abuse
>for
>camping on the name might be worthwhile?  That would at least get their
>attention.

+1
>
>Neil
>