[dpdk-dev] DPDK Technical Board Meeting, 2017-02-15

[dpdk-dev] DPDK Technical Board Meeting, 2017-02-15

[Richardson, Bruce]

A meeting of the DPDK technical board was held last Wednesday, 2017-02-15. Below are the meeting minutes.  

Please note that future meetings will be open to all to attend, as described below. The next meeting is planned for March 1st, and any topics to be referred to the tech board for discussion at that meeting should be emailed to techboard@dpdk.org by February 26th at the latest.

Regards,
/Bruce

------------------------------------------------------------------
Attendees:
Hemant Agrawal, Konstantin Ananyev, Jan Blunck, Stephen Hemminger, Jerin Jacob, Yuanhan Liu, Olivier Matz, Thomas Monjalon, Bruce Richardson

Agenda and Notes:
0. General Meeting Admin
 * Bruce Richardson volunteered to chair this meeting (Meeting chair was previously agreed to be a rotating role.)
 * Olivier Matz volunteered to chair the next meeting
 * At each future meetings, the chair for the next meeting will be decided and that person also has the responsibility of preparing the meeting agenda ahead of time.
 * Next meeting was agreed to be at the same time in two weeks - 1st March @ 3pm UTC (4pm France, 11pm PRC, 7am US Pacific). 
   - Olivier to send a reminder with agenda a few days in advance.

1. Welcome new members to the board, and answer any questions they had about technical board operations.
 * Hemant, Jan and Yuanhan were welcomed, but had no questions on tech board at this point.

2. Making meetings public
 * LF charter for DPDK specifies that meetings of the techboard should be public
 * Decision made to continue to hold meetings on #dpdk-board channel on IRC
 * Meetings will be announced ahead of time:
    - In minutes of previous meeting
    - Via separate email a couple of days before the meeting
 * Anyone interested in the board meeting can join the IRC channel and make their contribution.
 * Only items on the agreed agenda to be discussed. Any topics to be covered must be sent in advance to techboard@dpdk.org

3. Update board charter on website
 * The website details of the Tech board is out of date, both membership and its mode of operation and scope
 * Thomas has already sent one proposed update out via email, in the form of patch to the website, and Hemant has already provided one comment
 * Thomas to send out a new draft, and all tech board members to comment or ack it to signal agreement.
 * Update won't be applied until all members ack. 
 * If not agreed before next meeting, should be discussed at that meeting

4. Issue of review and timely feedback
 * Discussion focused on review of patches for existing DPDK components/libraries
 * Agreed that patch maintainers are primarily responsible for accepting patches in their area, and need to ensure sufficient reviews are done
 * Once patch has been sufficiently reviewed, the maintainer should ack the patch
 * Any patches which have been acked by the maintainer, and which have no subsequent issues raised with them, should be automatically merged to the relevant tree after 2 weeks.
     - in case where the ack is received less than 2 weeks before a merge deadline, the point at which it can be automatically merged is 2 days before the deadline
     - To make a release, a patch must be acked by the maintainer at least 2 days before the merge deadline.
 * If not merged at the automatic-merge deadline itself, e.g. unavoidable delay by committer, any subsequent feedback received should be considered "too late", and must be addressed by a follow-on patch, rather than via a revision of the original submission.
 * Trivial patches may be merged sooner, or without maintainer ack, at the tree committers discretion.

5. Accept and review new libraries
 * Discussion begun on this topic, but no consensus reached before time ran out
 * Most members felt this topic was closely tied to the "scope" of DPDK as a project
 * Discussion to continue over email, and to be discussed at the next meeting if not resolved before then.

Agenda Topics Not Covered:
* Community questions/issues
* Any action for some old patches/threads?

Action Summary:
* Olivier to prepare agenda for next meeting, and send out reminders
* Thomas to send revised draft of website update for tech board membership etc.

NEXT MEETING:
* 2017 - 03 - 01 @ 3pm UTC

[dpdk-dev] decision process to accept new libraries

[Thomas Monjalon]

2017-02-17 10:22, Richardson, Bruce:
> 4. Issue of review and timely feedback
>  * Discussion focused on review of patches for existing DPDK components/libraries
>  * Agreed that patch maintainers are primarily responsible for accepting patches in their area, and need to ensure sufficient reviews are done
>  * Once patch has been sufficiently reviewed, the maintainer should ack the patch
>  * Any patches which have been acked by the maintainer, and which have no subsequent issues raised with them, should be automatically merged to the relevant tree after 2 weeks.
>      - in case where the ack is received less than 2 weeks before a merge deadline, the point at which it can be automatically merged is 2 days before the deadline
>      - To make a release, a patch must be acked by the maintainer at least 2 days before the merge deadline.
>  * If not merged at the automatic-merge deadline itself, e.g. unavoidable delay by committer, any subsequent feedback received should be considered "too late", and must be addressed by a follow-on patch, rather than via a revision of the original submission.
>  * Trivial patches may be merged sooner, or without maintainer ack, at the tree committers discretion.

That's really good to have more rules like the one above.
It will make the process clearer to everyone,
and hopefully will speed up our workflow.

> 5. Accept and review new libraries
>  * Discussion begun on this topic, but no consensus reached before time ran out
>  * Most members felt this topic was closely tied to the "scope" of DPDK as a project
>  * Discussion to continue over email, and to be discussed at the next meeting if not resolved before then.

Let's discuss this important topic.

Deciding to add a library is about deciding the scope of the project.
However I think we do not need to define the DPDK scope now,
but we can define a process to help in scope decisions.
It is a first step which will help us to progress later in scope discussions.

Below are 3 separate (and related) topics to discuss.
Please let's target a decision for the first item only.

1/
I suggest that each new library must be developed in a separate repository
on dpdk.org. Then it can be asked to integrate it in the main project/repo.
Such discussion must happen on the mailing list and the techboard will vote
for the integration of the *existing* library.
I suggest such vote should be done by majority as stated in the LF charter.

2/
We could imagine the same kind of process for removal,
i.e. moving a library or an app outside of DPDK in a separate repository.
If there is a public request with enough discussion, the techboard could vote
for moving some DPDK parts in a separate repository.
The committer would be a volunteer or the current maintainer of the code.
A dedicated mailing-list will be created for the new project.

3/
Why a project scope should be defined?
I think it is important to agree on the long-term goal of defining a scope.
My view on the goal for the main DPDK project (git://dpdk.org/dpdk):
- a consistent set of libraries
- a good quality in every libraries
- contributors interested in every libraries (no community segmentation)
- contributors able to understand every libraries
In less words, a clear project by a strong community of contributors.

Re: [dpdk-dev] decision process to accept new libraries

[Bruce Richardson]

On Fri, Feb 17, 2017 at 12:16:59PM +0100, Thomas Monjalon wrote:
> 2017-02-17 10:22, Richardson, Bruce:
> > 4. Issue of review and timely feedback
> >  * Discussion focused on review of patches for existing DPDK components/libraries
> >  * Agreed that patch maintainers are primarily responsible for accepting patches in their area, and need to ensure sufficient reviews are done
> >  * Once patch has been sufficiently reviewed, the maintainer should ack the patch
> >  * Any patches which have been acked by the maintainer, and which have no subsequent issues raised with them, should be automatically merged to the relevant tree after 2 weeks.
> >      - in case where the ack is received less than 2 weeks before a merge deadline, the point at which it can be automatically merged is 2 days before the deadline
> >      - To make a release, a patch must be acked by the maintainer at least 2 days before the merge deadline.
> >  * If not merged at the automatic-merge deadline itself, e.g. unavoidable delay by committer, any subsequent feedback received should be considered "too late", and must be addressed by a follow-on patch, rather than via a revision of the original submission.
> >  * Trivial patches may be merged sooner, or without maintainer ack, at the tree committers discretion.
> 
> That's really good to have more rules like the one above.
> It will make the process clearer to everyone,
> and hopefully will speed up our workflow.
> 

I've just posted a patch to add this to our contributor guide doc.
Please review and comment.

> > 5. Accept and review new libraries
> >  * Discussion begun on this topic, but no consensus reached before time ran out
> >  * Most members felt this topic was closely tied to the "scope" of DPDK as a project
> >  * Discussion to continue over email, and to be discussed at the next meeting if not resolved before then.
> 
> Let's discuss this important topic.
> 
> Deciding to add a library is about deciding the scope of the project.
> However I think we do not need to define the DPDK scope now,
> but we can define a process to help in scope decisions.
> It is a first step which will help us to progress later in scope discussions.
> 
> Below are 3 separate (and related) topics to discuss.
> Please let's target a decision for the first item only.
> 
> 1/
> I suggest that each new library must be developed in a separate repository
> on dpdk.org. Then it can be asked to integrate it in the main project/repo.
> Such discussion must happen on the mailing list and the techboard will vote
> for the integration of the *existing* library.
> I suggest such vote should be done by majority as stated in the LF charter.

That sounds a reasonable starting point. Are you suggesting that each
library go in a separate repo, or that we have a common staging area
repo for all new libs?

> 
> 2/
> We could imagine the same kind of process for removal,
> i.e. moving a library or an app outside of DPDK in a separate repository.
> If there is a public request with enough discussion, the techboard could vote
> for moving some DPDK parts in a separate repository.
> The committer would be a volunteer or the current maintainer of the code.
> A dedicated mailing-list will be created for the new project.
> 
I always think that removing things should be harder than adding, so I
have a couple of concerns here.
* Having "a public request with enough discussion" seems a rather
  tenuous threshold for a library to be faced with removal. Should some
  more measurable metric be used instead? For example,
  - lack of a responsive maintainer for 2 release cycles
  - high-priority bug (that severely limits the usefulness of the
    library), open for one release cycle.
  - request of maintainer (in case anyone voluntarily wants their code
    removed)
* I think for removal we might want to consider needing more than a
  majority of the tech board. Majority +1 or something perhaps?

> 3/
> Why a project scope should be defined?
> I think it is important to agree on the long-term goal of defining a scope.
> My view on the goal for the main DPDK project (git://dpdk.org/dpdk):
> - a consistent set of libraries
> - a good quality in every libraries
> - contributors interested in every libraries (no community segmentation)
> - contributors able to understand every libraries
> In less words, a clear project by a strong community of contributors.
> 
The third one is the one I would query. I would view that as an
impossible goal to meet. If a significant subset of the community has an
interest in something, that is surely enough reason to have it in the
project.
One link of relevance to this discussion both in terms of scope and
removing things, is this case from the linux kernel:
http://news.softpedia.com/news/Linus-Tolvalds-Keeps-Code-in-the-Kernel-for-Just-One-User-470853.shtml


Regards,
/Bruce

Re: [dpdk-dev] decision process to accept new libraries

[Thomas Monjalon]

2017-02-21 13:46, Bruce Richardson:
> On Fri, Feb 17, 2017 at 12:16:59PM +0100, Thomas Monjalon wrote:
> > 2017-02-17 10:22, Richardson, Bruce:
> > > 5. Accept and review new libraries
> > >  * Discussion begun on this topic, but no consensus reached before time ran out
> > >  * Most members felt this topic was closely tied to the "scope" of DPDK as a project
> > >  * Discussion to continue over email, and to be discussed at the next meeting if not resolved before then.
> > 
> > Let's discuss this important topic.
> > 
> > Deciding to add a library is about deciding the scope of the project.
> > However I think we do not need to define the DPDK scope now,
> > but we can define a process to help in scope decisions.
> > It is a first step which will help us to progress later in scope discussions.
> > 
> > Below are 3 separate (and related) topics to discuss.
> > Please let's target a decision for the first item only.
> > 
> > 1/
> > I suggest that each new library must be developed in a separate repository
> > on dpdk.org. Then it can be asked to integrate it in the main project/repo.
> > Such discussion must happen on the mailing list and the techboard will vote
> > for the integration of the *existing* library.
> > I suggest such vote should be done by majority as stated in the LF charter.
> 
> That sounds a reasonable starting point. Are you suggesting that each
> library go in a separate repo, or that we have a common staging area
> repo for all new libs?

I suggest adding a new repo for each new lib or group some of them if relevant.
Examples: dpdk-qos, dpdk-metrics, dpdk-extra-examples

> > 2/
> > We could imagine the same kind of process for removal,
> > i.e. moving a library or an app outside of DPDK in a separate repository.
> > If there is a public request with enough discussion, the techboard could vote
> > for moving some DPDK parts in a separate repository.
> > The committer would be a volunteer or the current maintainer of the code.
> > A dedicated mailing-list will be created for the new project.
> > 
> I always think that removing things should be harder than adding, so I
> have a couple of concerns here.
> * Having "a public request with enough discussion" seems a rather
>   tenuous threshold for a library to be faced with removal. Should some
>   more measurable metric be used instead? For example,
>   - lack of a responsive maintainer for 2 release cycles
>   - high-priority bug (that severely limits the usefulness of the
>     library), open for one release cycle.
>   - request of maintainer (in case anyone voluntarily wants their code
>     removed)

I agree with these criteria.
I would like to add one more:
- there are some maintenance work and is considered out of scope

> * I think for removal we might want to consider needing more than a
>   majority of the tech board. Majority +1 or something perhaps?

I think the technical board majority will take the right decisions.
So I don't see a need to less consider this majority in some cases.

> > 3/
> > Why a project scope should be defined?
> > I think it is important to agree on the long-term goal of defining a scope.
> > My view on the goal for the main DPDK project (git://dpdk.org/dpdk):
> > - a consistent set of libraries
> > - a good quality in every libraries
> > - contributors interested in every libraries (no community segmentation)
> > - contributors able to understand every libraries
> > In less words, a clear project by a strong community of contributors.
> > 
> The third one is the one I would query. I would view that as an
> impossible goal to meet. If a significant subset of the community has an
> interest in something, that is surely enough reason to have it in the
> project.

Yes I tend to agree. If a driver is useful for one user and there is not
a lot more work to do on it, it deserves to be in DPDK.
But we must avoid adding a small library or example which is at the scope
border and grows so much that it becomes too important for the rest of
the project.

> One link of relevance to this discussion both in terms of scope and
> removing things, is this case from the linux kernel:
> http://news.softpedia.com/news/Linus-Tolvalds-Keeps-Code-in-the-Kernel-for-Just-One-User-470853.shtml

There is a big difference between Linux and DPDK: the first one is a kernel!
There is no alternative for a monolithic kernel: it is in or out.
In the DPDK case, we can host several libraries or examples in some
separate repositories.
The impact of having separate repositories is to reduce the work of a
contributor touching many areas in a rework. This cost is transfered
to the maintainer of the separate repository impacted by the change
in the main repository. So it becomes this question:
Do we prefer requiring some maintenance work from the contributors or
from the maintainers?

Re: [dpdk-dev] decision process to accept new libraries

[Dumitrescu, Cristian]

...<snip>
> > > 1/
> > > I suggest that each new library must be developed in a separate
> repository
> > > on dpdk.org. Then it can be asked to integrate it in the main project/repo.
> > > Such discussion must happen on the mailing list and the techboard will
> vote
> > > for the integration of the *existing* library.
> > > I suggest such vote should be done by majority as stated in the LF
> charter.
> >
> > That sounds a reasonable starting point. Are you suggesting that each
> > library go in a separate repo, or that we have a common staging area
> > repo for all new libs?
> 
> I suggest adding a new repo for each new lib or group some of them if
> relevant.
> Examples: dpdk-qos, dpdk-metrics, dpdk-extra-examples
> 

Are these repos still going to make it into the DPDK release package or left outside?

Re: [dpdk-dev] decision process to accept new libraries

[Dumitrescu, Cristian]

...<snip>

> The impact of having separate repositories is to reduce the work of a
> contributor touching many areas in a rework. This cost is transfered
> to the maintainer of the separate repository impacted by the change
> in the main repository. So it becomes this question:
> Do we prefer requiring some maintenance work from the contributors or
> from the maintainers?

IMO it is not fair for a contributor of the "main" repository to break stuff in other repos without fixing the other repos.

This essentially leads to the "other" repos becoming second class citizens that can be broken at any time without prior notice or the right to influence the change. The amount of maintenance work becomes very difficult to quantify (e.g. we all know what a ripple effect a chance in the mbuf structure can cause to any  of those "other" DPDK libraries). This is likely to lead to different release schedules for every of those "other" repos and big hassle in building a single unified DPDK release package. Or is it desired that DPDK release package should only contain the "main" repo?

What would be the advantages to this model, Thomas? And what are the issues with the current model of "you break it, you fix it"?

Re: [dpdk-dev] decision process to accept new libraries

[Remy Horton]

On 22/02/2017 19:06, Dumitrescu, Cristian wrote:
[..]
> This essentially leads to the "other" repos becoming second class
> citizens that can be broken at any time without prior notice or the
> right to influence the change. The amount of maintenance work becomes
> very difficult to quantify (e.g. we all know what a ripple effect a
> chance in the mbuf structure can cause to any  of those "other" DPDK
> libraries).

+1 - In my experience anything other than a single repository ends up in 
tears sooner or later. At a previous company I worked on a project where 
each "module" went into its own repo, all fourty-five of which were 
strung together using Gerrit/Jenkins, the result being I spent more time 
on rebases and build breakages than writing business logic. Patchsets 
that cross repo boundaries are a recipe for pain, and if DPDK goes down 
the same route, it will likley cripple development.

Re: [dpdk-dev] decision process to accept new libraries

[Thomas Monjalon]

2017-02-22 19:06, Dumitrescu, Cristian:
> ...<snip>
> 
> > The impact of having separate repositories is to reduce the work of a
> > contributor touching many areas in a rework. This cost is transfered
> > to the maintainer of the separate repository impacted by the change
> > in the main repository. So it becomes this question:
> > Do we prefer requiring some maintenance work from the contributors or
> > from the maintainers?
> 
> IMO it is not fair for a contributor of the "main" repository to break stuff in other repos without fixing the other repos.

The question is "is it fair to ask a contributor to fix every libraries using a core one?"

> This essentially leads to the "other" repos becoming second class citizens that can be broken at any time without prior notice or the right to influence the change. The amount of maintenance work becomes very difficult to quantify (e.g. we all know what a ripple effect a chance in the mbuf structure can cause to any  of those "other" DPDK libraries). This is likely to lead to different release schedules for every of those "other" repos and big hassle in building a single unified DPDK release package. Or is it desired that DPDK release package should only contain the "main" repo?

Yes the idea is to have a core package of the "main" repo.

> What would be the advantages to this model, Thomas?
> And what are the issues with the current model of "you break it, you fix it"?

That's a very good question Cristian.
As said above, it is a matter of deciding the scope of responsibility
of a contributor to a core library, or saying it differently,
who should do the work on other libs and multiple examples?
About the advantages, I think it could ease the contributions on core
libraries and let people who are not full-time on DPDK to contribute
to the core libraries.

That's a real question and feedbacks are very welcome.
I'd like to read opinions of more contributors. Thanks

Re: [dpdk-dev] decision process to accept new libraries

[Thomas Monjalon]

2017-02-24 11:33, Remy Horton:
> 
> On 22/02/2017 19:06, Dumitrescu, Cristian wrote:
> [..]
> > This essentially leads to the "other" repos becoming second class
> > citizens that can be broken at any time without prior notice or the
> > right to influence the change. The amount of maintenance work becomes
> > very difficult to quantify (e.g. we all know what a ripple effect a
> > chance in the mbuf structure can cause to any  of those "other" DPDK
> > libraries).
> 
> +1 - In my experience anything other than a single repository ends up in 
> tears sooner or later. At a previous company I worked on a project where 
> each "module" went into its own repo, all fourty-five of which were 
> strung together using Gerrit/Jenkins, the result being I spent more time 
> on rebases and build breakages than writing business logic. Patchsets 
> that cross repo boundaries are a recipe for pain, and if DPDK goes down 
> the same route, it will likley cripple development.

Indeed, that's the idea: give more work to the maintainers and require less
work from occasional contributors.

It may be a good or wrong idea. Anyway it deserves to be discussed.

Re: [dpdk-dev] decision process to accept new libraries

[Olivier Matz]

On Fri, 24 Feb 2017 11:33:11 +0000, Remy Horton <remy.horton@intel.com>
wrote:
> On 22/02/2017 19:06, Dumitrescu, Cristian wrote:
> [..]
> > This essentially leads to the "other" repos becoming second class
> > citizens that can be broken at any time without prior notice or the
> > right to influence the change. The amount of maintenance work
> > becomes very difficult to quantify (e.g. we all know what a ripple
> > effect a chance in the mbuf structure can cause to any  of those
> > "other" DPDK libraries).  
> 
> +1 - In my experience anything other than a single repository ends up
> in tears sooner or later. At a previous company I worked on a project
> where each "module" went into its own repo, all fourty-five of which
> were strung together using Gerrit/Jenkins, the result being I spent
> more time on rebases and build breakages than writing business logic.
> Patchsets that cross repo boundaries are a recipe for pain, and if
> DPDK goes down the same route, it will likley cripple development.

On the other hand, I think we can agree that everything that depends on
dpdk cannot be hosted in dpdk repository.

Many applications are hosted in other repositories: for instance pktgen
or vpp. A given version of these applications runs on a given version of
dpdk. It could also applies for libraries.

Having apps/libs outside the dpdk repo is more work for their
maintainers because they may need to revalidate (compilation + test) for
each dpdk version. Having them inside the dpdk repo is more work for
the maintainers of dpdk core libs, because they need to update all of
them when they do a big changes. This is sometimes not doable because
they don't have a test platform or knowledge for each pmd or lib.

That's why I think this is a decision that can be done by the technical
board on a case by case basis.

Olivier

Re: [dpdk-dev] decision process to accept new libraries

[Bruce Richardson]

On Fri, Feb 24, 2017 at 02:07:22PM +0100, Thomas Monjalon wrote:
> 2017-02-22 19:06, Dumitrescu, Cristian:
> > ...<snip>
> > 
> > > The impact of having separate repositories is to reduce the work of a
> > > contributor touching many areas in a rework. This cost is transfered
> > > to the maintainer of the separate repository impacted by the change
> > > in the main repository. So it becomes this question:
> > > Do we prefer requiring some maintenance work from the contributors or
> > > from the maintainers?
> > 
> > IMO it is not fair for a contributor of the "main" repository to break stuff in other repos without fixing the other repos.
> 
> The question is "is it fair to ask a contributor to fix every libraries using a core one?"
> 
> > This essentially leads to the "other" repos becoming second class citizens that can be broken at any time without prior notice or the right to influence the change. The amount of maintenance work becomes very difficult to quantify (e.g. we all know what a ripple effect a chance in the mbuf structure can cause to any  of those "other" DPDK libraries). This is likely to lead to different release schedules for every of those "other" repos and big hassle in building a single unified DPDK release package. Or is it desired that DPDK release package should only contain the "main" repo?
> 
> Yes the idea is to have a core package of the "main" repo.
> 
> > What would be the advantages to this model, Thomas?
> > And what are the issues with the current model of "you break it, you fix it"?
> 
> That's a very good question Cristian.
> As said above, it is a matter of deciding the scope of responsibility
> of a contributor to a core library, or saying it differently,
> who should do the work on other libs and multiple examples?
> About the advantages, I think it could ease the contributions on core
> libraries and let people who are not full-time on DPDK to contribute
> to the core libraries.
> 
> That's a real question and feedbacks are very welcome.
> I'd like to read opinions of more contributors. Thanks

I think the reduction in work for new contributors is an admirable one.
However, the idea of separating out libs to achieve this is not one I'm
so sure about. My thinking is:
* the examples, rather than the libs, might be a better source of "fat"
  to be cut. We're already looking at removing the old dpdk_qat sample,
  so perhaps more can follow that.
* are library dependencies really that much an issue?
* do we really want to make it that easy to change core library APIs? If
  the core library APIs change, that has an effect on everyone else
  using DPDK, so it's not an effort to be undertaken lightly. Having to
  change a bunch of other libraries and examples using that code will
  help contributors to understand a) what they are asking others to go
  through to update their own code for the change b) give insight into
  how the APIs are actually being used.

Regards,
/Bruce

Re: [dpdk-dev] decision process to accept new libraries

[Bruce Richardson]

On Fri, Feb 24, 2017 at 02:17:31PM +0100, Olivier Matz wrote:
> On Fri, 24 Feb 2017 11:33:11 +0000, Remy Horton <remy.horton@intel.com>
> wrote:
> > On 22/02/2017 19:06, Dumitrescu, Cristian wrote:
> > [..]
> > > This essentially leads to the "other" repos becoming second class
> > > citizens that can be broken at any time without prior notice or the
> > > right to influence the change. The amount of maintenance work
> > > becomes very difficult to quantify (e.g. we all know what a ripple
> > > effect a chance in the mbuf structure can cause to any  of those
> > > "other" DPDK libraries).  
> > 
> > +1 - In my experience anything other than a single repository ends up
> > in tears sooner or later. At a previous company I worked on a project
> > where each "module" went into its own repo, all fourty-five of which
> > were strung together using Gerrit/Jenkins, the result being I spent
> > more time on rebases and build breakages than writing business logic.
> > Patchsets that cross repo boundaries are a recipe for pain, and if
> > DPDK goes down the same route, it will likley cripple development.
> 
> On the other hand, I think we can agree that everything that depends on
> dpdk cannot be hosted in dpdk repository.
> 
> Many applications are hosted in other repositories: for instance pktgen
> or vpp. A given version of these applications runs on a given version of
> dpdk. It could also applies for libraries.
> 
> Having apps/libs outside the dpdk repo is more work for their
> maintainers because they may need to revalidate (compilation + test) for
> each dpdk version. Having them inside the dpdk repo is more work for
> the maintainers of dpdk core libs, because they need to update all of
> them when they do a big changes. This is sometimes not doable because
> they don't have a test platform or knowledge for each pmd or lib.

Maybe not, and I wouldn't expect someone making a change to have to test
every library affected by the change. However, I would expect them to have
enough knowledge of the change being made to update the affected code in
other libraries in a semi-mechanical way, and ensure it compiles. If you
are changing a core library and are not able to change all uses of the
API yourself, then you really need to question if the change is a good
one or not, or if you are qualified to make a change if you don't
understand how the old code was being used.

I also think it's likely that many users of DPDK code will have legacy
code using DPDK, for which the original programmer may be the one making
any updates. If, again, the change is such that it can't be done in a
relatively mechanical way, that is going to be a problem for all those
users.

As for review and testing, that is the responsibility of the maintainers
and validators responsible for the individual library being updated.

/Bruce

Re: [dpdk-dev] decision process to accept new libraries

[Olivier Matz]

On Fri, 24 Feb 2017 13:25:46 +0000, Bruce Richardson
<bruce.richardson@intel.com> wrote:
> On Fri, Feb 24, 2017 at 02:17:31PM +0100, Olivier Matz wrote:
> > On Fri, 24 Feb 2017 11:33:11 +0000, Remy Horton
> > <remy.horton@intel.com> wrote:  
> > > On 22/02/2017 19:06, Dumitrescu, Cristian wrote:
> > > [..]  
> > > > This essentially leads to the "other" repos becoming second
> > > > class citizens that can be broken at any time without prior
> > > > notice or the right to influence the change. The amount of
> > > > maintenance work becomes very difficult to quantify (e.g. we
> > > > all know what a ripple effect a chance in the mbuf structure
> > > > can cause to any  of those "other" DPDK libraries).    
> > > 
> > > +1 - In my experience anything other than a single repository
> > > ends up in tears sooner or later. At a previous company I worked
> > > on a project where each "module" went into its own repo, all
> > > fourty-five of which were strung together using Gerrit/Jenkins,
> > > the result being I spent more time on rebases and build breakages
> > > than writing business logic. Patchsets that cross repo boundaries
> > > are a recipe for pain, and if DPDK goes down the same route, it
> > > will likley cripple development.  
> > 
> > On the other hand, I think we can agree that everything that
> > depends on dpdk cannot be hosted in dpdk repository.
> > 
> > Many applications are hosted in other repositories: for instance
> > pktgen or vpp. A given version of these applications runs on a
> > given version of dpdk. It could also applies for libraries.
> > 
> > Having apps/libs outside the dpdk repo is more work for their
> > maintainers because they may need to revalidate (compilation +
> > test) for each dpdk version. Having them inside the dpdk repo is
> > more work for the maintainers of dpdk core libs, because they need
> > to update all of them when they do a big changes. This is sometimes
> > not doable because they don't have a test platform or knowledge for
> > each pmd or lib.  
> 
> Maybe not, and I wouldn't expect someone making a change to have to
> test every library affected by the change. However, I would expect
> them to have enough knowledge of the change being made to update the
> affected code in other libraries in a semi-mechanical way, and ensure
> it compiles. If you are changing a core library and are not able to
> change all uses of the API yourself, then you really need to question
> if the change is a good one or not, or if you are qualified to make a
> change if you don't understand how the old code was being used.

Yes. But the problem is when the guy that wants to do the change is
aware that he/she does not have the skills to update all the libraries.
At the end, the change may never be done because of the amount of work.

I can give some examples:
- remove the m->port field (requires to update some examples which
  hijack this field, maybe also some libs)
- move some mbuf fields in the second cache line (requires to update
  all the PMDs, including vector ones)

However I agree with what you said on the content :)
I just want to highlight that both approaches have their advantages and
drawbacks. And because to me there is no easy answer, I think it's sage
to let this decision to the technical board.

> 
> I also think it's likely that many users of DPDK code will have legacy
> code using DPDK, for which the original programmer may be the one
> making any updates. If, again, the change is such that it can't be
> done in a relatively mechanical way, that is going to be a problem
> for all those users.
> 
> As for review and testing, that is the responsibility of the
> maintainers and validators responsible for the individual library
> being updated.
> 
> /Bruce
>