[dpdk-dev] git trees organization

[dpdk-dev] git trees organization

[Thomas Monjalon]

Hi all,

As you know I am currently the only maintainer of the master tree.
It is very convenient because I need to synchronize with others
only when pulling "next-*" trees.
But the drawback is that I should be available very often to
avoid stalled patches waiting in patchwork backlog.

I feel it is the good time to move to a slightly different organization.
I am working closely with Ferruh Yigit for almost one year, as next-net
maintainer, and I think it would be very efficient to delegate him some
work for the master tree.
I mean that I would use the patchwork delegation to explicitly divide
the workload given our different experiences.
Ferruh, do you agree taking this new responsibility?

At the same time, we can think how to add more git sub-trees:

Should we create next-net-intel for Intel networking drivers?
Any volunteer?

Should we create next-bus for bus API and drivers?
Stephen Hemminger is working on a new bus.
Would you be interested by taking the responsibility of this git tree?

Should we create next-mem for malloc/mempool?

Should we take ethdev patches into next-net?

Other suggestions?

Re: [dpdk-dev] git trees organization

[Bruce Richardson]

On Tue, Sep 12, 2017 at 12:03:30AM +0200, Thomas Monjalon wrote:
> Hi all,
> 
> As you know I am currently the only maintainer of the master tree.
> It is very convenient because I need to synchronize with others
> only when pulling "next-*" trees.
> But the drawback is that I should be available very often to
> avoid stalled patches waiting in patchwork backlog.
> 
> I feel it is the good time to move to a slightly different organization.
> I am working closely with Ferruh Yigit for almost one year, as next-net
> maintainer, and I think it would be very efficient to delegate him some
> work for the master tree.

I think Ferruh has been doing an excellent job on the net tree, and
would be an excellent candidate to help with the workload on the master
tree.

> I mean that I would use the patchwork delegation to explicitly divide
> the workload given our different experiences.
> Ferruh, do you agree taking this new responsibility?
> 
> At the same time, we can think how to add more git sub-trees:

In principle, I'm in favour, but I think that the subtrees of the master
tree should be at a fairly coarse granularity, and not be too many of
them. The more subtrees, the more likely we are to have issues with
patchsets needing to be split across trees, or having to take bits from
multiple trees in order to test if everything is working.

> Should we create next-net-intel for Intel networking drivers?

Given the number and size of intel drivers, this seems reasonable to
start as a second-level subtree.

> Any volunteer?
> 
> Should we create next-bus for bus API and drivers?
> Stephen Hemminger is working on a new bus.
> Would you be interested by taking the responsibility of this git tree?

Is this something that is going to need ongoing work and maintenance, or
just something that would be needed while the current rework of
introducing bus types is being done? If the former, a tree makes sense,
but not if it's the latter case.

> 
> Should we create next-mem for malloc/mempool?
> 
Core libs tree, encompassing eal, mempool and 1 or 2 others? I don't
think memory should have its own tree initially.

> Should we take ethdev patches into next-net?
Definitely! I think not doing so was a bit of a mistake when net tree
was spun off.
> 
> Other suggestions?
Similar to above, cryptodev should be in crypto tree, eventdev in event
tree etc.

Other than that, all I can say is "let's do it!". We have quite a
backlog to get through for 17.11, so anything that moves things along
faster is to be welcomed.

/Bruce

Re: [dpdk-dev] git trees organization

[Thomas Monjalon]

12/09/2017 10:32, Bruce Richardson:
> On Tue, Sep 12, 2017 at 12:03:30AM +0200, Thomas Monjalon wrote:
[...]
> > At the same time, we can think how to add more git sub-trees:
> 
> In principle, I'm in favour, but I think that the subtrees of the master
> tree should be at a fairly coarse granularity, and not be too many of
> them. The more subtrees, the more likely we are to have issues with
> patchsets needing to be split across trees, or having to take bits from
> multiple trees in order to test if everything is working.
> 
> > Should we create next-net-intel for Intel networking drivers?
> 
> Given the number and size of intel drivers, this seems reasonable to
> start as a second-level subtree.

OK, we need the name of a volunteer :)

> > Any volunteer?
> > 
> > Should we create next-bus for bus API and drivers?
> > Stephen Hemminger is working on a new bus.
> > Would you be interested by taking the responsibility of this git tree?
> 
> Is this something that is going to need ongoing work and maintenance, or
> just something that would be needed while the current rework of
> introducing bus types is being done? If the former, a tree makes sense,
> but not if it's the latter case.

We are going to have many bus drivers (pci, vdev, fslmc, vmbus).
If we look only at PCI, there are always some new patches to improve
or fix things. So I think it is reasonnable to imagine that we will
have some real activity with all bus drivers.

> > Should we create next-mem for malloc/mempool?
> > 
> Core libs tree, encompassing eal, mempool and 1 or 2 others? I don't
> think memory should have its own tree initially.
> 
> > Should we take ethdev patches into next-net?
> 
> Definitely! I think not doing so was a bit of a mistake when net tree
> was spun off.

Sure it was a mistake, but it was assumed because net drivers is already
a big work. I hope we can add it now while moving Intel drivers to
a second level sub-tree.

> > Other suggestions?
> 
> Similar to above, cryptodev should be in crypto tree, eventdev in event
> tree etc.

It is already the case. No change to do here :)

> Other than that, all I can say is "let's do it!". We have quite a
> backlog to get through for 17.11, so anything that moves things along
> faster is to be welcomed.

Yes!

Re: [dpdk-dev] git trees organization

[Wiles, Keith]

> On Sep 12, 2017, at 3:48 AM, Thomas Monjalon <thomas@monjalon.net> wrote:
> 
> 12/09/2017 10:32, Bruce Richardson:
>> On Tue, Sep 12, 2017 at 12:03:30AM +0200, Thomas Monjalon wrote:
> [...]
>>> At the same time, we can think how to add more git sub-trees:
>> 
>> In principle, I'm in favour, but I think that the subtrees of the master
>> tree should be at a fairly coarse granularity, and not be too many of
>> them. The more subtrees, the more likely we are to have issues with
>> patchsets needing to be split across trees, or having to take bits from
>> multiple trees in order to test if everything is working.
>> 
>>> Should we create next-net-intel for Intel networking drivers?
>> 
>> Given the number and size of intel drivers, this seems reasonable to
>> start as a second-level subtree.
> 
> OK, we need the name of a volunteer :)

You miss-spelled ‘volunteer' it should be 'victim’ :-)

> 
>>> Any volunteer?
>>> 
>>> Should we create next-bus for bus API and drivers?
>>> Stephen Hemminger is working on a new bus.
>>> Would you be interested by taking the responsibility of this git tree?
>> 
>> Is this something that is going to need ongoing work and maintenance, or
>> just something that would be needed while the current rework of
>> introducing bus types is being done? If the former, a tree makes sense,
>> but not if it's the latter case.
> 
> We are going to have many bus drivers (pci, vdev, fslmc, vmbus).
> If we look only at PCI, there are always some new patches to improve
> or fix things. So I think it is reasonnable to imagine that we will
> have some real activity with all bus drivers.
> 
>>> Should we create next-mem for malloc/mempool?
>>> 
>> Core libs tree, encompassing eal, mempool and 1 or 2 others? I don't
>> think memory should have its own tree initially.
>> 
>>> Should we take ethdev patches into next-net?
>> 
>> Definitely! I think not doing so was a bit of a mistake when net tree
>> was spun off.
> 
> Sure it was a mistake, but it was assumed because net drivers is already
> a big work. I hope we can add it now while moving Intel drivers to
> a second level sub-tree.
> 
>>> Other suggestions?
>> 
>> Similar to above, cryptodev should be in crypto tree, eventdev in event
>> tree etc.
> 
> It is already the case. No change to do here :)
> 
>> Other than that, all I can say is "let's do it!". We have quite a
>> backlog to get through for 17.11, so anything that moves things along
>> faster is to be welcomed.
> 
> Yes!

Regards,
Keith

Re: [dpdk-dev] git trees organization

[Ferruh Yigit]

Hi Thomas,

On 9/11/2017 11:03 PM, Thomas Monjalon wrote:
> Hi all,
> 
> As you know I am currently the only maintainer of the master tree.
> It is very convenient because I need to synchronize with others
> only when pulling "next-*" trees.
> But the drawback is that I should be available very often to
> avoid stalled patches waiting in patchwork backlog.
> 
> I feel it is the good time to move to a slightly different organization.
> I am working closely with Ferruh Yigit for almost one year, as next-net
> maintainer, and I think it would be very efficient to delegate him some
> work for the master tree.
> I mean that I would use the patchwork delegation to explicitly divide
> the workload given our different experiences.
> Ferruh, do you agree taking this new responsibility?

ACK, I will do my best.

Re: [dpdk-dev] git trees organization

[Ferruh Yigit]

On 9/12/2017 9:48 AM, Thomas Monjalon wrote:
> 12/09/2017 10:32, Bruce Richardson:
>> On Tue, Sep 12, 2017 at 12:03:30AM +0200, Thomas Monjalon wrote:
> [...]
>>> At the same time, we can think how to add more git sub-trees:
>>
>> In principle, I'm in favour, but I think that the subtrees of the master
>> tree should be at a fairly coarse granularity, and not be too many of
>> them. The more subtrees, the more likely we are to have issues with
>> patchsets needing to be split across trees, or having to take bits from
>> multiple trees in order to test if everything is working.
>>
>>> Should we create next-net-intel for Intel networking drivers?
>>
>> Given the number and size of intel drivers, this seems reasonable to
>> start as a second-level subtree.
> 
> OK, we need the name of a volunteer :)

+1 to next-net-intel

Why not have next-net-mlx, next-net-cavium etc too.. This can reduce
load for next-net, which can be shifted to main repo.

> 
>>> Any volunteer?
>>>
>>> Should we create next-bus for bus API and drivers?
>>> Stephen Hemminger is working on a new bus.
>>> Would you be interested by taking the responsibility of this git tree?
>>
>> Is this something that is going to need ongoing work and maintenance, or
>> just something that would be needed while the current rework of
>> introducing bus types is being done? If the former, a tree makes sense,
>> but not if it's the latter case.
> 
> We are going to have many bus drivers (pci, vdev, fslmc, vmbus).
> If we look only at PCI, there are always some new patches to improve
> or fix things. So I think it is reasonnable to imagine that we will
> have some real activity with all bus drivers.
> 
>>> Should we create next-mem for malloc/mempool?
>>>
>> Core libs tree, encompassing eal, mempool and 1 or 2 others? I don't
>> think memory should have its own tree initially.
>>
>>> Should we take ethdev patches into next-net?
>>
>> Definitely! I think not doing so was a bit of a mistake when net tree
>> was spun off.
> 
> Sure it was a mistake, but it was assumed because net drivers is already
> a big work. I hope we can add it now while moving Intel drivers to
> a second level sub-tree.

+1

> 
>>> Other suggestions?
>>
>> Similar to above, cryptodev should be in crypto tree, eventdev in event
>> tree etc.
> 
> It is already the case. No change to do here :)
> 
>> Other than that, all I can say is "let's do it!". We have quite a
>> backlog to get through for 17.11, so anything that moves things along
>> faster is to be welcomed.
> 
> Yes!
> 

Re: [dpdk-dev] git trees organization

[Thomas Monjalon]

12/09/2017 18:32, Ferruh Yigit:
> Hi Thomas,
> 
> On 9/11/2017 11:03 PM, Thomas Monjalon wrote:
> > I feel it is the good time to move to a slightly different organization.
> > I am working closely with Ferruh Yigit for almost one year, as next-net
> > maintainer, and I think it would be very efficient to delegate him some
> > work for the master tree.
> > I mean that I would use the patchwork delegation to explicitly divide
> > the workload given our different experiences.
> > Ferruh, do you agree taking this new responsibility?
> 
> ACK, I will do my best.

Thank you so much Ferruh!

As discussed privately, we will wait for techboard validation.

Re: [dpdk-dev] git trees organization

[Adrien Mazarguil]

Hi,

On Tue, Sep 12, 2017 at 09:32:07AM +0100, Bruce Richardson wrote:
> On Tue, Sep 12, 2017 at 12:03:30AM +0200, Thomas Monjalon wrote:
> > Hi all,
> > 
> > As you know I am currently the only maintainer of the master tree.
> > It is very convenient because I need to synchronize with others
> > only when pulling "next-*" trees.
> > But the drawback is that I should be available very often to
> > avoid stalled patches waiting in patchwork backlog.
> > 
> > I feel it is the good time to move to a slightly different organization.
> > I am working closely with Ferruh Yigit for almost one year, as next-net
> > maintainer, and I think it would be very efficient to delegate him some
> > work for the master tree.
> 
> I think Ferruh has been doing an excellent job on the net tree, and
> would be an excellent candidate to help with the workload on the master
> tree.
> 
> > I mean that I would use the patchwork delegation to explicitly divide
> > the workload given our different experiences.
> > Ferruh, do you agree taking this new responsibility?
> > 
> > At the same time, we can think how to add more git sub-trees:
> 
> In principle, I'm in favour, but I think that the subtrees of the master
> tree should be at a fairly coarse granularity, and not be too many of
> them. The more subtrees, the more likely we are to have issues with
> patchsets needing to be split across trees, or having to take bits from
> multiple trees in order to test if everything is working.
<snip>

About that, how about we start allowing true merge commits instead of
rebasing (rewriting history) in order to ease things for maintainers?

This approach makes pull requests show up as a merge commits that contain
the (ideally trivial) changes needed to resolve any conflicts; this has the
following benefits:

- The work done by a maintainer during that merge is tracked, not silently
  ignored or lost. The merge commit itself is signed-off by its author.

- This allows tracing mistakes or bugs to the conflict resolution itself.

- Upstream can reject pull requests on the basis that merging it is not
  trivial enough (i.e. downstream must merge upstream changes first).

- Sub-trees can merge among themselves in case they need features that
  encompass several trees, not necessarily always against the master
  tree. Everything is tracked.

- Maintainers do not ever modify the commits they get from other trees,
  which keep their SHAs unmodified as part of the history. A given commit ID
  is truly unique among all trees (back-port trees remain the only exception
  since commits are cherry-picked).

- It shifts the entire responsibility to the maintainers of sub-trees.

The only downside is that commits have several parents, history becomes a
graph that developers need to get used to (some might call it a mess),
however that's probably not an issue for those already used to Linux kernel
development and other large projects.

I know this was already discussed in the past, however I think adding more
sub-trees will make rebasing too complex otherwise.

Thoughts?

-- 
Adrien Mazarguil
6WIND

Re: [dpdk-dev] git trees organization

[Ferruh Yigit]

On 9/13/2017 8:58 AM, Adrien Mazarguil wrote:
> Hi,
> 
> On Tue, Sep 12, 2017 at 09:32:07AM +0100, Bruce Richardson wrote:
>> On Tue, Sep 12, 2017 at 12:03:30AM +0200, Thomas Monjalon wrote:
>>> Hi all,
>>>
>>> As you know I am currently the only maintainer of the master tree.
>>> It is very convenient because I need to synchronize with others
>>> only when pulling "next-*" trees.
>>> But the drawback is that I should be available very often to
>>> avoid stalled patches waiting in patchwork backlog.
>>>
>>> I feel it is the good time to move to a slightly different organization.
>>> I am working closely with Ferruh Yigit for almost one year, as next-net
>>> maintainer, and I think it would be very efficient to delegate him some
>>> work for the master tree.
>>
>> I think Ferruh has been doing an excellent job on the net tree, and
>> would be an excellent candidate to help with the workload on the master
>> tree.
>>
>>> I mean that I would use the patchwork delegation to explicitly divide
>>> the workload given our different experiences.
>>> Ferruh, do you agree taking this new responsibility?
>>>
>>> At the same time, we can think how to add more git sub-trees:
>>
>> In principle, I'm in favour, but I think that the subtrees of the master
>> tree should be at a fairly coarse granularity, and not be too many of
>> them. The more subtrees, the more likely we are to have issues with
>> patchsets needing to be split across trees, or having to take bits from
>> multiple trees in order to test if everything is working.
> <snip>
> 
> About that, how about we start allowing true merge commits instead of
> rebasing (rewriting history) in order to ease things for maintainers?
> 
> This approach makes pull requests show up as a merge commits that contain
> the (ideally trivial) changes needed to resolve any conflicts; this has the
> following benefits:
> 
> - The work done by a maintainer during that merge is tracked, not silently
>   ignored or lost. The merge commit itself is signed-off by its author.
> 
> - This allows tracing mistakes or bugs to the conflict resolution itself.
> 
> - Upstream can reject pull requests on the basis that merging it is not
>   trivial enough (i.e. downstream must merge upstream changes first).
> 
> - Sub-trees can merge among themselves in case they need features that
>   encompass several trees, not necessarily always against the master
>   tree. Everything is tracked.
> 
> - Maintainers do not ever modify the commits they get from other trees,
>   which keep their SHAs unmodified as part of the history. A given commit ID
>   is truly unique among all trees (back-port trees remain the only exception
>   since commits are cherry-picked).
> 
> - It shifts the entire responsibility to the maintainers of sub-trees.
> 
> The only downside is that commits have several parents, history becomes a
> graph that developers need to get used to (some might call it a mess),
> however that's probably not an issue for those already used to Linux kernel
> development and other large projects.
> 
> I know this was already discussed in the past, however I think adding more
> sub-trees will make rebasing too complex otherwise>
> Thoughts?
>

Using git merge looks more proper git usage, but I have one question /
concern:

For next-net, sometimes there are dependent patches in main tree, and
what I am doing is rebasing sub-tree on top of latest main tree.

When switched to merge method, how dependent patches can be get into the
sub-tree? Merge from main tree to sub-tree? Won't this bidirectional
merging confusing?


And following are notes from my current experience:

- Having re-writable history gives some flexibility to sub-trees.
Possible to update commit logs and amend patches even after pushed.

- It is hard to confirm pulled commits in main tree, I guess merge
commit will make this easier.

- To track main tree, continuously rebasing and continuously re-writing
history, I am doing this almost daily, this may be hard for people
working on top of next-net.

- Conflict resolving done by sub-trees during rebase, instead of done by
main tree during merge. So this may be more distributed effort.

- Rebasing gives more straight forward history in main repo, merge
commits looks more confusing, although I would expect it won't be as
complex as Linux tree, so may not be a problem.

Thanks,
ferruh

Re: [dpdk-dev] git trees organization

[Adrien Mazarguil]

On Wed, Sep 13, 2017 at 12:38:37PM +0100, Ferruh Yigit wrote:
> On 9/13/2017 8:58 AM, Adrien Mazarguil wrote:
> > Hi,
> > 
> > On Tue, Sep 12, 2017 at 09:32:07AM +0100, Bruce Richardson wrote:
> >> On Tue, Sep 12, 2017 at 12:03:30AM +0200, Thomas Monjalon wrote:
> >>> Hi all,
> >>>
> >>> As you know I am currently the only maintainer of the master tree.
> >>> It is very convenient because I need to synchronize with others
> >>> only when pulling "next-*" trees.
> >>> But the drawback is that I should be available very often to
> >>> avoid stalled patches waiting in patchwork backlog.
> >>>
> >>> I feel it is the good time to move to a slightly different organization.
> >>> I am working closely with Ferruh Yigit for almost one year, as next-net
> >>> maintainer, and I think it would be very efficient to delegate him some
> >>> work for the master tree.
> >>
> >> I think Ferruh has been doing an excellent job on the net tree, and
> >> would be an excellent candidate to help with the workload on the master
> >> tree.
> >>
> >>> I mean that I would use the patchwork delegation to explicitly divide
> >>> the workload given our different experiences.
> >>> Ferruh, do you agree taking this new responsibility?
> >>>
> >>> At the same time, we can think how to add more git sub-trees:
> >>
> >> In principle, I'm in favour, but I think that the subtrees of the master
> >> tree should be at a fairly coarse granularity, and not be too many of
> >> them. The more subtrees, the more likely we are to have issues with
> >> patchsets needing to be split across trees, or having to take bits from
> >> multiple trees in order to test if everything is working.
> > <snip>
> > 
> > About that, how about we start allowing true merge commits instead of
> > rebasing (rewriting history) in order to ease things for maintainers?
> > 
> > This approach makes pull requests show up as a merge commits that contain
> > the (ideally trivial) changes needed to resolve any conflicts; this has the
> > following benefits:
> > 
> > - The work done by a maintainer during that merge is tracked, not silently
> >   ignored or lost. The merge commit itself is signed-off by its author.
> > 
> > - This allows tracing mistakes or bugs to the conflict resolution itself.
> > 
> > - Upstream can reject pull requests on the basis that merging it is not
> >   trivial enough (i.e. downstream must merge upstream changes first).
> > 
> > - Sub-trees can merge among themselves in case they need features that
> >   encompass several trees, not necessarily always against the master
> >   tree. Everything is tracked.
> > 
> > - Maintainers do not ever modify the commits they get from other trees,
> >   which keep their SHAs unmodified as part of the history. A given commit ID
> >   is truly unique among all trees (back-port trees remain the only exception
> >   since commits are cherry-picked).
> > 
> > - It shifts the entire responsibility to the maintainers of sub-trees.
> > 
> > The only downside is that commits have several parents, history becomes a
> > graph that developers need to get used to (some might call it a mess),
> > however that's probably not an issue for those already used to Linux kernel
> > development and other large projects.
> > 
> > I know this was already discussed in the past, however I think adding more
> > sub-trees will make rebasing too complex otherwise>
> > Thoughts?
> >
> 
> Using git merge looks more proper git usage, but I have one question /
> concern:
> 
> For next-net, sometimes there are dependent patches in main tree, and
> what I am doing is rebasing sub-tree on top of latest main tree.
> 
> When switched to merge method, how dependent patches can be get into the
> sub-tree? Merge from main tree to sub-tree?

Yes, that's the idea. On the other hand, as a maintainer, you are not
responsible for the contents of what's merged from other official
trees. Commits are taken as they are, this implies trust between tree
maintainers.

> Won't this bidirectional merging confusing?

Probably at first, this certainly needs some getting used to. We can attempt
to avoid such merges as much as possible with proper coordination. Avoiding
them is likely not possible though if we want to keep history intact.

> And following are notes from my current experience:
> 
> - Having re-writable history gives some flexibility to sub-trees.
> Possible to update commit logs and amend patches even after pushed.

This is both a good and a bad thing. Thanks to that, history is currently
linear and extremely clean, I think we haven't had a single patch that
doesn't compile or a bad merge artifact for a very long time.

On the other hand, if you look at the effort required to maintain a single
sub-tree that way, it likely becomes exponential for several. All of them
need to constantly rebase their stuff, with conflicts to address. The more
people, the more mistakes and so on.

Once part of an official tree, a commit cannot be amended anymore, it's too
late; revert commits will be more common. We need to accept this first.

> - It is hard to confirm pulled commits in main tree, I guess merge
> commit will make this easier.
> 
> - To track main tree, continuously rebasing and continuously re-writing
> history, I am doing this almost daily, this may be hard for people
> working on top of next-net.

Yes that's one of the "bad" things with the current approach. Consider
automatic non-regression testing against disappearing commit IDs. Tracking
their status is difficult when history is changing. For instance we usually
add local tags to track CI successes/failures. All of them now point to
nonexistent commits.

> - Conflict resolving done by sub-trees during rebase, instead of done by
> main tree during merge. So this may be more distributed effort.

It's not too different actually. With merge you can reject PRs on the basis
that there are too many conflicts to take care of, not unlike a request for
rebase.

On the plus side if you make any changes yourself in order to solve them,
they are made part of the merge commit, not in the original commits which
remain unmodified.

> - Rebasing gives more straight forward history in main repo, merge
> commits looks more confusing, although I would expect it won't be as
> complex as Linux tree, so may not be a problem.

Right, that's the main drawback. I think there's no way to know the impact
unless we attempt it as an experiment.

-- 
Adrien Mazarguil
6WIND

Re: [dpdk-dev] git trees organization

[Ferruh Yigit]

On 9/13/2017 1:25 PM, Adrien Mazarguil wrote:
> On Wed, Sep 13, 2017 at 12:38:37PM +0100, Ferruh Yigit wrote:
>> On 9/13/2017 8:58 AM, Adrien Mazarguil wrote:
>>> Hi,
>>>
>>> On Tue, Sep 12, 2017 at 09:32:07AM +0100, Bruce Richardson wrote:
>>>> On Tue, Sep 12, 2017 at 12:03:30AM +0200, Thomas Monjalon wrote:
>>>>> Hi all,
>>>>>
>>>>> As you know I am currently the only maintainer of the master tree.
>>>>> It is very convenient because I need to synchronize with others
>>>>> only when pulling "next-*" trees.
>>>>> But the drawback is that I should be available very often to
>>>>> avoid stalled patches waiting in patchwork backlog.
>>>>>
>>>>> I feel it is the good time to move to a slightly different organization.
>>>>> I am working closely with Ferruh Yigit for almost one year, as next-net
>>>>> maintainer, and I think it would be very efficient to delegate him some
>>>>> work for the master tree.
>>>>
>>>> I think Ferruh has been doing an excellent job on the net tree, and
>>>> would be an excellent candidate to help with the workload on the master
>>>> tree.
>>>>
>>>>> I mean that I would use the patchwork delegation to explicitly divide
>>>>> the workload given our different experiences.
>>>>> Ferruh, do you agree taking this new responsibility?
>>>>>
>>>>> At the same time, we can think how to add more git sub-trees:
>>>>
>>>> In principle, I'm in favour, but I think that the subtrees of the master
>>>> tree should be at a fairly coarse granularity, and not be too many of
>>>> them. The more subtrees, the more likely we are to have issues with
>>>> patchsets needing to be split across trees, or having to take bits from
>>>> multiple trees in order to test if everything is working.
>>> <snip>
>>>
>>> About that, how about we start allowing true merge commits instead of
>>> rebasing (rewriting history) in order to ease things for maintainers?
>>>
>>> This approach makes pull requests show up as a merge commits that contain
>>> the (ideally trivial) changes needed to resolve any conflicts; this has the
>>> following benefits:
>>>
>>> - The work done by a maintainer during that merge is tracked, not silently
>>>   ignored or lost. The merge commit itself is signed-off by its author.
>>>
>>> - This allows tracing mistakes or bugs to the conflict resolution itself.
>>>
>>> - Upstream can reject pull requests on the basis that merging it is not
>>>   trivial enough (i.e. downstream must merge upstream changes first).
>>>
>>> - Sub-trees can merge among themselves in case they need features that
>>>   encompass several trees, not necessarily always against the master
>>>   tree. Everything is tracked.
>>>
>>> - Maintainers do not ever modify the commits they get from other trees,
>>>   which keep their SHAs unmodified as part of the history. A given commit ID
>>>   is truly unique among all trees (back-port trees remain the only exception
>>>   since commits are cherry-picked).
>>>
>>> - It shifts the entire responsibility to the maintainers of sub-trees.
>>>
>>> The only downside is that commits have several parents, history becomes a
>>> graph that developers need to get used to (some might call it a mess),
>>> however that's probably not an issue for those already used to Linux kernel
>>> development and other large projects.
>>>
>>> I know this was already discussed in the past, however I think adding more
>>> sub-trees will make rebasing too complex otherwise>
>>> Thoughts?
>>>
>>
>> Using git merge looks more proper git usage, but I have one question /
>> concern:
>>
>> For next-net, sometimes there are dependent patches in main tree, and
>> what I am doing is rebasing sub-tree on top of latest main tree.
>>
>> When switched to merge method, how dependent patches can be get into the
>> sub-tree? Merge from main tree to sub-tree?
> 
> Yes, that's the idea. On the other hand, as a maintainer, you are not
> responsible for the contents of what's merged from other official
> trees. Commits are taken as they are, this implies trust between tree
> maintainers.
> 
>> Won't this bidirectional merging confusing?
> 
> Probably at first, this certainly needs some getting used to. We can attempt
> to avoid such merges as much as possible with proper coordination. Avoiding
> them is likely not possible though if we want to keep history intact.

Let's assume I need to merge from main tree to next-net three times
before the integration, because of dependencies.
When Thomas merged next-net to main tree for rc1, will those three merge
commits visible in main tree?

For Linux I guess sub-trees not merging from Linus' tree because of
dependencies, I assume they are only merging after release to get new
commits not in their sub-tree.
But for DPDK main tree is also getting new patches on its own.

> 
>> And following are notes from my current experience:
>>
>> - Having re-writable history gives some flexibility to sub-trees.
>> Possible to update commit logs and amend patches even after pushed.
> 
> This is both a good and a bad thing. Thanks to that, history is currently
> linear and extremely clean, I think we haven't had a single patch that
> doesn't compile or a bad merge artifact for a very long time.
> 
> On the other hand, if you look at the effort required to maintain a single
> sub-tree that way, it likely becomes exponential for several. All of them
> need to constantly rebase their stuff, with conflicts to address. The more
> people, the more mistakes and so on.
> 
> Once part of an official tree, a commit cannot be amended anymore, it's too
> late; revert commits will be more common. We need to accept this first.
> 
>> - It is hard to confirm pulled commits in main tree, I guess merge
>> commit will make this easier.
>>
>> - To track main tree, continuously rebasing and continuously re-writing
>> history, I am doing this almost daily, this may be hard for people
>> working on top of next-net.
> 
> Yes that's one of the "bad" things with the current approach. Consider
> automatic non-regression testing against disappearing commit IDs. Tracking
> their status is difficult when history is changing. For instance we usually
> add local tags to track CI successes/failures. All of them now point to
> nonexistent commits.
> 
>> - Conflict resolving done by sub-trees during rebase, instead of done by
>> main tree during merge. So this may be more distributed effort.
> 
> It's not too different actually. With merge you can reject PRs on the basis
> that there are too many conflicts to take care of, not unlike a request for
> rebase.
> 
> On the plus side if you make any changes yourself in order to solve them,
> they are made part of the merge commit, not in the original commits which
> remain unmodified.

You are right, we are loosing original code when there is merge
conflict, and I think there is no way to trace back to the original
commit in repo, but from mail list and patchwork perhaps.

And very hard to find out what has been changed for conflict resolving,
and if something went wrong!

> 
>> - Rebasing gives more straight forward history in main repo, merge
>> commits looks more confusing, although I would expect it won't be as
>> complex as Linux tree, so may not be a problem.
> 
> Right, that's the main drawback. I think there's no way to know the impact
> unless we attempt it as an experiment.
> 

Re: [dpdk-dev] git trees organization

[Adrien Mazarguil]

On Wed, Sep 13, 2017 at 02:21:00PM +0100, Ferruh Yigit wrote:
> On 9/13/2017 1:25 PM, Adrien Mazarguil wrote:
> > On Wed, Sep 13, 2017 at 12:38:37PM +0100, Ferruh Yigit wrote:
> >> On 9/13/2017 8:58 AM, Adrien Mazarguil wrote:
> >>> Hi,
> >>>
> >>> On Tue, Sep 12, 2017 at 09:32:07AM +0100, Bruce Richardson wrote:
> >>>> On Tue, Sep 12, 2017 at 12:03:30AM +0200, Thomas Monjalon wrote:
> >>>>> Hi all,
> >>>>>
> >>>>> As you know I am currently the only maintainer of the master tree.
> >>>>> It is very convenient because I need to synchronize with others
> >>>>> only when pulling "next-*" trees.
> >>>>> But the drawback is that I should be available very often to
> >>>>> avoid stalled patches waiting in patchwork backlog.
> >>>>>
> >>>>> I feel it is the good time to move to a slightly different organization.
> >>>>> I am working closely with Ferruh Yigit for almost one year, as next-net
> >>>>> maintainer, and I think it would be very efficient to delegate him some
> >>>>> work for the master tree.
> >>>>
> >>>> I think Ferruh has been doing an excellent job on the net tree, and
> >>>> would be an excellent candidate to help with the workload on the master
> >>>> tree.
> >>>>
> >>>>> I mean that I would use the patchwork delegation to explicitly divide
> >>>>> the workload given our different experiences.
> >>>>> Ferruh, do you agree taking this new responsibility?
> >>>>>
> >>>>> At the same time, we can think how to add more git sub-trees:
> >>>>
> >>>> In principle, I'm in favour, but I think that the subtrees of the master
> >>>> tree should be at a fairly coarse granularity, and not be too many of
> >>>> them. The more subtrees, the more likely we are to have issues with
> >>>> patchsets needing to be split across trees, or having to take bits from
> >>>> multiple trees in order to test if everything is working.
> >>> <snip>
> >>>
> >>> About that, how about we start allowing true merge commits instead of
> >>> rebasing (rewriting history) in order to ease things for maintainers?
> >>>
> >>> This approach makes pull requests show up as a merge commits that contain
> >>> the (ideally trivial) changes needed to resolve any conflicts; this has the
> >>> following benefits:
> >>>
> >>> - The work done by a maintainer during that merge is tracked, not silently
> >>>   ignored or lost. The merge commit itself is signed-off by its author.
> >>>
> >>> - This allows tracing mistakes or bugs to the conflict resolution itself.
> >>>
> >>> - Upstream can reject pull requests on the basis that merging it is not
> >>>   trivial enough (i.e. downstream must merge upstream changes first).
> >>>
> >>> - Sub-trees can merge among themselves in case they need features that
> >>>   encompass several trees, not necessarily always against the master
> >>>   tree. Everything is tracked.
> >>>
> >>> - Maintainers do not ever modify the commits they get from other trees,
> >>>   which keep their SHAs unmodified as part of the history. A given commit ID
> >>>   is truly unique among all trees (back-port trees remain the only exception
> >>>   since commits are cherry-picked).
> >>>
> >>> - It shifts the entire responsibility to the maintainers of sub-trees.
> >>>
> >>> The only downside is that commits have several parents, history becomes a
> >>> graph that developers need to get used to (some might call it a mess),
> >>> however that's probably not an issue for those already used to Linux kernel
> >>> development and other large projects.
> >>>
> >>> I know this was already discussed in the past, however I think adding more
> >>> sub-trees will make rebasing too complex otherwise>
> >>> Thoughts?
> >>>
> >>
> >> Using git merge looks more proper git usage, but I have one question /
> >> concern:
> >>
> >> For next-net, sometimes there are dependent patches in main tree, and
> >> what I am doing is rebasing sub-tree on top of latest main tree.
> >>
> >> When switched to merge method, how dependent patches can be get into the
> >> sub-tree? Merge from main tree to sub-tree?
> > 
> > Yes, that's the idea. On the other hand, as a maintainer, you are not
> > responsible for the contents of what's merged from other official
> > trees. Commits are taken as they are, this implies trust between tree
> > maintainers.
> > 
> >> Won't this bidirectional merging confusing?
> > 
> > Probably at first, this certainly needs some getting used to. We can attempt
> > to avoid such merges as much as possible with proper coordination. Avoiding
> > them is likely not possible though if we want to keep history intact.
> 
> Let's assume I need to merge from main tree to next-net three times
> before the integration, because of dependencies.
> When Thomas merged next-net to main tree for rc1, will those three merge
> commits visible in main tree?

Yes, all merge commits will remain visible and part of history. There's only
one case when such commits are optional: fast-forwards. For instance
assuming the HEAD of your current branch is part of upstream's history, you
may not get a merge commit if you pull from upstream before applying a
series instead of doing the reverse.

Whoever gets the fast-forward wins, therefore merging often is better.

Even with many subsequent merges, it's not all that bad. Graph views such as
"git log --oneline --graph" help a lot in clarifying things (once you get
used to them).

> For Linux I guess sub-trees not merging from Linus' tree because of
> dependencies, I assume they are only merging after release to get new
> commits not in their sub-tree.

Linux does that all the time and not necessarily after a release, even among
sub-trees. We don't plan to have as many different trees as Linux so it
should remain much simpler for us in any case.

> But for DPDK main tree is also getting new patches on its own.

Same for Linux even if those come in minority, I think it's not a problem
either way, Git is really good at merging histories.

Note that a maintainer can also add glue commits of his own on top of his
tree in order to simplify a subsequent merge. This is better than addressing
everything in the merge commit itself in non-trivial cases (although it
should be the job of the requester).

> >> And following are notes from my current experience:
> >>
> >> - Having re-writable history gives some flexibility to sub-trees.
> >> Possible to update commit logs and amend patches even after pushed.
> > 
> > This is both a good and a bad thing. Thanks to that, history is currently
> > linear and extremely clean, I think we haven't had a single patch that
> > doesn't compile or a bad merge artifact for a very long time.
> > 
> > On the other hand, if you look at the effort required to maintain a single
> > sub-tree that way, it likely becomes exponential for several. All of them
> > need to constantly rebase their stuff, with conflicts to address. The more
> > people, the more mistakes and so on.
> > 
> > Once part of an official tree, a commit cannot be amended anymore, it's too
> > late; revert commits will be more common. We need to accept this first.
> > 
> >> - It is hard to confirm pulled commits in main tree, I guess merge
> >> commit will make this easier.
> >>
> >> - To track main tree, continuously rebasing and continuously re-writing
> >> history, I am doing this almost daily, this may be hard for people
> >> working on top of next-net.
> > 
> > Yes that's one of the "bad" things with the current approach. Consider
> > automatic non-regression testing against disappearing commit IDs. Tracking
> > their status is difficult when history is changing. For instance we usually
> > add local tags to track CI successes/failures. All of them now point to
> > nonexistent commits.
> > 
> >> - Conflict resolving done by sub-trees during rebase, instead of done by
> >> main tree during merge. So this may be more distributed effort.
> > 
> > It's not too different actually. With merge you can reject PRs on the basis
> > that there are too many conflicts to take care of, not unlike a request for
> > rebase.
> > 
> > On the plus side if you make any changes yourself in order to solve them,
> > they are made part of the merge commit, not in the original commits which
> > remain unmodified.
> 
> You are right, we are loosing original code when there is merge
> conflict, and I think there is no way to trace back to the original
> commit in repo, but from mail list and patchwork perhaps.
> 
> And very hard to find out what has been changed for conflict resolving,
> and if something went wrong!
> 
> > 
> >> - Rebasing gives more straight forward history in main repo, merge
> >> commits looks more confusing, although I would expect it won't be as
> >> complex as Linux tree, so may not be a problem.
> > 
> > Right, that's the main drawback. I think there's no way to know the impact
> > unless we attempt it as an experiment.

-- 
Adrien Mazarguil
6WIND

Re: [dpdk-dev] git trees organization

[Stephen Hemminger]

On Sep 13, 2017 7:54 AM, "Adrien Mazarguil" <adrien.mazarguil@6wind.com>
wrote:

On Wed, Sep 13, 2017 at 02:21:00PM +0100, Ferruh Yigit wrote:
> On 9/13/2017 1:25 PM, Adrien Mazarguil wrote:
> > On Wed, Sep 13, 2017 at 12:38:37PM +0100, Ferruh Yigit wrote:
> >> On 9/13/2017 8:58 AM, Adrien Mazarguil wrote:
> >>> Hi,
> >>>
> >>> On Tue, Sep 12, 2017 at 09:32:07AM +0100, Bruce Richardson wrote:
> >>>> On Tue, Sep 12, 2017 at 12:03:30AM +0200, Thomas Monjalon wrote:
> >>>>> Hi all,
> >>>>>
> >>>>> As you know I am currently the only maintainer of the master tree.
> >>>>> It is very convenient because I need to synchronize with others
> >>>>> only when pulling "next-*" trees.
> >>>>> But the drawback is that I should be available very often to
> >>>>> avoid stalled patches waiting in patchwork backlog.
> >>>>>
> >>>>> I feel it is the good time to move to a slightly different
organization.
> >>>>> I am working closely with Ferruh Yigit for almost one year, as
next-net
> >>>>> maintainer, and I think it would be very efficient to delegate him
some
> >>>>> work for the master tree.
> >>>>
> >>>> I think Ferruh has been doing an excellent job on the net tree, and
> >>>> would be an excellent candidate to help with the workload on the
master
> >>>> tree.
> >>>>
> >>>>> I mean that I would use the patchwork delegation to explicitly
divide
> >>>>> the workload given our different experiences.
> >>>>> Ferruh, do you agree taking this new responsibility?
> >>>>>
> >>>>> At the same time, we can think how to add more git sub-trees:
> >>>>
> >>>> In principle, I'm in favour, but I think that the subtrees of the
master
> >>>> tree should be at a fairly coarse granularity, and not be too many of
> >>>> them. The more subtrees, the more likely we are to have issues with
> >>>> patchsets needing to be split across trees, or having to take bits
from
> >>>> multiple trees in order to test if everything is working.
> >>> <snip>
> >>>
> >>> About that, how about we start allowing true merge commits instead of
> >>> rebasing (rewriting history) in order to ease things for maintainers?
> >>>
> >>> This approach makes pull requests show up as a merge commits that
contain
> >>> the (ideally trivial) changes needed to resolve any conflicts; this
has the
> >>> following benefits:
> >>>
> >>> - The work done by a maintainer during that merge is tracked, not
silently
> >>>   ignored or lost. The merge commit itself is signed-off by its
author.
> >>>
> >>> - This allows tracing mistakes or bugs to the conflict resolution
itself.
> >>>
> >>> - Upstream can reject pull requests on the basis that merging it is
not
> >>>   trivial enough (i.e. downstream must merge upstream changes first).
> >>>
> >>> - Sub-trees can merge among themselves in case they need features that
> >>>   encompass several trees, not necessarily always against the master
> >>>   tree. Everything is tracked.
> >>>
> >>> - Maintainers do not ever modify the commits they get from other
trees,
> >>>   which keep their SHAs unmodified as part of the history. A given
commit ID
> >>>   is truly unique among all trees (back-port trees remain the only
exception
> >>>   since commits are cherry-picked).
> >>>
> >>> - It shifts the entire responsibility to the maintainers of sub-trees.
> >>>
> >>> The only downside is that commits have several parents, history
becomes a
> >>> graph that developers need to get used to (some might call it a mess),
> >>> however that's probably not an issue for those already used to Linux
kernel
> >>> development and other large projects.
> >>>
> >>> I know this was already discussed in the past, however I think adding
more
> >>> sub-trees will make rebasing too complex otherwise>
> >>> Thoughts?
> >>>
> >>
> >> Using git merge looks more proper git usage, but I have one question /
> >> concern:
> >>
> >> For next-net, sometimes there are dependent patches in main tree, and
> >> what I am doing is rebasing sub-tree on top of latest main tree.
> >>
> >> When switched to merge method, how dependent patches can be get into
the
> >> sub-tree? Merge from main tree to sub-tree?
> >
> > Yes, that's the idea. On the other hand, as a maintainer, you are not
> > responsible for the contents of what's merged from other official
> > trees. Commits are taken as they are, this implies trust between tree
> > maintainers.
> >
> >> Won't this bidirectional merging confusing?
> >
> > Probably at first, this certainly needs some getting used to. We can
attempt
> > to avoid such merges as much as possible with proper coordination.
Avoiding
> > them is likely not possible though if we want to keep history intact.
>
> Let's assume I need to merge from main tree to next-net three times
> before the integration, because of dependencies.
> When Thomas merged next-net to main tree for rc1, will those three merge
> commits visible in main tree?

Yes, all merge commits will remain visible and part of history. There's only
one case when such commits are optional: fast-forwards. For instance
assuming the HEAD of your current branch is part of upstream's history, you
may not get a merge commit if you pull from upstream before applying a
series instead of doing the reverse.

Whoever gets the fast-forward wins, therefore merging often is better.

Even with many subsequent merges, it's not all that bad. Graph views such as
"git log --oneline --graph" help a lot in clarifying things (once you get
used to them).

> For Linux I guess sub-trees not merging from Linus' tree because of
> dependencies, I assume they are only merging after release to get new
> commits not in their sub-tree.

Linux does that all the time and not necessarily after a release, even among
sub-trees. We don't plan to have as many different trees as Linux so it
should remain much simpler for us in any case.

> But for DPDK main tree is also getting new patches on its own.

Same for Linux even if those come in minority, I think it's not a problem
either way, Git is really good at merging histories.

Note that a maintainer can also add glue commits of his own on top of his
tree in order to simplify a subsequent merge. This is better than addressing
everything in the merge commit itself in non-trivial cases (although it
should be the job of the requester).

> >> And following are notes from my current experience:
> >>
> >> - Having re-writable history gives some flexibility to sub-trees.
> >> Possible to update commit logs and amend patches even after pushed.
> >
> > This is both a good and a bad thing. Thanks to that, history is
currently
> > linear and extremely clean, I think we haven't had a single patch that
> > doesn't compile or a bad merge artifact for a very long time.
> >
> > On the other hand, if you look at the effort required to maintain a
single
> > sub-tree that way, it likely becomes exponential for several. All of
them
> > need to constantly rebase their stuff, with conflicts to address. The
more
> > people, the more mistakes and so on.
> >
> > Once part of an official tree, a commit cannot be amended anymore, it's
too
> > late; revert commits will be more common. We need to accept this first.
> >
> >> - It is hard to confirm pulled commits in main tree, I guess merge
> >> commit will make this easier.
> >>
> >> - To track main tree, continuously rebasing and continuously re-writing
> >> history, I am doing this almost daily, this may be hard for people
> >> working on top of next-net.
> >
> > Yes that's one of the "bad" things with the current approach. Consider
> > automatic non-regression testing against disappearing commit IDs.
Tracking
> > their status is difficult when history is changing. For instance we
usually
> > add local tags to track CI successes/failures. All of them now point to
> > nonexistent commits.
> >
> >> - Conflict resolving done by sub-trees during rebase, instead of done
by
> >> main tree during merge. So this may be more distributed effort.
> >
> > It's not too different actually. With merge you can reject PRs on the
basis
> > that there are too many conflicts to take care of, not unlike a request
for
> > rebase.
> >
> > On the plus side if you make any changes yourself in order to solve
them,
> > they are made part of the merge commit, not in the original commits
which
> > remain unmodified.
>
> You are right, we are loosing original code when there is merge
> conflict, and I think there is no way to trace back to the original
> commit in repo, but from mail list and patchwork perhaps.
>
> And very hard to find out what has been changed for conflict resolving,
> and if something went wrong!
>
> >
> >> - Rebasing gives more straight forward history in main repo, merge
> >> commits looks more confusing, although I would expect it won't be as
> >> complex as Linux tree, so may not be a problem.
> >
> > Right, that's the main drawback. I think there's no way to know the
impact
> > unless we attempt it as an experiment.

--
Adrien Mazarguil
6WIND


Bisecting a tree with lots of subtree merges is terrible. That is why Linus
rebases and doesn't directly take linux-next

Re: [dpdk-dev] git trees organization

[Thomas Monjalon]

14/09/2017 04:25, Stephen Hemminger:
> Bisecting a tree with lots of subtree merges is terrible. That is why Linus
> rebases and doesn't directly take linux-next

I agree, bisecting with subtree merges is not pleasant at all.
That's why I chose the rebase method until now.

Adrien mentioned some drawbacks with the rebase method.
Ferruh mentioned some drawbacks and some advantages of rebase.
Stephen mentioned another advantage of rebase.
Such decisions are really difficult.
One thing is sure: there will be always someone unhappy,
no matter the decision :)

When we want to take such decision or re-consider it,
we ask the techboard to vote...

Re: [dpdk-dev] git trees organization

[Bruce Richardson]

On Thu, Sep 14, 2017 at 10:22:23AM +0200, Thomas Monjalon wrote:
> 14/09/2017 04:25, Stephen Hemminger:
> > Bisecting a tree with lots of subtree merges is terrible. That is why Linus
> > rebases and doesn't directly take linux-next
> 
> I agree, bisecting with subtree merges is not pleasant at all.
> That's why I chose the rebase method until now.
> 
> Adrien mentioned some drawbacks with the rebase method.
> Ferruh mentioned some drawbacks and some advantages of rebase.
> Stephen mentioned another advantage of rebase.
> Such decisions are really difficult.
> One thing is sure: there will be always someone unhappy,
> no matter the decision :)
> 
> When we want to take such decision or re-consider it,
> we ask the techboard to vote...

I'm not sure the techboard needs to vote on this, this is an issue for
the tree maintainers/committers is it not? If you do want techboard
input on this, I suggest the committers come to an agreement among
themselves, with community input, and then just look for tech board to
ratify it.

/Bruce

Re: [dpdk-dev] git trees organization

[Nélio Laranjeiro]

On Thu, Sep 14, 2017 at 10:22:23AM +0200, Thomas Monjalon wrote:
> 14/09/2017 04:25, Stephen Hemminger:
> > Bisecting a tree with lots of subtree merges is terrible. That is why Linus
> > rebases and doesn't directly take linux-next
> 
> I agree, bisecting with subtree merges is not pleasant at all.
> That's why I chose the rebase method until now.

I don't see what is un-pleasant, if we start a bisect what we expect is to
find the commit introducing the issue, bisect is able to do it even on large
tree with a lot of merges.  Moreover, the probability the issue will be
searched in a specific section within its own subtree is high which also means
locally it will be linear, is not it equivalent to the actual situation?

> Adrien mentioned some drawbacks with the rebase method.
> Ferruh mentioned some drawbacks and some advantages of rebase.
> Stephen mentioned another advantage of rebase.
> Such decisions are really difficult.
> One thing is sure: there will be always someone unhappy,
> no matter the decision :)
> 
> When we want to take such decision or re-consider it,
> we ask the techboard to vote...

-- 
Nélio Laranjeiro
6WIND

Re: [dpdk-dev] git trees organization

[Thomas Monjalon]

14/09/2017 11:03, Bruce Richardson:
> On Thu, Sep 14, 2017 at 10:22:23AM +0200, Thomas Monjalon wrote:
> > 14/09/2017 04:25, Stephen Hemminger:
> > > Bisecting a tree with lots of subtree merges is terrible. That is why Linus
> > > rebases and doesn't directly take linux-next
> > 
> > I agree, bisecting with subtree merges is not pleasant at all.
> > That's why I chose the rebase method until now.
> > 
> > Adrien mentioned some drawbacks with the rebase method.
> > Ferruh mentioned some drawbacks and some advantages of rebase.
> > Stephen mentioned another advantage of rebase.
> > Such decisions are really difficult.
> > One thing is sure: there will be always someone unhappy,
> > no matter the decision :)
> > 
> > When we want to take such decision or re-consider it,
> > we ask the techboard to vote...
> 
> I'm not sure the techboard needs to vote on this, this is an issue for
> the tree maintainers/committers is it not? If you do want techboard
> input on this, I suggest the committers come to an agreement among
> themselves, with community input, and then just look for tech board to
> ratify it.

No, it is an issue for everybody.
Rebase makes tracking of subtrees difficult for developpers.
Merge makes reading and bisecting difficult for developpers.
We cannot have an agreement in the community because both arguments
are valid.

I add that merge can slow down subtree integration if a change is needed.
>From my point of view, it is OK as it is currently.

It is maybe as difficult as choosing between vim and emacs ;)
(although it seems crazy to use emacs)

Re: [dpdk-dev] git trees organization

[Wiles, Keith]

> On Sep 14, 2017, at 4:18 AM, Thomas Monjalon <thomas@monjalon.net> wrote:
> 
> 14/09/2017 11:03, Bruce Richardson:
>> On Thu, Sep 14, 2017 at 10:22:23AM +0200, Thomas Monjalon wrote:
>>> 14/09/2017 04:25, Stephen Hemminger:
>>>> Bisecting a tree with lots of subtree merges is terrible. That is why Linus
>>>> rebases and doesn't directly take linux-next
>>> 
>>> I agree, bisecting with subtree merges is not pleasant at all.
>>> That's why I chose the rebase method until now.
>>> 
>>> Adrien mentioned some drawbacks with the rebase method.
>>> Ferruh mentioned some drawbacks and some advantages of rebase.
>>> Stephen mentioned another advantage of rebase.
>>> Such decisions are really difficult.
>>> One thing is sure: there will be always someone unhappy,
>>> no matter the decision :)
>>> 
>>> When we want to take such decision or re-consider it,
>>> we ask the techboard to vote...
>> 
>> I'm not sure the techboard needs to vote on this, this is an issue for
>> the tree maintainers/committers is it not? If you do want techboard
>> input on this, I suggest the committers come to an agreement among
>> themselves, with community input, and then just look for tech board to
>> ratify it.
> 
> No, it is an issue for everybody.
> Rebase makes tracking of subtrees difficult for developpers.
> Merge makes reading and bisecting difficult for developpers.
> We cannot have an agreement in the community because both arguments
> are valid.

It seems to me the lesser of the two evils is rebase, but I have not used bisect much. I would think we pick rebase and see how it goes as I know merging can be a bit of a problem. We can always pick merge later if we find a specific issue we need to fix or bisect. I know it is not ideal, but we need to pick one and move on.

> 
> I add that merge can slow down subtree integration if a change is needed.
> From my point of view, it is OK as it is currently.
> 
> It is maybe as difficult as choosing between vim and emacs ;)
> (although it seems crazy to use emacs)

What Emacs does everything for you even writes your code for you, that is why I use vim and visual studio code to write my programs just so I feel useful and not replaced by Emacs :-)



Regards,
Keith

Re: [dpdk-dev] git trees organization

[Stephen Hemminger]

On Thu, 14 Sep 2017 11:11:56 +0200
Nélio Laranjeiro <nelio.laranjeiro@6wind.com> wrote:

> On Thu, Sep 14, 2017 at 10:22:23AM +0200, Thomas Monjalon wrote:
> > 14/09/2017 04:25, Stephen Hemminger:  
> > > Bisecting a tree with lots of subtree merges is terrible. That is why Linus
> > > rebases and doesn't directly take linux-next  
> > 
> > I agree, bisecting with subtree merges is not pleasant at all.
> > That's why I chose the rebase method until now.  
> 
> I don't see what is un-pleasant, if we start a bisect what we expect is to
> find the commit introducing the issue, bisect is able to do it even on large
> tree with a lot of merges.  Moreover, the probability the issue will be
> searched in a specific section within its own subtree is high which also means
> locally it will be linear, is not it equivalent to the actual situation?
> 
> > Adrien mentioned some drawbacks with the rebase method.
> > Ferruh mentioned some drawbacks and some advantages of rebase.
> > Stephen mentioned another advantage of rebase.
> > Such decisions are really difficult.
> > One thing is sure: there will be always someone unhappy,
> > no matter the decision :)
> > 
> > When we want to take such decision or re-consider it,
> > we ask the techboard to vote...  

A recent git bisect gives an example of the problem.

I needed to bisect between two daily versions of linux-next.
Linux-next is intentionally not a serial tree, it is recreated every day.

The big bisect wanted to go through 10,000 commits and back track from
4.14-rc1 into 4.13-rc5 to get down into some subtree. 

On upstream tree it nevers goes back into ancient history.