From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <adrien.mazarguil@6wind.com> Received: from mail-wm0-f46.google.com (mail-wm0-f46.google.com [74.125.82.46]) by dpdk.org (Postfix) with ESMTP id DDEC9325B for <dev@dpdk.org>; Wed, 13 Sep 2017 16:54:13 +0200 (CEST) Received: by mail-wm0-f46.google.com with SMTP id a137so8807776wma.0 for <dev@dpdk.org>; Wed, 13 Sep 2017 07:54:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=Xn3kE/WcZP5qHSB7OMyIBGh9wRkC9T02DlxHqO1zN/I=; b=tMuoChy9u77J76rMhS4BER5iAxIkDtvh9bbTpSwZpyeR695wmy5mFYTV7p85N9xzwt qcdwyBxN6gYCGVX+6yhrH4r9Zo3onrYjQANK4L3ZmnYzYN4FB6TIhaTjchVQIJQcrvKR n5tYlipzZOGp1UPulL5LYQ242ZAgOdVpESuaHMMRrAIuL4IXRTw5SvzpARc+axq+ACBE kce9HjPfkdVZCOUxPujpRVRlb7zCL+2PnWxiwLj231T0W8tFmB7X3O3Azd6iI2e+PJKC kMi5RaopLWnJtBN6RXWuVzP5QH75sZxcmfNGWLNzLtJSK8jJAW+rTKuUkMt1XINS03L2 jsBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=Xn3kE/WcZP5qHSB7OMyIBGh9wRkC9T02DlxHqO1zN/I=; b=Nkiu58DRoMmNNoTOSR/FPSAnUabgsmF7F3vEuYY8KjtU73LpVsbcstu38zdwq21Hcr 46ABARLbsic3XhqV986UeehE7HQ74bmvcez1mFJIneL3OSFCd1BK50kfekngSWkCOtlR DsryiYXOaY3A4BHJwZhpEVO7MQZUGf722GCiP7Nc/5yuy6VmqU7Ssf2xDrQ4YVJudwOm 9Y5AcLqkanWXsTHZPm45fieffkoN4cgFX8NwswzW5Y+omXjGxW0nJeBBCaOb6lJC29cO PJ7yKwlxLWSGELl9QaJDCEHkbyadNJGdyLMwXd61gIA0cC86ppyeVMRd8Oo+OqLvpxAJ m3GA== X-Gm-Message-State: AHPjjUh4Ex2HJO5We9NhyYSxKvxEdAn2QPhbDuMVfp1Gh156ooiYJIfj p6aWW5e0yuWVEmUl X-Google-Smtp-Source: AOwi7QCoBrs+g6VS6TxRCX5aZGBE5JJnn+tHJybWtZ/ER7Q+FgK8v8KpKVJdfN89Ql0kJqZkEpK3Cg== X-Received: by 10.28.9.135 with SMTP id 129mr2941979wmj.93.1505314453459; Wed, 13 Sep 2017 07:54:13 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id l72sm2093615wmi.1.2017.09.13.07.54.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Sep 2017 07:54:12 -0700 (PDT) Date: Wed, 13 Sep 2017 16:54:02 +0200 From: Adrien Mazarguil <adrien.mazarguil@6wind.com> To: Ferruh Yigit <ferruh.yigit@intel.com> Cc: Bruce Richardson <bruce.richardson@intel.com>, Thomas Monjalon <thomas@monjalon.net>, stephen@networkplumber.org, dev@dpdk.org Message-ID: <20170913145402.GA2481@6wind.com> References: <2737351.pD9poAUtZC@xps> <20170912083207.GC40060@bricha3-MOBL3.ger.corp.intel.com> <20170913075845.GR2481@6wind.com> <a7de54a6-7f59-6192-7d41-f2ed6ebb2339@intel.com> <20170913122517.GZ2481@6wind.com> <296a5b75-6581-3877-5b3b-4f38d739b6da@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <296a5b75-6581-3877-5b3b-4f38d739b6da@intel.com> Subject: Re: [dpdk-dev] git trees organization X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <http://dpdk.org/ml/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <http://dpdk.org/ml/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Wed, 13 Sep 2017 14:54:14 -0000 On Wed, Sep 13, 2017 at 02:21:00PM +0100, Ferruh Yigit wrote: > On 9/13/2017 1:25 PM, Adrien Mazarguil wrote: > > On Wed, Sep 13, 2017 at 12:38:37PM +0100, Ferruh Yigit wrote: > >> On 9/13/2017 8:58 AM, Adrien Mazarguil wrote: > >>> Hi, > >>> > >>> On Tue, Sep 12, 2017 at 09:32:07AM +0100, Bruce Richardson wrote: > >>>> On Tue, Sep 12, 2017 at 12:03:30AM +0200, Thomas Monjalon wrote: > >>>>> Hi all, > >>>>> > >>>>> As you know I am currently the only maintainer of the master tree. > >>>>> It is very convenient because I need to synchronize with others > >>>>> only when pulling "next-*" trees. > >>>>> But the drawback is that I should be available very often to > >>>>> avoid stalled patches waiting in patchwork backlog. > >>>>> > >>>>> I feel it is the good time to move to a slightly different organization. > >>>>> I am working closely with Ferruh Yigit for almost one year, as next-net > >>>>> maintainer, and I think it would be very efficient to delegate him some > >>>>> work for the master tree. > >>>> > >>>> I think Ferruh has been doing an excellent job on the net tree, and > >>>> would be an excellent candidate to help with the workload on the master > >>>> tree. > >>>> > >>>>> I mean that I would use the patchwork delegation to explicitly divide > >>>>> the workload given our different experiences. > >>>>> Ferruh, do you agree taking this new responsibility? > >>>>> > >>>>> At the same time, we can think how to add more git sub-trees: > >>>> > >>>> In principle, I'm in favour, but I think that the subtrees of the master > >>>> tree should be at a fairly coarse granularity, and not be too many of > >>>> them. The more subtrees, the more likely we are to have issues with > >>>> patchsets needing to be split across trees, or having to take bits from > >>>> multiple trees in order to test if everything is working. > >>> <snip> > >>> > >>> About that, how about we start allowing true merge commits instead of > >>> rebasing (rewriting history) in order to ease things for maintainers? > >>> > >>> This approach makes pull requests show up as a merge commits that contain > >>> the (ideally trivial) changes needed to resolve any conflicts; this has the > >>> following benefits: > >>> > >>> - The work done by a maintainer during that merge is tracked, not silently > >>> ignored or lost. The merge commit itself is signed-off by its author. > >>> > >>> - This allows tracing mistakes or bugs to the conflict resolution itself. > >>> > >>> - Upstream can reject pull requests on the basis that merging it is not > >>> trivial enough (i.e. downstream must merge upstream changes first). > >>> > >>> - Sub-trees can merge among themselves in case they need features that > >>> encompass several trees, not necessarily always against the master > >>> tree. Everything is tracked. > >>> > >>> - Maintainers do not ever modify the commits they get from other trees, > >>> which keep their SHAs unmodified as part of the history. A given commit ID > >>> is truly unique among all trees (back-port trees remain the only exception > >>> since commits are cherry-picked). > >>> > >>> - It shifts the entire responsibility to the maintainers of sub-trees. > >>> > >>> The only downside is that commits have several parents, history becomes a > >>> graph that developers need to get used to (some might call it a mess), > >>> however that's probably not an issue for those already used to Linux kernel > >>> development and other large projects. > >>> > >>> I know this was already discussed in the past, however I think adding more > >>> sub-trees will make rebasing too complex otherwise> > >>> Thoughts? > >>> > >> > >> Using git merge looks more proper git usage, but I have one question / > >> concern: > >> > >> For next-net, sometimes there are dependent patches in main tree, and > >> what I am doing is rebasing sub-tree on top of latest main tree. > >> > >> When switched to merge method, how dependent patches can be get into the > >> sub-tree? Merge from main tree to sub-tree? > > > > Yes, that's the idea. On the other hand, as a maintainer, you are not > > responsible for the contents of what's merged from other official > > trees. Commits are taken as they are, this implies trust between tree > > maintainers. > > > >> Won't this bidirectional merging confusing? > > > > Probably at first, this certainly needs some getting used to. We can attempt > > to avoid such merges as much as possible with proper coordination. Avoiding > > them is likely not possible though if we want to keep history intact. > > Let's assume I need to merge from main tree to next-net three times > before the integration, because of dependencies. > When Thomas merged next-net to main tree for rc1, will those three merge > commits visible in main tree? Yes, all merge commits will remain visible and part of history. There's only one case when such commits are optional: fast-forwards. For instance assuming the HEAD of your current branch is part of upstream's history, you may not get a merge commit if you pull from upstream before applying a series instead of doing the reverse. Whoever gets the fast-forward wins, therefore merging often is better. Even with many subsequent merges, it's not all that bad. Graph views such as "git log --oneline --graph" help a lot in clarifying things (once you get used to them). > For Linux I guess sub-trees not merging from Linus' tree because of > dependencies, I assume they are only merging after release to get new > commits not in their sub-tree. Linux does that all the time and not necessarily after a release, even among sub-trees. We don't plan to have as many different trees as Linux so it should remain much simpler for us in any case. > But for DPDK main tree is also getting new patches on its own. Same for Linux even if those come in minority, I think it's not a problem either way, Git is really good at merging histories. Note that a maintainer can also add glue commits of his own on top of his tree in order to simplify a subsequent merge. This is better than addressing everything in the merge commit itself in non-trivial cases (although it should be the job of the requester). > >> And following are notes from my current experience: > >> > >> - Having re-writable history gives some flexibility to sub-trees. > >> Possible to update commit logs and amend patches even after pushed. > > > > This is both a good and a bad thing. Thanks to that, history is currently > > linear and extremely clean, I think we haven't had a single patch that > > doesn't compile or a bad merge artifact for a very long time. > > > > On the other hand, if you look at the effort required to maintain a single > > sub-tree that way, it likely becomes exponential for several. All of them > > need to constantly rebase their stuff, with conflicts to address. The more > > people, the more mistakes and so on. > > > > Once part of an official tree, a commit cannot be amended anymore, it's too > > late; revert commits will be more common. We need to accept this first. > > > >> - It is hard to confirm pulled commits in main tree, I guess merge > >> commit will make this easier. > >> > >> - To track main tree, continuously rebasing and continuously re-writing > >> history, I am doing this almost daily, this may be hard for people > >> working on top of next-net. > > > > Yes that's one of the "bad" things with the current approach. Consider > > automatic non-regression testing against disappearing commit IDs. Tracking > > their status is difficult when history is changing. For instance we usually > > add local tags to track CI successes/failures. All of them now point to > > nonexistent commits. > > > >> - Conflict resolving done by sub-trees during rebase, instead of done by > >> main tree during merge. So this may be more distributed effort. > > > > It's not too different actually. With merge you can reject PRs on the basis > > that there are too many conflicts to take care of, not unlike a request for > > rebase. > > > > On the plus side if you make any changes yourself in order to solve them, > > they are made part of the merge commit, not in the original commits which > > remain unmodified. > > You are right, we are loosing original code when there is merge > conflict, and I think there is no way to trace back to the original > commit in repo, but from mail list and patchwork perhaps. > > And very hard to find out what has been changed for conflict resolving, > and if something went wrong! > > > > >> - Rebasing gives more straight forward history in main repo, merge > >> commits looks more confusing, although I would expect it won't be as > >> complex as Linux tree, so may not be a problem. > > > > Right, that's the main drawback. I think there's no way to know the impact > > unless we attempt it as an experiment. -- Adrien Mazarguil 6WIND