From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from youngberry.canonical.com (youngberry.canonical.com [91.189.89.112]) by dpdk.org (Postfix) with ESMTP id 949FF2A66 for ; Tue, 1 Dec 2015 13:37:37 +0100 (CET) Received: from 1.general.rbasak.uk.vpn ([10.172.195.6] helo=localhost) by youngberry.canonical.com with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.76) (envelope-from ) id 1a3kBd-00031s-Ao; Tue, 01 Dec 2015 12:37:37 +0000 Date: Tue, 1 Dec 2015 12:37:37 +0000 From: Robie Basak To: Panu Matilainen , dev@dpdk.org Message-ID: <20151201123736.GJ28475@mal.justgohome.co.uk> References: <079fa1cfc3550c8147ea8b137fa1bc0f34d051dc.1448375477.git.pmatilai@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <079fa1cfc3550c8147ea8b137fa1bc0f34d051dc.1448375477.git.pmatilai@redhat.com> Subject: Re: [dpdk-dev] [PATCH] mk: fix the combined library problems by replacing it with a linker script X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Dec 2015 12:37:37 -0000 Re-sending this unsigned since the ML rejected my signed email. -1 from Ubuntu without further discussion since it will break us. Please don't commit this patch yet. I don't understand why we must have the complexity of so many shared libraries. From a distribution packaging perspective, all I see is that this multiplies potential work by twenty times and makes it awkward to work with without special tooling (which then needs maintaining). Before I go into details, it would be nice if someone could please explain why DPDK has to be "special" in needing to do this? I don't understand why DPDK must be different to every other userspace library out there. If DPDK has a good need to be different then that's fair enough. But I feel that if DPDK is deviating from the norm then we need to frame the discussion from the perspective of "why DPDK must be different", rather than having me trying to explain why the norm is the right way to do it. On Tue, Nov 24, 2015 at 04:31:17PM +0200, Panu Matilainen wrote: > That's how Fedora and RHEL are shipping it already and nobody has so much > as noticed anything strange, much less complained about it. 20 libraries > is but a drop in the ocean on a average distro. No, it is 20 times the work from the perspective of DPDK package maintenance. Let me explain why. In Debian and Ubuntu, we manage a library transition (an ABI bump in a library together with all dependencies moving to use the new ABI) by concurrently packaging both the old and new libraries at once. This works well with the norm for libraries. We ship one binary package per soname, with the major version as part of the package name. This allows a system to have two (or more) ABIs installed simultaneously. For a library transition, we just package the new version and then that can land and work concurrently as we then individually update every dependent (library-consuming) package. This works because of conventions around sonames, which DPDK breaks unless we treat it as twenty different libraries which changes our work from easy to painful. Usually a library transition is managed by hand by the package maintainer. It's not taxing because it's straightforward and well understood. Update and upload the new ABI source package, then find all reverse dependencies and sort them out, recursively. But if the maintainer must do it twenty times, then it becomes taxing and prone to error. And if the reverse dependency tree differs depending on the split library used by library consumers, then it gets far more complex to follow. Admittedly we could tool around this to make it easier, but that's extra work (both initially and in maintenance) and prone to error (because we'd only be doing it for DPDK). Packaging a library is usually virtually a no-op in Debian and Ubuntu nowadays. Our tooling does it all for us. But packaging DPDK is far from this currently because of all this added complexity. From my perspective this is unnecessary and makes no sense. We could do all kinds of things to work around it (that's what packaging is about) but then we'd have to maintain that specialness and I don't see why it must be awkward like this instead of just doing it the same way as every other library. > The combined library as it is simply is no longer a viable option. > Besides just being broken (witness the strange hacks people are coming > up with to work around issues in it) its ugly because it basically gives > the middle finger to all the effort going into version compatibility, > and its also big. Few projects will use every library in DPDK, but with > the combined library they're forced to lug the 800 pound gorilla along > needlessly. It's broken because it's broken upstream, and that's what we should fix. Why is it not viable? How does it give the middle finger to effort going into version compatibility? Doing it the right way like every other userspace library is what *gives us* version compatibility because then distributions can straightforwardly install multiple ABI versions at once. Finally, I fail to see any "lug the 800 pound gorilla along" saving. We (Ubuntu and Fedora) are both shipping all the libraries in one package, whether split or combined, so they are all being lugged onto disk anyway. Whether split or combined, there is no saving there. And memory is hardly saved either because the kernel will just page in and out what is needed in both cases. So how does this proposed change give us any saving at all? If distributions are expected to ship everything lumped together on one package, then we don't get any benefit of having the library split up. I did bring this up on this list[1] and my understanding of the outcome then was that it would be fine for us to use the combined library, and in time we could better define its ABI. Thus I'm not happy that you're proposing to change tack on this, both because I'm far from convinced it's a good idea for the project and wider ecosystem and also because it creates more work for us. I understand that in the embedded world you might want to build a subset of the split libraries, and I'm not suggesting that we take away this ability if there users who want it (though in that scenario I think static builds probably make more sense). But in the distribution world, for binaries we ship, I'd prefer to see things work the standard way with standard tooling and nothing special going on when there is no need for it. Please can you explain why having a combined library is so much of a problem? Thanks, Robie [1] Message-ID: <20150902134900.GO467@mal.justgohome.co.uk> Archive: http://dpdk.org/ml/archives/dev/2015-September/023180.html