From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id A9666A045E for ; Thu, 30 May 2019 13:59:49 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 737961B944; Thu, 30 May 2019 13:59:49 +0200 (CEST) Received: from mailout1.w1.samsung.com (mailout1.w1.samsung.com [210.118.77.11]) by dpdk.org (Postfix) with ESMTP id 5F08137A2 for ; Thu, 30 May 2019 13:59:48 +0200 (CEST) Received: from eucas1p1.samsung.com (unknown [182.198.249.206]) by mailout1.w1.samsung.com (KnoxPortal) with ESMTP id 20190530115947euoutp01e296423125ec0cbb50b77712841c9051~jdCJ-5sme3140031400euoutp019 for ; Thu, 30 May 2019 11:59:47 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.w1.samsung.com 20190530115947euoutp01e296423125ec0cbb50b77712841c9051~jdCJ-5sme3140031400euoutp019 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1559217587; bh=b9pxhyl5tg5HntJcgA1RABkrSO2JbvFqoNahBpi/Z8k=; h=Subject:To:Cc:From:Date:In-Reply-To:References:From; b=f11b7yXgszS2cUokZCloWKXWews0L5zfirjGU0QzMOhKzIYs28f4SBjCrn4Bm0osm LMhnE0MMzR0cQPWPOAoMU1lGNo1OtSttclO0LYDbpYFNZ2NcAszlFVRdq3FWp/Dmm9 8hd+QrygLAYHzCqR8Wcizo615a884kvLIYzPrkk8= Received: from eusmges2new.samsung.com (unknown [203.254.199.244]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20190530115946eucas1p110797b053e0ecf558ae20b78ca9a432a~jdCJgI-Z02575725757eucas1p1g; Thu, 30 May 2019 11:59:46 +0000 (GMT) Received: from eucas1p2.samsung.com ( [182.198.249.207]) by eusmges2new.samsung.com (EUCPMTA) with SMTP id 48.0E.04377.2B5CFEC5; Thu, 30 May 2019 12:59:46 +0100 (BST) Received: from eusmtrp1.samsung.com (unknown [182.198.249.138]) by eucas1p1.samsung.com (KnoxPortal) with ESMTPA id 20190530115946eucas1p19832257ee50dfa4c451e438b8cf1ec49~jdCItJZ0l2588825888eucas1p1W; Thu, 30 May 2019 11:59:46 +0000 (GMT) Received: from eusmgms2.samsung.com (unknown [182.198.249.180]) by eusmtrp1.samsung.com (KnoxPortal) with ESMTP id 20190530115945eusmtrp1a9892c8c390ddd0f88fba2f3ba980897~jdCIdnMb22833828338eusmtrp1e; Thu, 30 May 2019 11:59:45 +0000 (GMT) X-AuditID: cbfec7f4-12dff70000001119-f8-5cefc5b2e661 Received: from eusmtip1.samsung.com ( [203.254.199.221]) by eusmgms2.samsung.com (EUCPMTA) with SMTP id 6A.82.04140.1B5CFEC5; Thu, 30 May 2019 12:59:45 +0100 (BST) Received: from [106.109.129.180] (unknown [106.109.129.180]) by eusmtip1.samsung.com (KnoxPortal) with ESMTPA id 20190530115945eusmtip1c6865cc55f9cd0db3d1027a92b0dea58~jdCIBO9Tm0745607456eusmtip1J; Thu, 30 May 2019 11:59:45 +0000 (GMT) To: Luca Boccassi , dev@dpdk.org, Thomas Monjalon Cc: Bruce Richardson , Aaron Conole , Kevin Traynor From: Ilya Maximets Message-ID: Date: Thu, 30 May 2019 14:59:45 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.7.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-GB Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrLKsWRmVeSWpSXmKPExsWy7djP87qbjr6PMVgxT93i15sH7BZzmtax W9xYZW/x7tN2Josr7T/ZLdYe+sBu8enBCRYHdo9fbXOZPX4tWMrqsXjPSyaPYzensXu833eV zaNvyyrGALYoLpuU1JzMstQifbsEroy+teeYC3q9K658W87ewHjQqouRk0NCwERix6L17CC2 kMAKRonpKwW7GLmA7C+MEs+/tLFCOJ8ZJTadvsYG09EwdQozRGI5o8SsfQeg2j8ySkzd4wNi Cws4STyc9I0ZxBYRiJZoeXcUrIZZoFri8vwpYDabgI7EqdVHGEFsXgE7iZ8n54DFWQRUJc6f 6GcFsUUFIiS+7NwEVSMocXLmExYQm1PAQ2L6w7fMEDPFJZq+rGSFsOUltr+dA3achMA2dond k7vYIa52kWif2cwKYQtLvDq+BSouI3F6cg8LhF0vcb/lJSNEcwcwLA79Y4JI2EtseX0OqIED aIOmxPpd+iCmhICjxMxudwiTT+LGW0GIE/gkJm2bzgwR5pXoaBOCmKEi8fvgcmYIW0ri5rvP 7BMYlWYheWwWkmdmIXlmFsLaBYwsqxjFU0uLc9NTi43yUsv1ihNzi0vz0vWS83M3MQIT0el/ x7/sYNz1J+kQowAHoxIPr8DB9zFCrIllxZW5hxglOJiVRHh/Ln8XI8SbklhZlVqUH19UmpNa fIhRmoNFSZy3muFBtJBAemJJanZqakFqEUyWiYNTqoGRL4rVw9iCUUlXu1q1MiXjwSTHGx11 b750SejOXHbCSPKrpETeI81FPRy37jcLc59dzfDMIkRxTkETx9mGZTdCPt1fFr3rQ9uKmNhf lbUrHE7NuLdz6rOjMnIV2QKqGxQmlqVJPf+7aUGVyqbrwlVGtYdWr+mb7iLlf6CSg/dagFp9 ovSHDyVKLMUZiYZazEXFiQBhC9RXQAMAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrMIsWRmVeSWpSXmKPExsVy+t/xu7obj76PMWibZWLx680Ddos5TevY LW6ssrd492k7k8WV9p/sFmsPfWC3+PTgBIsDu8evtrnMHr8WLGX1WLznJZPHsZvT2D3e77vK 5tG3ZRVjAFuUnk1RfmlJqkJGfnGJrVK0oYWRnqGlhZ6RiaWeobF5rJWRqZK+nU1Kak5mWWqR vl2CXkbf2nPMBb3eFVe+LWdvYDxo1cXIySEhYCLRMHUKcxcjF4eQwFJGib55j5kgElISP35d YIWwhSX+XOtigyh6zyjxseklO0hCWMBJ4uGkb8wgtohAtMTXputgDcwC1RK/LvezQDR8YpLY 9nMiI0iCTUBH4tTqI2A2r4CdxM+Tc8AGsQioSpw/0Q/WLCoQITF7VwMLRI2gxMmZT8BsTgEP iekP3zJDLFCX+DPvEpQtLtH0ZSXUYnmJ7W/nME9gFJqFpH0WkpZZSFpmIWlZwMiyilEktbQ4 Nz232EivODG3uDQvXS85P3cTIzACtx37uWUHY9e74EOMAhyMSjy8AgffxwixJpYVV+YeYpTg YFYS4f25/F2MEG9KYmVValF+fFFpTmrxIUZToOcmMkuJJucDk0NeSbyhqaG5haWhubG5sZmF kjhvh8DBGCGB9MSS1OzU1ILUIpg+Jg5OqQbGCAvG5Wn3g4tuHOfu3a1nkv5EJEj2Ad8lZqms myuu1iXHLIwO/fni3u6VhUs/n5oX+JSl7td564Or2WbE/kuIsV3n9mnOIaWI38rbn796G1P7 4gujV5v+QcYzzsdYVU+KunGtrA65sm7/1OL4P5NDc7wi1wra32ETsjER0VVMjS2KY5ue0dOg xFKckWioxVxUnAgApeMMytYCAAA= X-CMS-MailID: 20190530115946eucas1p19832257ee50dfa4c451e438b8cf1ec49 X-Msg-Generator: CA Content-Type: text/plain; charset="utf-8" X-RootMTR: 20190529164009eucas1p289f1dcf87012ecf049efc8eee2c2ea9d X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20190529164009eucas1p289f1dcf87012ecf049efc8eee2c2ea9d References: <20190529163958.30796-1-i.maximets@samsung.com> <20190529163958.30796-3-i.maximets@samsung.com> <1a07d1cd59d84dce84e56c10fdabf5e5504560a6.camel@debian.org> <0cc060dc-36e6-88ea-491e-e3b9c552bd7a@samsung.com> Subject: Re: [dpdk-dev] [PATCH 2/2] meson: make build configurable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 30.05.2019 14:06, Luca Boccassi wrote: > On Thu, 2019-05-30 at 13:03 +0300, Ilya Maximets wrote: >> On 29.05.2019 23:37, Luca Boccassi wrote: >>> On Wed, 2019-05-29 at 19:39 +0300, Ilya Maximets wrote: >>>> The first thing many developers do before start building DPDK is >>>> disabling all the not needed divers and libraries. This happens >>>> just because more than a half of DPDK dirvers and libraries are >>>> not >>>> needed for the particular reason. For example, you don't need >>>> dpaa*, octeon*, various croypto devices, eventdev, etc. if you're >>>> only want to build OVS for x86_64 with static linking. >>>> >>>> By disabling everything you don't need, build speeds up literally >>>> 10x >>>> times. This is important for CI systems. For example, TravisCI >>>> wastes >>>> 10 minutes for the default DPDK build just to check linking with >>>> OVS. >>>> >>>> Another thing is the binary size. Number of DPDK libraries and, >>>> as a result, size of resulted statically linked application >>>> decreases >>>> significantly. >>>> >>>> Important thing also that you're able to not install some >>>> dependencies >>>> if you don't have them on a target platform. Just disable >>>> libs/drivers >>>> that depends on it. Similar thing for the glibc version mismatch >>>> between build and target platforms. >>>> >>>> Also, I have to note that less code means less probability of >>>> failures and less number of attack vectors. >>>> >>>> This patch gives 'meson' the power of configurability that we >>>> have with 'make'. Using new options it's possible to enable just >>>> what you need and nothing more. >>>> >>>> For example, following cmdline could be used to build almost >>>> minimal >>>> set of DPDK libs and drivers to check OVS build: >>>> >>>> $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false >>>> \ >>>> -Ddrivers_bus=pci,vdev \ >>>> -Ddrivers_mempool=ring \ >>>> -Ddrivers_net=null,virtio,ring \ >>>> -Ddrivers_crypto=virtio \ >>>> -Ddrivers_compress=none \ >>>> -Ddrivers_event=none \ >>>> -Ddrivers_baseband=none \ >>>> -Ddrivers_raw=none \ >>>> -Ddrivers_common=none \ >>>> >>>> -Dlibs=kvargs,eal,cmdline,ring,mempool,mbuf,net,meter,\ >>>> ethdev,pci,hash,cryptodev,pdump,vhost \ >>>> -Dapps=none >>>> >>>> Adding a few real net drivers will give configuration that can be >>>> used >>>> in production environment. >>>> >>>> Looks not very pretty, but this could be moved to a script. >>>> >>>> Build details: >>>> >>>> Build targets in project: 57 >>>> >>>> $ time ninja >>>> real 0m11,528s >>>> user 1m4,137s >>>> sys 0m4,935s >>>> >>>> $ du -sh ../dpdk_meson_install/ >>>> 3,5M ../dpdk_meson_install/ >>>> >>>> To compare with what we have without these options: >>>> >>>> $ meson build -Dexamples='' -Dtests=false -Denable_kmods=false >>>> Build targets in project: 434 >>>> >>>> $ time ninja >>>> real 1m38,963s >>>> user 10m18,624s >>>> sys 0m45,478s >>>> >>>> $ du -sh ../dpdk_meson_install/ >>>> 27M ../dpdk_meson_install/ >>>> >>>> 10x speed up for the user time. >>>> 7.7 times size decrease. >>>> >>>> This is probably not much user-friendly because it's not a >>>> Kconfig >>>> and dependency tracking in meson is really poor, so it requires >>>> usually few iterations to pick correct set of libraries to >>>> satisfy >>>> all dependencies. However, it's not a big deal. Options intended >>>> for a proficient users who knows what they need. >>> >>> Hi, >>> >>> We talked about this a few times in the past, and it was actually >>> one >>> of the design goals to _avoid_ replicating the octopus-like config >>> system of the makefiles. That's because it makes the test matrix >>> insanely complicated, not to mention the harm to user friendliness, >>> among other things. >>> >>> If someone doesn't want to use a PMD, they can just avoid >>> installing it >>> - it's simple enough. >> >> So how can I do this? I don't think 'ninja install' has such option. >> Also, if you think that it is safe to skip some libs/drivers in >> installation >> process, it must be safe to not build them at all. It's just a waste >> of >> time and computational resources to build something known to be not >> used. >> And if you're going to ship DPDK libraries separately in distros, >> you'll >> have to test their different combinations anyway. If they're so >> independent >> that you don't need to test them in various combinations, than your >> point >> about test matrix is not valid. > > It can be done in the packaging step, or post-install if there's no > packaging. An operating system vendor is free to do its own test and > support plan, and decide to leave out some PMDs from it. This technically means doing this manually/write custom scripts. > Canonical does > something similar: in Debian/Ubuntu and derivatives we package PMDs > individually, and then Canonical groups them in 2 sets - a subset they > guarantee to support from their own resources, and the full set as > delivered by the community. But the point is that it's a step that they > decide to take, and pay the price for it in terms of time investment in > validating that particular combination, rather than the onus being on > our very limited and already stretched resources to validate all > combinations. > > We can focus our resources into making sure the few combinations that > _must_ be supported, for example due to external dependencies, work > fine. > >>> Sorry, but from me it's a very strong NACK. >> >> Sorry, but let me disagree with you. For me, meson configurability is >> the >> essential thing to have in terms of deprecating the 'make' build >> system. >> DPDK was and keeps being (in most cases) the library that users >> statically >> linking to a single application built for particular platform and not >> using >> for anything else. This means that user in most cases knows which >> parts >> needed and which parts will never be used. Current meson build system >> doesn't allow to disable anything forcing users to link with the >> whole bunch >> of unused code. >> >> One major case is that you have to have build environment equal to >> your >> target platform in terms of availability of external libraries. So, >> if I >> have some external library on build system, meson will build all the >> modules >> it depends from and will link them to my application. As a result >> I'll not >> be able to run my application on a target platform without installing >> additional dependencies which is not acceptable. This patch will >> allow to >> specifically disable all the libs that has unsatisfiable dependencies >> on >> target. Without the patch it's required to manually remove resulted >> libs and >> fix pkg-config and stuff before building apps. This is far less user- >> friendly >> than options I proposed. And yes, I still have to waste time for >> building >> libraries I'll remove right after. >> >> While testing OVS on TravisCI, DPDK was built far more than 30K times >> which is more than half of a year of a wasted computational resources >> (if we'll count 10 minutes per build). >> I think this time could be used more wisely. >> >> Best regards, Ilya Maximets. > > But that's the thing: as it was discussed recently, we need to move > away from DPDK being by default a "special sauce" toolkit with millions > of customizations that is custom built and statically linked, like > busybox, and to a situation where it's just another set of system > libraries like any other, shipped by the operating system, like glibc - > see the threads about stable API/ABI and OS-driven delivery by Ray. > The status quo of an insanely granular build configuration that means > everyone is using something different from each other is a bug - not a > feature. Excessive _build time_ configuration exacerbates and > encourages this bug. > > Sure, an initial build from scratch will take a couple of minutes more > - mildly annoying, but worth the price. Caching takes care of that > problem already pretty well. Also standardizing on radically fewer > build configurations means you _don't_ have to rebuild DPDK for third > party testing - you just consume the pre-built binaries from the OS of > choice, or the PPA or similar for backports and HEAD builds. That will > save even more time and resources in third-party build systems. > CI systems usually build everything from scratch. So caching doesn't help. And I don't think that distros will have any pre-built binaries for the old systems used in public CI systems. For example, TravisCI has 'Trusty' as a default environment. At most you may use 'Xenial'. But I don't think that DPDK 18.11 will ever be packaged for them. Also, distros mostly has dynamic libraries in their packages not providing static ones. It might be changed in the future, but still we'll not have them for 'xenial'. So, OVS will stuck with slow building DPDK each time from scratch on Travis. And developers will maintain local patches for meson for stripping extra libraries or local build scripts that will package only what they need. Best regards, Ilya Maximets.