From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CBDA8432B9; Mon, 6 Nov 2023 11:28:53 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B998F402B6; Mon, 6 Nov 2023 11:28:53 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 95F7A402AA for ; Mon, 6 Nov 2023 11:28:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1699266532; x=1730802532; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=LebhchGzmd2w8DM9rxAwY1Nb/gAnUVt8cj6O6NCRcJE=; b=MSRHSrKRGHf13i2fX0HmZv3ELgQ5cKIbDcX4odz+i7sjQz/BYjTV2e59 j21mzx2hyburDVa8EcfSsejGAeYDELV+uP83Etg+f5pN7ktSOpwtM2g2i bq79cIM4kE/wK13272RsokpUKuKXqvBw8oiyapvtSHedKAC7O3eSfRyKw jM3kB1cGN43cIqYgRCZyrnwlI6WHq3uqPhu2ut+Qm6FgUguXNfsEzU8lX aVLUqqfQzbJxx7d8JEULlswVBaCksz63btvF1FPYFuW4lFoD/jfGZQtwU QE6VFu/4YAhTeOCLCV1wy/oyrPJh6Pi6XL3gM+Qovnh7dj/88POI9FG0r g==; X-IronPort-AV: E=McAfee;i="6600,9927,10885"; a="392116944" X-IronPort-AV: E=Sophos;i="6.03,281,1694761200"; d="scan'208";a="392116944" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Nov 2023 02:28:51 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10885"; a="832683602" X-IronPort-AV: E=Sophos;i="6.03,281,1694761200"; d="scan'208";a="832683602" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by fmsmga004.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 06 Nov 2023 02:28:50 -0800 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.34; Mon, 6 Nov 2023 02:28:49 -0800 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.34 via Frontend Transport; Mon, 6 Nov 2023 02:28:49 -0800 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (104.47.70.101) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.34; Mon, 6 Nov 2023 02:28:48 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Cr3qpBr04dxkgMxLjcCs45TbPqtKUTKXaEWx2CvQXvXTbS5TvbXnxYDiaWSPFOwGOBtixueahHdVaF/oTR1NUrojFYadQvGS0e2OpZZwLRLrEgotAGbiaVOz4Uqw3WGmYIeoBjNrxrh+yDLwqMhvGnyQgaz+XCzrOz+fV96ZvA+jKsIxIa0e1TwRXJHD4XPmAUqGp/582zqCmrLaViqMGanPcuUEAn6P2Eq2tpjbRyrzS4iKHiL1ufRVR3aaKlufOo6dca+NdskcYpyDm9x8ndbrlbBjIkEFBuFNC44Oi6gpjnAaFqNnEZjeIE0Jr6cqO2QvUG9ll29eFwv30IXgig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Vrj/rFRU+pjTQajD2VvrzOoO529aXQPFzipmBVguLGA=; b=bldFO9plmjjb4E4N7+T6UMIeIequz5DAcnCLVsCr9OiHwNizKGic79uE4tWr65dfHr78aiZp3dWhJOKHmy9GYlZRn+s+SgOSXYNTuAfLVftzdthH0u3YhMcJfpZgixfRXfs28cDRnzQ55elaQZf5oIL/R61p5JFnizQ/R4hBW0gWOi5gKCWDcu0ZwlFovvPAMjrllfTgfW3dP7rad9/DuGwYp6QfixFpX4DPIWGVXw33uycxn5KMKFwIt4gH2+WNBXlRVqUJLkF0z7yTCWdzGvPr9pLZ/sn0MYqg/BkuIXMkXQGHgGM8ITEn7GyLHqXnmbTI6QQIkqhqAStv1G/hOA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by SA1PR11MB8374.namprd11.prod.outlook.com (2603:10b6:806:385::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.28; Mon, 6 Nov 2023 10:28:45 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::8645:d921:ce8a:12ba]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::8645:d921:ce8a:12ba%5]) with mapi id 15.20.6954.028; Mon, 6 Nov 2023 10:28:45 +0000 Date: Mon, 6 Nov 2023 10:28:40 +0000 From: Bruce Richardson To: Morten =?iso-8859-1?Q?Br=F8rup?= CC: Subject: Re: [PATCH 24.03 v2] build: track mandatory rather than optional libs Message-ID: References: <20231103162830.593702-1-bruce.richardson@intel.com> <20231103165208.1210182-1-bruce.richardson@intel.com> <98CBD80474FA8B44BF855DF32C47DC35E9EFCF@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35E9EFD0@smartserver.smartshare.dk> Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35E9EFD0@smartserver.smartshare.dk> X-ClientProxiedBy: DU2P251CA0005.EURP251.PROD.OUTLOOK.COM (2603:10a6:10:230::34) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|SA1PR11MB8374:EE_ X-MS-Office365-Filtering-Correlation-Id: 125657a4-296b-4ba0-a76c-08dbdeb3281e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: G2H9MDFWXL/dXWCOBZqYZ9UnGQgBIvFS8QRdbqLvq865hU0nhxFzrgCj3nW45iL+hnS4Q7XwY7KNIAVfQ2vJY3MFlKdPGY1Q5uQ9Vt+LDfANhTiZ38FsYLHkGWxHQU4xSK4U+rVanDYWA1b4wlpBiF9bfnz/sbkfRo1LM7tJxF2JTi7vGbJ9njDbYa0uLVIBvPBBg+m0uMiZvJV+N8kQkZWxHYmmY+112U59NHA+m81/xI1OwEf7GBd+umReFGazcCwX54byV6rdChbjQF7Jkr1EH4LkKLSZTx88dZhz3Es2ULLgBJzfEFfeUusfRpuZEf1F2SAW/SfKuob3eoJDDqcS4/ueNLotiHSseDrRXtt2KcT7CQTllIANWPsAh0NrUa4ayclxHUskPkrPFjdQ5lGON+2KDC+qvzE/I/xWQzPE5R0f/xgJDpbOCidLZ0sFEmT5NidFq3UQPZ9Abj+T28DlAsQ8PbJfkvPTsFW6ykd4UKUNLZXukRAD1xYyvlz8UqljR0Ha816hfgvHoPaUzjb8Ur9x1FrqMIB5Slf4qL/0ki3LT6DKS+a5knsy2lj1 X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(39860400002)(366004)(396003)(376002)(346002)(136003)(230922051799003)(186009)(451199024)(1800799009)(64100799003)(2906002)(41300700001)(6916009)(478600001)(316002)(66476007)(66556008)(66946007)(38100700002)(8936002)(8676002)(4326008)(86362001)(5660300002)(44832011)(6486002)(83380400001)(66574015)(82960400001)(6666004)(6506007)(26005)(6512007); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?BCWPZoTKWZg4+gKfsn3oP0eFriuUieFTabfahCLzaqQgKSQDHlxJb7cKKb?= =?iso-8859-1?Q?28s4iov35BW7/amuN+fEmzuS+mZM44Mgp26K2HALEKTVY9nSHbiatFZ/IZ?= =?iso-8859-1?Q?3QyMmgXDAdlfAaBrbxsiUis7t89HeT1lYMiVQ66IG5fgj0Rw7cQdajXwlt?= =?iso-8859-1?Q?4o88hcxKuFBkIiV0AytTVbPuSY1n9N+0xIp5m10p1UFOSLihOVyRGB259K?= =?iso-8859-1?Q?hamVblHK7fkQMQo8GbiUeamZMbjrx2LS4vOk7KZGHkWSqVPPQhnTgVF6/j?= =?iso-8859-1?Q?giiGXaMYg+JvCPu2OTEC/xbpPzO8Z4OFZAkxRO77fcaQfPdeNfzvrv6nO2?= =?iso-8859-1?Q?s47vj0sg3KgXxmqotREMAKF1eLxDePLWh5U0OyyvenT6bnNIcwKIywexqB?= =?iso-8859-1?Q?UkH/UOF80mGoRK/+cHYAAWb7T+gMlCUhAANBgohvnF17ZfaafZsZi/0oro?= =?iso-8859-1?Q?9gi0vWFwf5Z+lEizRE+LHfvRizm+8JjNYz9o5PZzXckmKuAcVqJquzRdRF?= =?iso-8859-1?Q?CmuPy13e5WfGRamAw2kiIu+Cfr9nbAuI6itPcH3l8dWTZElLFbe1Tumm+y?= =?iso-8859-1?Q?SGpJBTGuaECgPU86xoHL8vOzV/7s0DzDYpFcB4SKcacy0JnnXw2uw3p44K?= =?iso-8859-1?Q?aaNbfbvyPecGx3YCsmiAsC8TIr7YeDNxMNpHfKjn5gf/hCLocxQHtBdsMK?= =?iso-8859-1?Q?ys0+kW5LqvuraWnmyXsvuQp2punTlHBpKTJXxh89jIZ/TGvE716q1eGqFi?= =?iso-8859-1?Q?zX4PlcRGJVq03+7Kae9Z3wlU3Bn9dmtIhAVxDBlCly9vIdm0D3uNOBxwRj?= =?iso-8859-1?Q?J54zmmpELnYRCD77zTqiyN6fa2olwGdrfPJK915+sDzZ60eCpZp2Un8YL1?= =?iso-8859-1?Q?ehS4fwOgyg5EJhM2smacMTfVHzCoJi6uR5PsJaoZVoPvsWeUlr8j9rNSMD?= =?iso-8859-1?Q?SArxJVLwmSJ2F7l4Sn24T1pnw5eJOezWEzUcKZlWEOLqm2y7Nv/PdkjXL5?= =?iso-8859-1?Q?L+UQU8wGLeIgS7Ii4AXP85hHDqsTH+xmB5GndJQz+pxJBVOH2JA+MDcQFQ?= =?iso-8859-1?Q?jrSRGn1ZGD85sY+Dz+0yaSwnaqKBVZawJHoVnolNzUl8YMDK69l0ipEbu+?= =?iso-8859-1?Q?Q2HnozCWUxbAwBkDHUJlwySJUkxTJmhHqQpitoFjINjkVWTzBuqrDHr4oa?= =?iso-8859-1?Q?THg8YRqVgXUbPYTeS3iXV4/i/J9bsGoSMBd4edhlVoLfmX1yLfvF/0c8GD?= =?iso-8859-1?Q?t0sxsFF3PZM4hfVQvfefNx7r9JkFtmf3SQqXF2FrAsDu6rGzc4t+8W7vRw?= =?iso-8859-1?Q?t41o7bp3p4wRPupJn0+5YL/k2bJaMwyzwbNdNPh9QqIWrjR3c7yFjmrwqZ?= =?iso-8859-1?Q?jteg069+b7cXvWM9aGTSTTluVUaLKIVF3S9EV15nUc8jzWbRSCmVVnintf?= =?iso-8859-1?Q?o8nQkqmRBymkyWKJGwiKWE7nP/ultvr2/ZMXx5aNfxr6wPYQ1uj8O4J2DQ?= =?iso-8859-1?Q?ZLGjobcruV2hioD49iYJUtVLt524LlmqVRRjLoQWURg/bvicIlkobu1P32?= =?iso-8859-1?Q?ENj8O2aOWEXmy9X+Ln89liRlnTnwxP3ijTX9UnOkHFv/LMD7q/8AHTpv0Q?= =?iso-8859-1?Q?pFzshLVI1GOK8OBOquOeMnpmpN9/FgpDrZKo+N1cboLmO65DOKlhc0QQ?= =?iso-8859-1?Q?=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 125657a4-296b-4ba0-a76c-08dbdeb3281e X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Nov 2023 10:28:45.7782 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MYnrKIG9Q5wZVCCDPREqwnfKd0xwFhk+Wsh7wac7BbsHtx7y8Zu48+AevxLZLLtn+ToTLYEBOpQGmRwh4JV7ccQYgPH9d7DqxP6SrJOEqcg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB8374 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Fri, Nov 03, 2023 at 09:19:53PM +0100, Morten Brørup wrote: > > From: Bruce Richardson [mailto:bruce.richardson@intel.com] > > Sent: Friday, 3 November 2023 19.09 > > > > On Fri, Nov 03, 2023 at 06:31:30PM +0100, Morten Brørup wrote: > > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com] > > > > Sent: Friday, 3 November 2023 17.52 > > > > > > > > DPDK now has more optional libraries than mandatory ones, so invert > > the > > > > list stored in the meson.build file from the optional ones to the > > > > "always_enable" ones. As well as being a shorter list: > > > > > > > > * we can remove the loop building up the "always_enable" list > > > > dynamically from the optional list > > > > * it better aligns with the drivers/meson.build file which > > maintains an > > > > always_enable list. > > > > > > > > Signed-off-by: Bruce Richardson > > > > > > Excellent! > > > > > > It really shows how bloated DPDK CORE still is. I would like to see > > these go optional: > > > > > > > For some I agree, but we need to decide what optional really means. :-) > > > > For my mind, there are 3 (maybe 4) key components that need to be built > > for > > me to consider a build to be a valid DPDK one: > > * EAL obviously, > > * testpmd, because everyone seems to use it > > * l3fwd, becaues it's the most commonly referenced example and used for > > benchmarking, and build testing in test-meson-builds. (There are > > others, > > but they are pretty likely to build if l3fwd does!) > > * dpdk-test - I feel this should always be buildable, but for me it's > > the > > optional 4th component. > > > > Now, the obviously one to relax here is l3fwd, since it is just an > > example, > > but I wonder if that may cause some heartache. > > I don't consider any DPDK lib CORE just because the lib is used by testpmd and/or l3fwd. I agree that all libs should be included by default, so you can run testpmd, l3fwd, and other apps and examples. > > However, many libs are not needed for *all* DPDK applications, so I would like other apps to be able to build DPDK without superfluous libs. > > E.g. our StraightShaper CSP appliance is deployed at Layer 2, and doesn't use any of DPDK's L3 libs, so why should the DPDK L3 libs be considered CORE and thus included in our application? I suppose other companies are also using DPDK for other purposes than L3 routing, and don't need the DPDK L3 libs. > > Furthermore, I suppose that some Layer 3 applications use their own RIB/FIB/LPM libraries. Does OVS use DPDK's rib/fib/lpm libraries? > > > Overall, if we want to make more libs optional, I would start looking > > at > > l3fwd and making it a bit more modular. I previously made its support > > for > > eventdev optional, we should do the same for lpm and fib. Beyond that, > > we > > need to decide what core really means. > > Yes - defining CORE is the key to setting the goal here. > > In my mind, CORE is the minimum requirement to running an absolutely minimal DPDK application. > > A primary DPDK application would probably need to do some packet I/O; but it might be a simple layer two bridge, not using any of the L3 libs. > > And a secondary DPDK application might attach to a primary DPDK application only to work on its data structures, e.g. to collect statistics, but not do any packet processing, so that application doesn't need any of those libs (not even the ethdev lib). > > In reality, DPDK applications would probably need to build more libs than just CORE. But some application might need CORE + lib A, and some other application might need CORE + lib B. In essence, I don't want application A to drag around some unused lib B, and application B to drag around some unused lib A. > > It's an optimization only available a build time. Distros should continue providing all DPDK libs. > > There's also system testing and system attack surface to consider... all that bloat makes production systems more fragile and vulnerable. > I largely agree, though I do think that trying to split primary-secondary as having different builds could lead to some headaches, so I'd push any work around that further down the road. Some thoughts on next steps: * From looks of my original list above, it appears the low-hanging fruit is largely gone, in terms of being able to turn off libs that have few dependencies, timer being one possible exception * I think it's worth looking into making l3fwd more modular so it can be build only with backend x or y or z in it. However, if agreeable, we can just start marking lpm and rib/fib libs as optional directly and have l3fwd not buildable in those cases. * For libs that depend on other libs for bits of functionality, we are getting into the realm of using ifdefs to start selectively removing bits. This is the not-so-nice bit as: - it makes it a lot harder to do proper build testing, as we now have to test with individual bits on and off. So long as we were just enabling/ disabling whole components, the build-minimal target was good enough to test we had a working build. With some libs partially depending on others - both of which may be disablable independently - our build test matrix just explodes. - #ifdefs are just really, really ugly in the code, and make it far harder to maintain and manage. Therefore, I'm wondering if we can come up with some sort of neater solution here. For example, can we add support to the build system that allows some form of stubbing out of libraries when they are disabled? That would save the putting of ifdefs throughout other parts of DPDK and keep the management of the disabling of the library someway inside the library itself. One way to do this might be to have a "stub" folder inside a library folder, where that contains a minimal header file to be used to provide empty functions in case where the lib itself is disabled. Other, more interesting schemes, might involve the automatic creation - from the version.map file - of dummy functions for dependent libs to link against on build. /Bruce