From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3D5DC42AF8; Wed, 17 May 2023 01:24:36 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C135C410E4; Wed, 17 May 2023 01:24:35 +0200 (CEST) Received: from mail.redfish-solutions.com (mail.redfish-solutions.com [24.116.100.90]) by mails.dpdk.org (Postfix) with ESMTP id 76B7F40EE1 for ; Wed, 17 May 2023 01:24:34 +0200 (CEST) Received: from smtpclient.apple (macbook3-3.redfish-solutions.com [192.168.8.12]) (authenticated bits=0) by mail.redfish-solutions.com (8.17.1/8.16.1) with ESMTPSA id 34GNOWTq584487 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 16 May 2023 17:24:32 -0600 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: DPDK packaging and OpenWrt From: Philip Prindeville In-Reply-To: <20230516134323.4e2d5db9@hermes.local> Date: Tue, 16 May 2023 17:24:21 -0600 Cc: dev@dpdk.org Content-Transfer-Encoding: quoted-printable Message-Id: <0ED3E358-1635-4B3B-8DCB-01CDA08A0F25@redfish-solutions.com> References: <60A57C56-D30C-4B0F-BAE9-01BA6FD16E66@redfish-solutions.com> <20230516134323.4e2d5db9@hermes.local> To: Stephen Hemminger X-Mailer: Apple Mail (2.3731.500.231) X-Scanned-By: MIMEDefang 3.3 on 192.168.8.3 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hey Stephen, it's been a while... > On May 16, 2023, at 2:43 PM, Stephen Hemminger = wrote: >=20 > On Tue, 16 May 2023 13:18:40 -0600 > Philip Prindeville wrote: >=20 >> Hi, >>=20 >> I'm a packaging maintainer for some of the OpenWrt ancillary = packages, and I'm looking at upstreaming DPDK support into OpenWrt. = Apologies in advance if this is a bit dense (a brain dump) or hops = between too many topics. >>=20 >> I was looking at what's been done to date, and it seems there are a = few shortcomings which I hope to address. >>=20 >> Amongst them are: >>=20 >> * getting DPDK support into OpenWrt's main repo for the kmod's and = into the packages repo for the user-space support; >=20 > DPDK kernel modules are deprecated, creating more usage of them is = problematic. Well, some kernel modules are still required, aren't they? What's been = deprecated? >=20 >>=20 >> * making DPDK supported on all available architectures (i.e. = agnostic, not just x86_64 specific); >>=20 >> * integrating into the OpenWrt "make menuconfig" system, so that = editing packages directly isn't required; >=20 > Does openwrt build system support meson? >=20 >> * supporting cross-building and avoiding the [flawed] assumption that = the micro-architecture of the build server is in any way related to the = processor type of the target machine(s); >>=20 >> * integration into the OpenWrt CI/CD tests; >>=20 >> * making the kernel support as secure/robust as possible (i.e. = avoiding an ill-behaved application taking down the kernel, since this = is a firewall after all); >=20 > Not a problem with vfio >=20 >>=20 >> * avoiding conflict with other existing module or package = functionality; >>=20 >> * avoiding, to the extent possible, introducing one-off toolchain = dependencies that unnecessarily complicate the build ecosystem; >>=20 >> To this end, I'm asking the mailing list for guidance on the best = packaging practices that satisfy these goals. I'm willing to do = whatever heavy lifting that's within my ability. >>=20 >> I have some related questions. >>=20 >> 1. OpenWrt already bakes "vfio.ko" and "vfio-pci.ko" here: >>=20 >> = https://github.com/openwrt/openwrt/blob/master/package/kernel/linux/module= s/virt.mk#L77-L117 >>=20 >> but does so with "CONFIG_VFIO_PCI_IGD=3Dy", which seems to be = specifically for VGA acceleration of guests in a virtualized = environment. That seems to be an odd corner case, and unlikely given = that OpenWrt almost always runs on headless hardware. Should this be = reverted? >=20 > Yes. Okay, thanks. >=20 >>=20 >> 2. "vfio.ko" is built with "CONFIG_VFIO_NOIOMMU=3Dy", which seems = insecure. Can this be reverted? >=20 > No. Most of OpenWrt's systems do not have an IOMMU. And most of OpenWrt isn't going to need DPDK, either. We're thinking of = Xeon, Xeon-D, and Atom64 (C26xx) based Intel hardware, or high-end = multicore ARM64 designs like the Traverse Technologies ten64. In other words, Enterprise-class firewalls and data center appliances. >=20 >> 3. Is "uio_pci_generic.ko" worth the potential insecurity/instability = of a misbehaving application? My understanding is that it's only needed = on SR-IOV hardware where MSI/MSI-X interrupts aren't supported: is there = even any current hardware that doesn't support MSI/MSI-X? Or am I = misunderstanding the use case? >=20 > Use VFIO noiommu, it is more supported and tested upstream. With PCI = generic, no interrupts work. What is the risk/reward of using CONFIG_NOIOMMU=3Dn? It's a non-trivial = bit of logic to include in a processor design: it must have had some = scenario where it would be useful otherwise that's a lot of wasted = gates... >=20 >> 4. Can most functionality be achieved with VFIO + IOMMU support? >=20 > Yes. >=20 >> 5. I've seen packaging for the "iommu_v2.ko" module done here: >>=20 >> = https://github.com/k13132/openwrt-dpdk/blob/main/packages/kmod-iommu_v2/Ma= kefile#L22-L42 >>=20 >> Is this potentially useful? What platforms/architectures use this = driver? >>=20 >> 6. Hand editing a wrapper for dpdk.tar.gz is a non-starter. I'd = rather add Kconfig adornments to OpenWrt packaging for the wrapper so = that options for "-Denable_drivers=3D" and "-Dcpu_instruction_set=3D" = can be passed in once global build options for OpenWrt have been = selected. Defaulting the instruction set to the build host is going to = be wrong at least some of the time, if not most of the time. For = x86_64, what is a decent compromise for a micro-architecture that will = build and run on most AMD and Intel hardware? What's a decent baseline = for an ARM64 micro-architecture that will build and run on most ARM = hardware? >>=20 >> 7. Many embedded systems don't build with glibc because it's too = bloated (and because critical fixes sometimes take too long to roll = out), and instead use MUSL, eglibc, or uClibc (although the last one = seems to be waning). Only glibc supports from what I can = tell. Can broader support for other C runtimes be added? Can = RTE_BACKTRACE be made a parameter or conditionally defined based on the = runtime headers? (autotools would be and HAVE_EXECINFO_H would be = really handy here, but I'm not sure how to make this play well with = meson/ninja, and truth be told I'm an old school Makefile + autotools = knuckle-dragger). >=20 > You could do without backtrace, but then if DPDK applications you are = flying blind. Yeah, that occurred to me too, but it seems that libbacktrace couldn't = be shimmed in because it does require heap integrity if I remember... >=20 >> 8. How do I validate that DPDK is properly being built with the = cross-tools and not native tools? Even when building x86_64 targets on = an x86_64 build host, we end up using a custom toolchain and not the = "native" compiler that comes with the distro. >>=20 >> 9. What is the parity between AMD64 and ARM64? Do both platforms = offer equal functionality and security, if not performance? >=20 > Apples/Oranges. >=20 >> 10. Who else is using DPDK with OpenWrt that is open to = collaboration? >>=20 >> 11. What is the user-space TCP/IP stack of choice (or reference) for = use with DPDK? >=20 > No user-space TCP/IP stack is really robust and that great. > VPP has one but it is likely to be specific to using VPP and not sure = if you want to go there. > I don't think Fedora/Suse/Debian/Ubuntu have packaged any userspace = TCP yet. Not robust how? In terms of performance, security, handling error = conditions, having complete feature handlings... ? -Philip