From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8D92E48B0C; Fri, 14 Nov 2025 18:40:38 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 51EF140E26; Fri, 14 Nov 2025 18:40:38 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.15]) by mails.dpdk.org (Postfix) with ESMTP id F29C840288 for ; Fri, 14 Nov 2025 18:40:35 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1763142036; x=1794678036; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=dlKiDifZcxMTxHvhRbB2d1MRjD9Bt2JWhqHxcN2Lias=; b=I58QSaBN+pPVIAEsFEGivX6ynDNLt+BVZLwBVq8X1FPeKcXsE5bSiuKT VNhmJg53kCZ3cZ7xUaKMJ1BPHHeohkUoPo167M6WjbfLKtEpyhPVTdr+Q kfqdwuNDJVkPaIlesGKiN7Yle0v5ue1S6L+MbhiFJMsvtSNnYMNYT6X5S FXNECAUip77x+A4IvfGLnqyyg4U4XRA5kulSOrlkKaGlj0NQh2ypkLcVF bzoEkahHr+cLco4t9jDhLecX15wbLOTh0J+66fp5gHPl9VTjbJilkItic dNjW49nAYlYNG0/+l18JW/M9rbYMRL0rXqMJNcP3EB5RvrDszsZNE8LhG A==; X-CSE-ConnectionGUID: HUQUXQIITvm+nI/O6cDyjQ== X-CSE-MsgGUID: ieWtbbAwQzOORDb5V7G6HA== X-IronPort-AV: E=McAfee;i="6800,10657,11613"; a="68864248" X-IronPort-AV: E=Sophos;i="6.19,305,1754982000"; d="scan'208";a="68864248" Received: from orviesa009.jf.intel.com ([10.64.159.149]) by orvoesa107.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Nov 2025 09:40:35 -0800 X-CSE-ConnectionGUID: S+/ZhembSyK2p/x5fJrw/w== X-CSE-MsgGUID: MYThoymAT9O58HVRhc7LLQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.19,305,1754982000"; d="scan'208";a="189650707" Received: from silpixa00401119.ir.intel.com ([10.20.224.206]) by orviesa009.jf.intel.com with ESMTP; 14 Nov 2025 09:40:34 -0800 From: Anatoly Burakov To: dev@dpdk.org Subject: [PATCH v2 00/19] Support VFIO cdev API in DPDK Date: Fri, 14 Nov 2025 17:40:10 +0000 Message-ID: X-Mailer: git-send-email 2.47.3 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patchset introduces a major refactor of the VFIO subsystem in DPDK to support character device (cdev) interface introduced in Linux kernel, as well as make the API more streamlined and useful. The goal is to simplify device management, improve compatibility, and clarify API responsibilities. The following sections outline the key issues addressed by this patchset and the corresponding changes introduced. 1. Only group mode is supported =============================== Since kernel version 4.14.327 (LTS), VFIO supports the new character device (cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This is a device-centric mode and does away with all the complexity regarding groups and IOMMU types, delegating it all to the kernel, and exposes a much simpler interface to userspace. The old group interface is still around, and will need to be kept in DPDK both for compatibility reasons, as well as supporting special cases (FSLMC bus, NBL driver, etc.). To enable this, VFIO is heavily refactored, so that the code can support both modes while relying on (mostly) common infrastructure. Note that the existing `rte_vfio_device_setup/release` model is fundamentally incompatible with cdev mode, because for custom container cases, the expected flow is that the user binds the IOMMU group (and thus, implicitly, the device itself) to a specific container using `rte_vfio_container_group_bind`, whereas this step is not needed for cdev as the device fd is assigned to the container straight away. Therefore, what we do instead is introduce a new API for container device assignment which, semantically, will assign a device to specified container, so that when it is mapped using `rte_pci_map_device`, the appropriate container is selected. Under the hood though, we essentially transition to getting device fd straight away at assign stage, so that by the time the PCI bus attempts to map the device, it is already mapped and we just return an fd. Additionally, a new `rte_vfio_get_mode` API is added for those cases that need some introspection into VFIO's internals, with three new modes: group (old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode). Although no-IOMMU is technically a variant of group mode, the distinction is largely irrelevant to the user, as all usages of noiommu checks in our codebase are for deciding whether to use IOVA or PA, not anything to do with managing groups. The current plan for kernel community is to *not* introduce no-IOMMU cdev implementation, which is why this will be kept for compatibility for these use cases. There were other users of VFIO which relied on group API but only for convenience purposes; no actual VFIO functionality depended on those API's. Therefore, group API's are removed and, where appropriate, replaced with the new API's. List of removed API's: * `rte_vfio_get_group_fd` * `rte_vfio_clear_group` * `rte_vfio_container_group_bind` * `rte_vfio_container_group_unbind` * `rte_vfio_noiommu_is_enabled` (replaced by new mode API) 2. The API responsibilities aren't clear and bleed into each other ================================================================== Some API's do multiple things at once. In particular: * `rte_vfio_get_device_info` will setup the device * `rte_vfio_setup_device` will get device info These API's have been adjusted to do one thing only. v2: - Make the entire API internal - More aggressive API pruning, complete removal of group API - Fixed a bug in group mode where device could not be used - Better documentation and deprecation notice patches - Moved doc patches to beginning of patchset Anatoly Burakov (19): doc: add deprecation notice for VFIO API doc: add deprecation notice for vDPA driver API uapi: update to v6.17 and add iommufd.h vfio: make all functions internal vfio: add container device assignment API vfio: split get device info from setup net/nbl: do not use VFIO group bind API net/ntnic: use container device assignment API vdpa/ifc: use container device assignment API vdpa/nfp: use container device assignment API vdpa/sfc: use container device assignment API vhost: remove group-related API from drivers vfio: cleanup and refactor bus/pci: use the new VFIO mode API bus/fslmc: use the new VFIO mode API net/hinic3: use the new VFIO mode API net/ntnic: use the new VFIO mode API vfio: remove group API functions vfio: introduce cdev mode config/arm/meson.build | 1 + config/meson.build | 1 + doc/guides/prog_guide/vhost_lib.rst | 4 - doc/guides/rel_notes/deprecation.rst | 7 + drivers/bus/cdx/cdx_vfio.c | 21 +- drivers/bus/fslmc/fslmc_bus.c | 10 +- drivers/bus/fslmc/fslmc_vfio.c | 2 +- drivers/bus/pci/linux/pci.c | 2 +- drivers/bus/pci/linux/pci_vfio.c | 29 +- drivers/bus/platform/platform.c | 8 +- drivers/crypto/bcmfs/bcmfs_vfio.c | 14 +- drivers/net/hinic3/base/hinic3_hwdev.c | 2 +- drivers/net/nbl/nbl_common/nbl_userdev.c | 18 +- drivers/net/nbl/nbl_include/nbl_include.h | 1 + drivers/net/ntnic/ntnic_ethdev.c | 2 +- drivers/net/ntnic/ntnic_vfio.c | 30 +- drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 1 - drivers/vdpa/nfp/nfp_vdpa.c | 37 +- drivers/vdpa/sfc/sfc_vdpa.c | 39 +- drivers/vdpa/sfc/sfc_vdpa.h | 2 - kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++ kernel/linux/uapi/linux/vduse.h | 2 +- kernel/linux/uapi/linux/vfio.h | 12 +- kernel/linux/uapi/version | 2 +- lib/eal/freebsd/eal.c | 98 +- lib/eal/include/rte_vfio.h | 386 ++-- lib/eal/linux/eal_vfio.c | 2439 ++++++++------------- lib/eal/linux/eal_vfio.h | 167 +- lib/eal/linux/eal_vfio_cdev.c | 389 ++++ lib/eal/linux/eal_vfio_group.c | 983 +++++++++ lib/eal/linux/eal_vfio_mp_sync.c | 80 +- lib/eal/linux/meson.build | 2 + lib/eal/windows/eal.c | 4 +- lib/vhost/vdpa_driver.h | 3 - 35 files changed, 4263 insertions(+), 1861 deletions(-) create mode 100644 kernel/linux/uapi/linux/iommufd.h create mode 100644 lib/eal/linux/eal_vfio_cdev.c create mode 100644 lib/eal/linux/eal_vfio_group.c -- 2.47.3