Re: [PATCH v3 1/1] bus/pci: introduce get_iova_mode for pci dev

[Stephen Hemminger <stephen@networkplumber.org>]

On Sun, 27 Apr 2025 02:27:34 +0000
Kyo Liu <kyo.liu@nebula-matrix.com> wrote:

> I propose this patch for DPDK to enable coexistence between
> DPDK and kernel drivers for regular NICs.This solution requires
> adding a new pci_ops in rte_pci_driver, through which DPDK will
> retrieve the required IOVA mode from the vendor driver.
> This mechanism is necessary to handle different IOMMU
> configurations and operating modes. Below is a detailed
> analysis of various scenarios:
> 
> 1. When IOMMU is enabled:
> 1.1 With PT (Pass-Through) enabled:
> In this case, the domain type is IOMMU_DOMAIN_IDENTITY,
> which prevents vendor drivers from setting IOVA->PA mapping tables.
> Therefore, DPDK must use PA mode. To achieve this:
> The vendor kernel driver will register a character device (cdev) to
> communicate with DPDK. This cdev handles device operations
> (open, mmap, etc.) and ultimately
> programs the hardware registers.
> 
> 1.2 With PT disabled:
> Here, the vendor driver doesn't enforce specific IOVA mode requirements.
> Our implementation will:
> Integrate a mediated device (mdev) in the vendor driver.
> This mdev interacts with DPDK and manages IOVA->PA mapping configurations.
> 
> 2. When IOMMU is disabled:
> The vendor driver mandates PA mode (consistent with DPDK's PA mode
> requirement in this scenario).
> A character device (cdev) will similarly be registered for DPDK
> communication.
> 
> Summary:
> The solution leverages multiple technologies:
> mdev for IOVA management when IOMMU is partially enabled.
> VFIO for device passthrough operations.
> cdev for register programming coordination.
> A new pci_ops interface in DPDK to dynamically determine IOVA modes.
> This architecture enables clean coexistence by establishing standardized
> communication channels between DPDK and vendor drivers across different
> IOMMU configurations.
> 
> Motivation for the Patch:
> This patch is introduced to prepare for the upcoming open-source
> contribution of our NebulaMatrix SNIC driver to DPDK. We aim to
> ensure that our SNIC can seamlessly coexist with kernel drivers
> using this mechanism. By adopting the proposed
> architecture—leveraging dynamic IOVA mode negotiation via pci_ops,
> mediated devices (mdev), and character device (cdev)
> interactions—we enable our SNIC to operate in hybrid environments
> here both DPDK and kernel drivers may manage the same hardware.
> This design aligns with DPDK’s scalability goals and ensures
> compatibility across diverse IOMMU configurations, which is critical
> for real-world deployment scenarios.
> 
> Signed-off-by: Kyo Liu <kyo.liu@nebula-matrix.com>

Looks like this might also end up being ABI change so may have
to wait for 25.11