From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F17F1A0524; Tue, 1 Jun 2021 05:17:19 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6BD6940041; Tue, 1 Jun 2021 05:17:19 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id D2D9B40040 for ; Tue, 1 Jun 2021 05:17:17 +0200 (CEST) IronPort-SDR: jRRBRCZ2H0AE7QS60BNkStKtIOtT1PR/i1vLFHkfysk+sBe6ZM7DDh0+YRlCLJKT0htFBR+0qC xALlPdo7lw/Q== X-IronPort-AV: E=McAfee;i="6200,9189,10001"; a="183821620" X-IronPort-AV: E=Sophos;i="5.83,239,1616482800"; d="scan'208";a="183821620" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 31 May 2021 20:17:17 -0700 IronPort-SDR: 3jifRUV3gD3F028nf6RQfNMoCZYvSrEGJLDS/slYEuNObuc+GAEraU2rE+70THDaVF0ePImbic lHWW2ZRLS/0A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.83,239,1616482800"; d="scan'208";a="482315313" Received: from npg-dpdk-virtio-xiachenbo-nw.sh.intel.com ([10.67.118.250]) by fmsmga002.fm.intel.com with ESMTP; 31 May 2021 20:17:11 -0700 From: Chenbo Xia To: dev@dpdk.org, thomas@monjalon.net, cunming.liang@intel.com, jingjing.wu@intel.com Cc: anatoly.burakov@intel.com, ferruh.yigit@intel.com, mdr@ashroe.eu, nhorman@tuxdriver.com, bruce.richardson@intel.com, david.marchand@redhat.com, stephen@networkplumber.org, konstantin.ananyev@intel.com Date: Tue, 1 Jun 2021 11:06:38 +0800 Message-Id: <20210601030644.3318-1-chenbo.xia@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190715075214.16616-6-tiwei.bie@intel.com> References: <20190715075214.16616-6-tiwei.bie@intel.com> Subject: [dpdk-dev] [RFC v3 0/6] Add mdev (Mediated device) support in DPDK X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi everyone, This is a draft implementation of the mdev (Mediated device [1]) support in DPDK PCI bus driver. Mdev is a way to virtualize devices in Linux kernel. Based on the device-api (mdev_type/device_api), there could be different types of mdev devices (e.g. vfio-pci). In this patchset, the PCI bus driver is extended to support scanning and probing the mdev devices whose device-api is "vfio-pci". +---------+ | PCI bus | +----+----+ | +--------+-------+-------+--------+ | | | | Physical PCI devices ... Mediated PCI devices ... The first four patches in this patchset are mainly preparation of mdev bus support. The left two patches are the key implementation of mdev bus. The implementation of mdev bus in DPDK has several options: 1: Embed mdev bus in current pci bus This patchset takes this option for an example. Mdev has several device types: pci/platform/amba/ccw/ap. DPDK currently only cares pci devices in all mdev device types so we could embed the mdev bus into current pci bus. Then pci bus with mdev support will scan/plug/ unplug/.. not only normal pci devices but also mediated pci devices. 2: A new mdev bus that scans mediated pci devices and probes mdev driver to plug-in pci devices to pci bus If we took this option, a new mdev bus will be implemented to scan mediated pci devices and a new mdev driver for pci devices will be implemented in pci bus to plug-in mediated pci devices to pci bus. Our RFC v1 takes this option: http://patchwork.dpdk.org/project/dpdk/cover/20190403071844.21126-1-tiwei.bie@intel.com/ Note that: for either option 1 or 2, device drivers do not know the implementation difference but only use structs/functions exposed by pci bus. Mediated pci devices are different from normal pci devices on: 1. Mediated pci devices use UUID as address but normal ones use BDF. 2. Mediated pci devices may have some capabilities that normal pci devices do not have. For example, mediated pci devices could have regions that have sparse mmap capability, which allows a region to have multiple mmap areas. Another example is mediated pci devices may have regions/part of regions not mmaped but need to access them. Above difference will change the current ABI (i.e., struct rte_pci_device). Please check 5th and 6th patch for details. 3. A brand new mdev bus that does everything This option will implement a new and standalone mdev bus. This option does not need any changes in current pci bus but only needs some shared code (linux vfio part) in pci bus. Drivers of devices that support mdev will register itself as a mdev driver and do not rely on pci bus anymore. This option, IMHO, will make the code clean. The only potential problem may be code duplication, which could be solved by making code of linux vfio part of pci bus common and shared. Your comments on above three options are welcomed and appreciated! Thanks! Chenbo ---------------------------------------------------------------------------- RFC v3: - Add sparse mmap support - Minor fixes and improvements RFC v2: - Let PCI bus scan mediated PCI devices directly - Address Keith's comments - Merge below patch into this series (David) http://patches.dpdk.org/patch/55927/ - Add internal representation of PCI device (David) - Minor fixes and improvements [1] https://github.com/torvalds/linux/blob/master/Documentation/driver-api/vfio-mediated-device.rst Chenbo Xia (1): bus/pci: add sparse mmap support for mediated PCI devices Tiwei Bie (5): bus/pci: introduce an internal representation of PCI device bus/pci: avoid depending on private value in kernel source bus/pci: introduce helper for MMIO read and write eal: add a helper for reading string from sysfs bus/pci: add mdev support drivers/bus/pci/bsd/pci.c | 36 +- drivers/bus/pci/linux/pci.c | 107 ++++- drivers/bus/pci/linux/pci_init.h | 29 +- drivers/bus/pci/linux/pci_uio.c | 22 + drivers/bus/pci/linux/pci_vfio.c | 586 ++++++++++++++++++++++---- drivers/bus/pci/linux/pci_vfio_mdev.c | 277 ++++++++++++ drivers/bus/pci/meson.build | 1 + drivers/bus/pci/pci_common.c | 86 ++-- drivers/bus/pci/pci_params.c | 36 +- drivers/bus/pci/private.h | 40 ++ drivers/bus/pci/rte_bus_pci.h | 83 +++- drivers/bus/pci/version.map | 4 + lib/eal/common/eal_filesystem.h | 10 + lib/eal/freebsd/eal.c | 22 + lib/eal/linux/eal.c | 39 +- lib/eal/version.map | 3 + 16 files changed, 1224 insertions(+), 157 deletions(-) create mode 100644 drivers/bus/pci/linux/pci_vfio_mdev.c -- 2.17.1