From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 86E80A04AF; Fri, 21 Aug 2020 18:30:47 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7B7471C0D9; Fri, 21 Aug 2020 18:30:06 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by dpdk.org (Postfix) with ESMTP id 2A0841C0D7 for ; Fri, 21 Aug 2020 18:30:04 +0200 (CEST) IronPort-SDR: GHXz0OUqFFCqYYloPBWwfYcSG6oSxJJl1WP3ECfnDkMXiTVbCE6XolHoWfdryppdEmnDdyGsmF 8paoW0urV2Sg== X-IronPort-AV: E=McAfee;i="6000,8403,9719"; a="143209468" X-IronPort-AV: E=Sophos;i="5.76,337,1592895600"; d="scan'208";a="143209468" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Aug 2020 09:30:03 -0700 IronPort-SDR: K4TA+Y39mPlZceasBiffxaIcBvJQy1ZdoNQSD5qIzg0mRKlhyCYYS+ZKXwc1xYvfunQMMyO+sk nnZAu8hFv3xg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.76,337,1592895600"; d="scan'208";a="297992696" Received: from silpixa00399126.ir.intel.com ([10.237.222.56]) by orsmga006.jf.intel.com with ESMTP; 21 Aug 2020 09:30:01 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: cheng1.jiang@intel.com, patrick.fu@intel.com, ping.yu@intel.com, kevin.laatz@intel.com, Bruce Richardson Date: Fri, 21 Aug 2020 17:29:31 +0100 Message-Id: <20200821162944.29840-6-bruce.richardson@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200821162944.29840-1-bruce.richardson@intel.com> References: <20200721095140.719297-1-bruce.richardson@intel.com> <20200821162944.29840-1-bruce.richardson@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v2 05/18] raw/ioat: add skeleton for VFIO/UIO based DSA device X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add in the basic probe/remove skeleton code for DSA devices which are bound directly to vfio or uio driver. The kernel module for supporting these uses the "idxd" name, so that name is used as function and file prefix to avoid conflict with existing "ioat" prefixed functions. Since we are adding new files to the driver and there will be common definitions shared between the various files, we create a new internal header file ioat_private.h to hold common macros and function prototypes. Signed-off-by: Bruce Richardson --- doc/guides/rawdevs/ioat.rst | 69 ++++++++++----------------------- drivers/raw/ioat/idxd_pci.c | 56 ++++++++++++++++++++++++++ drivers/raw/ioat/ioat_private.h | 27 +++++++++++++ drivers/raw/ioat/ioat_rawdev.c | 9 +---- drivers/raw/ioat/meson.build | 6 ++- 5 files changed, 108 insertions(+), 59 deletions(-) create mode 100644 drivers/raw/ioat/idxd_pci.c create mode 100644 drivers/raw/ioat/ioat_private.h diff --git a/doc/guides/rawdevs/ioat.rst b/doc/guides/rawdevs/ioat.rst index c46460ff45..b83cf0f7db 100644 --- a/doc/guides/rawdevs/ioat.rst +++ b/doc/guides/rawdevs/ioat.rst @@ -3,10 +3,12 @@ .. include:: -IOAT Rawdev Driver for Intel\ |reg| QuickData Technology -====================================================================== +IOAT Rawdev Driver +=================== The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg| +Data Streaming Accelerator `(Intel DSA) +`_ and for Intel\ |reg| QuickData Technology, part of Intel\ |reg| I/O Acceleration Technology `(Intel I/OAT) `_. @@ -17,61 +19,30 @@ be done by software, freeing up CPU cycles for other tasks. Hardware Requirements ---------------------- -On Linux, the presence of an Intel\ |reg| QuickData Technology hardware can -be detected by checking the output of the ``lspci`` command, where the -hardware will be often listed as "Crystal Beach DMA" or "CBDMA". For -example, on a system with Intel\ |reg| Xeon\ |reg| CPU E5-2699 v4 @2.20GHz, -lspci shows: - -.. code-block:: console - - # lspci | grep DMA - 00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01) - 00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01) - 00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01) - 00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01) - 00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01) - 00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01) - 00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01) - 00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01) - -On a system with Intel\ |reg| Xeon\ |reg| Gold 6154 CPU @ 3.00GHz, lspci -shows: - -.. code-block:: console - - # lspci | grep DMA - 00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - +The ``dpdk-devbind.py`` script, included with DPDK, +can be used to show the presence of supported hardware. +Running ``dpdk-devbind.py --status-dev misc`` will show all the miscellaneous, +or rawdev-based devices on the system. +For Intel\ |reg| QuickData Technology devices, the hardware will be often listed as "Crystal Beach DMA", +or "CBDMA". +For Intel\ |reg| DSA devices, they are currently (at time of writing) appearing as devices with type "0b25", +due to the absence of pci-id database entries for them at this point. Compilation ------------ -For builds done with ``make``, the driver compilation is enabled by the -``CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV`` build configuration option. This is -enabled by default in builds for x86 platforms, and disabled in other -configurations. - -For builds using ``meson`` and ``ninja``, the driver will be built when the -target platform is x86-based. +For builds using ``meson`` and ``ninja``, the driver will be built when the target platform is x86-based. +No additional compilation steps are necessary. Device Setup ------------- -The Intel\ |reg| QuickData Technology HW devices will need to be bound to a -user-space IO driver for use. The script ``dpdk-devbind.py`` script -included with DPDK can be used to view the state of the devices and to bind -them to a suitable DPDK-supported kernel driver. When querying the status -of the devices, they will appear under the category of "Misc (rawdev) -devices", i.e. the command ``dpdk-devbind.py --status-dev misc`` can be -used to see the state of those devices alone. +The HW devices to be used will need to be bound to a user-space IO driver for use. +The ``dpdk-devbind.py`` script can be used to view the state of the devices +and to bind them to a suitable DPDK-supported kernel driver, such as ``vfio-pci``. +For example:: + + $ dpdk-devbind.py -b vfio-pci 00:04.0 00:04.1 Device Probing and Initialization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/raw/ioat/idxd_pci.c b/drivers/raw/ioat/idxd_pci.c new file mode 100644 index 0000000000..1a30e9c316 --- /dev/null +++ b/drivers/raw/ioat/idxd_pci.c @@ -0,0 +1,56 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2020 Intel Corporation + */ + +#include + +#include "ioat_private.h" + +#define IDXD_VENDOR_ID 0x8086 +#define IDXD_DEVICE_ID_SPR 0x0B25 + +#define IDXD_PMD_RAWDEV_NAME_PCI rawdev_idxd_pci + +const struct rte_pci_id pci_id_idxd_map[] = { + { RTE_PCI_DEVICE(IDXD_VENDOR_ID, IDXD_DEVICE_ID_SPR) }, + { .vendor_id = 0, /* sentinel */ }, +}; + +static int +idxd_rawdev_probe_pci(struct rte_pci_driver *drv, struct rte_pci_device *dev) +{ + int ret = 0; + char name[PCI_PRI_STR_SIZE]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); + dev->device.driver = &drv->driver; + + return ret; +} + +static int +idxd_rawdev_remove_pci(struct rte_pci_device *dev) +{ + char name[PCI_PRI_STR_SIZE]; + int ret = 0; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + + IOAT_PMD_INFO("Closing %s on NUMA node %d", + name, dev->device.numa_node); + + return ret; +} + +struct rte_pci_driver idxd_pmd_drv_pci = { + .id_table = pci_id_idxd_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING, + .probe = idxd_rawdev_probe_pci, + .remove = idxd_rawdev_remove_pci, +}; + +RTE_PMD_REGISTER_PCI(IDXD_PMD_RAWDEV_NAME_PCI, idxd_pmd_drv_pci); +RTE_PMD_REGISTER_PCI_TABLE(IDXD_PMD_RAWDEV_NAME_PCI, pci_id_idxd_map); +RTE_PMD_REGISTER_KMOD_DEP(IDXD_PMD_RAWDEV_NAME_PCI, + "* igb_uio | uio_pci_generic | vfio-pci"); diff --git a/drivers/raw/ioat/ioat_private.h b/drivers/raw/ioat/ioat_private.h new file mode 100644 index 0000000000..d87d4d055e --- /dev/null +++ b/drivers/raw/ioat/ioat_private.h @@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2020 Intel Corporation + */ + +#ifndef _IOAT_PRIVATE_H_ +#define _IOAT_PRIVATE_H_ + +/** + * @file idxd_private.h + * + * Private data structures for the idxd/DSA part of ioat device driver + * + * @warning + * @b EXPERIMENTAL: these structures and APIs may change without prior notice + */ + +extern int ioat_pmd_logtype; + +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args) + +#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) +#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) +#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) +#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) + +#endif /* _IOAT_PRIVATE_H_ */ diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c index bba072f2a7..20c8b671a6 100644 --- a/drivers/raw/ioat/ioat_rawdev.c +++ b/drivers/raw/ioat/ioat_rawdev.c @@ -10,6 +10,7 @@ #include "rte_ioat_rawdev.h" #include "ioat_spec.h" +#include "ioat_private.h" static struct rte_pci_driver ioat_pmd_drv; @@ -29,14 +30,6 @@ static struct rte_pci_driver ioat_pmd_drv; RTE_LOG_REGISTER(ioat_pmd_logtype, rawdev.ioat, INFO); -#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ - ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args) - -#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) -#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) -#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) -#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) - #define DESC_SZ sizeof(struct rte_ioat_generic_hw_desc) #define COMPLETION_SZ sizeof(__m128i) diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build index 06636f8a9f..3529635e9c 100644 --- a/drivers/raw/ioat/meson.build +++ b/drivers/raw/ioat/meson.build @@ -3,8 +3,10 @@ build = dpdk_conf.has('RTE_ARCH_X86') reason = 'only supported on x86' -sources = files('ioat_rawdev.c', - 'ioat_rawdev_test.c') +sources = files( + 'idxd_pci.c', + 'ioat_rawdev.c', + 'ioat_rawdev_test.c') deps += ['rawdev', 'bus_pci', 'mbuf'] install_headers('rte_ioat_rawdev.h', -- 2.25.1