* [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices
@ 2021-08-27 17:25 Conor Walsh
2021-08-27 17:25 ` [dpdk-dev] [PATCH 1/8] dma/ioat: add device probe and removal functions Conor Walsh
` (13 more replies)
0 siblings, 14 replies; 132+ messages in thread
From: Conor Walsh @ 2021-08-27 17:25 UTC (permalink / raw)
To: dev; +Cc: bruce.richardson, fengchengwen, jerinj, kevin.laatz, Conor Walsh
This patchset adds a dmadev driver and associated documentation to support
Intel QuickData Technology devices, part of the Intel I/O Acceleration
Technology (Intel I/OAT). This driver is intended to ultimately replace
the current IOAT rawdev driver
NOTE: This patchset has several dependencies:
- v16 of the dmadev set [1]
- rfc of the dmadev test suite [2]
- v1 of the IDXD driver [3]
[1] http://patches.dpdk.org/project/dpdk/list/?series=18391
[2] http://patches.dpdk.org/project/dpdk/list/?series=18477
[3] http://patches.dpdk.org/project/dpdk/list/?series=18500
Conor Walsh (8):
dma/ioat: add device probe and removal functions
dma/ioat: create dmadev instances on PCI probe
dma/ioat: add datapath structures
dma/ioat: add configuration functions
dma/ioat: add start and stop functions
dma/ioat: add data path job submission functions
dma/ioat: add data path completion functions
dma/ioat: add statistics
MAINTAINERS | 6 +
doc/guides/dmadevs/index.rst | 1 +
doc/guides/dmadevs/ioat.rst | 214 ++++++++
doc/guides/rel_notes/release_21_11.rst | 7 +-
drivers/dma/ioat/ioat_dmadev.c | 674 +++++++++++++++++++++++++
drivers/dma/ioat/ioat_hw_defs.h | 296 +++++++++++
drivers/dma/ioat/ioat_internal.h | 44 ++
drivers/dma/ioat/meson.build | 7 +
drivers/dma/ioat/version.map | 3 +
drivers/dma/meson.build | 1 +
10 files changed, 1250 insertions(+), 3 deletions(-)
create mode 100644 doc/guides/dmadevs/ioat.rst
create mode 100644 drivers/dma/ioat/ioat_dmadev.c
create mode 100644 drivers/dma/ioat/ioat_hw_defs.h
create mode 100644 drivers/dma/ioat/ioat_internal.h
create mode 100644 drivers/dma/ioat/meson.build
create mode 100644 drivers/dma/ioat/version.map
--
2.25.1
^ permalink raw reply [flat|nested] 132+ messages in thread* [dpdk-dev] [PATCH 1/8] dma/ioat: add device probe and removal functions 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh @ 2021-08-27 17:25 ` Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 2/8] dma/ioat: create dmadev instances on PCI probe Conor Walsh ` (12 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-08-27 17:25 UTC (permalink / raw) To: dev; +Cc: bruce.richardson, fengchengwen, jerinj, kevin.laatz, Conor Walsh Add the basic device probe/remove skeleton code and initial documentation for new IOAT DMA driver. Maintainers update is also included in this patch. Signed-off-by: Conor Walsh <conor.walsh@intel.com> --- MAINTAINERS | 6 +++ doc/guides/dmadevs/index.rst | 1 + doc/guides/dmadevs/ioat.rst | 64 ++++++++++++++++++++++++ doc/guides/rel_notes/release_21_11.rst | 7 +-- drivers/dma/ioat/ioat_dmadev.c | 69 ++++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 35 +++++++++++++ drivers/dma/ioat/ioat_internal.h | 20 ++++++++ drivers/dma/ioat/meson.build | 7 +++ drivers/dma/ioat/version.map | 3 ++ drivers/dma/meson.build | 1 + 10 files changed, 210 insertions(+), 3 deletions(-) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map diff --git a/MAINTAINERS b/MAINTAINERS index 3d275c6bb4..2561bda1af 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1208,6 +1208,12 @@ M: Kevin Laatz <kevin.laatz@intel.com> F: drivers/dma/idxd/ F: doc/guides/dmadevs/idxd.rst +Intel IOAT - EXPERIMENTAL +M: Bruce Richardson <bruce.richardson@intel.com> +M: Conor Walsh <conor.walsh@intel.com> +F: drivers/dma/ioat/ +F: doc/guides/dmadevs/ioat.rst + RegEx Drivers ------------- diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst index b30004fd65..0b01493c5a 100644 --- a/doc/guides/dmadevs/index.rst +++ b/doc/guides/dmadevs/index.rst @@ -12,3 +12,4 @@ application through DMAdev API. :numbered: idxd + ioat diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst new file mode 100644 index 0000000000..45a2e65d70 --- /dev/null +++ b/doc/guides/dmadevs/ioat.rst @@ -0,0 +1,64 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2021 Intel Corporation. + +.. include:: <isonum.txt> + +IOAT DMA Device Driver +======================= + +The ``ioat`` dmadev driver provides a poll-mode driver (PMD) for Intel\ +|reg| QuickData Technology which is part of part of Intel\ |reg| I/O +Acceleration Technology (`Intel I/OAT +<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_). +This PMD, when used on supported hardware, allows data copies, for example, +cloning packet data, to be accelerated by IOAT hardware rather than having to +be done by software, freeing up CPU cycles for other tasks. + +Hardware Requirements +---------------------- + +The ``dpdk-devbind.py`` script, included with DPDK, can be used to show the +presence of supported hardware. Running ``dpdk-devbind.py --status-dev dma`` +will show all the DMA devices on the system, IOAT devices are included in this +list. For Intel\ |reg| IOAT devices, the hardware will often be listed as +"Crystal Beach DMA", or "CBDMA" or on some newer systems '0b00' due to the +absence of pci-id database entries for them at this point. + +Compilation +------------ + +For builds using ``meson`` and ``ninja``, the driver will be built when the +target platform is x86-based. No additional compilation steps are necessary. + +Device Setup +------------- + +Intel\ |reg| IOAT devices will need to be bound to a suitable DPDK-supported +user-space IO driver such as ``vfio-pci`` in order to be used by DPDK. + +The ``dpdk-devbind.py`` script can be used to view the state of the devices using:: + + $ dpdk-devbind.py --status-dev dma + +The ``dpdk-devbind.py`` script can also be used to bind devices to a suitable driver. +For example:: + + $ dpdk-devbind.py -b vfio-pci 00:01.0 00:01.1 + +Device Probing and Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For devices bound to a suitable DPDK-supported driver (``vfio-pci``), the HW +devices will be found as part of the device scan done at application +initialization time without the need to pass parameters to the application. + +If the application does not require all the devices available an allowlist can +be used in the same way that other DPDK devices use them. + +For example:: + + $ dpdk-test -a <b:d:f> + +Once probed successfully, the device will appear as a ``dmadev``, that is a +"DMA device type" inside DPDK, and can be accessed using APIs from the +``rte_dmadev`` library. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 98ccc41e7b..129fc44af5 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -61,10 +61,11 @@ New Features provisioning of hardware and software DMA poll mode drivers, defining generic APIs which support a number of different DMA operations. -* **Added IDXD dmadev driver implementation.** +* **Added Intel dmadev driver implementations.** - The IDXD dmadev driver provide device drivers for the Intel DSA devices. - This device driver can be used through the generic dmadev API. + The IDXD and IOAT dmadev drivers provide device drivers for Intel DSA + and IOAT devices. These device drivers can be used through the generic + dmadev API. Removed Items diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c new file mode 100644 index 0000000000..f3491d45b1 --- /dev/null +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#include <rte_bus_pci.h> +#include <rte_dmadev_pmd.h> + +#include "ioat_internal.h" + +static struct rte_pci_driver ioat_pmd_drv; + +RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); + +#define IOAT_PMD_NAME dmadev_ioat +#define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) + +/* Probe DMA device. */ +static int +ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); + + dev->device.driver = &drv->driver; + return 0; +} + +/* Remove DMA device. */ +static int +ioat_dmadev_remove(struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + + IOAT_PMD_INFO("Closing %s on NUMA node %d", + name, dev->device.numa_node); + + return 0; +} + +static const struct rte_pci_id pci_id_ioat_map[] = { + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_SKX) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX0) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX1) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX2) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX3) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX4) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX5) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX6) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX7) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXE) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXF) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_ICX) }, + { .vendor_id = 0, /* sentinel */ }, +}; + +static struct rte_pci_driver ioat_pmd_drv = { + .id_table = pci_id_ioat_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC, + .probe = ioat_dmadev_probe, + .remove = ioat_dmadev_remove, +}; + +RTE_PMD_REGISTER_PCI(IOAT_PMD_NAME, ioat_pmd_drv); +RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_NAME, pci_id_ioat_map); +RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_NAME, "* igb_uio | uio_pci_generic | vfio-pci"); diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h new file mode 100644 index 0000000000..eeabba41ef --- /dev/null +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#ifndef IOAT_HW_DEFS_H +#define IOAT_HW_DEFS_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> + +#define IOAT_VER_3_0 0x30 +#define IOAT_VER_3_3 0x33 + +#define IOAT_VENDOR_ID 0x8086 +#define IOAT_DEVICE_ID_SKX 0x2021 +#define IOAT_DEVICE_ID_BDX0 0x6f20 +#define IOAT_DEVICE_ID_BDX1 0x6f21 +#define IOAT_DEVICE_ID_BDX2 0x6f22 +#define IOAT_DEVICE_ID_BDX3 0x6f23 +#define IOAT_DEVICE_ID_BDX4 0x6f24 +#define IOAT_DEVICE_ID_BDX5 0x6f25 +#define IOAT_DEVICE_ID_BDX6 0x6f26 +#define IOAT_DEVICE_ID_BDX7 0x6f27 +#define IOAT_DEVICE_ID_BDXE 0x6f2E +#define IOAT_DEVICE_ID_BDXF 0x6f2F +#define IOAT_DEVICE_ID_ICX 0x0b00 + +#ifdef __cplusplus +} +#endif + +#endif /* IOAT_HW_DEFS_H */ diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h new file mode 100644 index 0000000000..f1ec12a919 --- /dev/null +++ b/drivers/dma/ioat/ioat_internal.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021 Intel Corporation + */ + +#ifndef _IOAT_INTERNAL_H_ +#define _IOAT_INTERNAL_H_ + +#include "ioat_hw_defs.h" + +extern int ioat_pmd_logtype; + +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + ioat_pmd_logtype, "IOAT: %s(): " fmt "\n", __func__, ##args) + +#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) +#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) +#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) +#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) + +#endif /* _IOAT_INTERNAL_H_ */ diff --git a/drivers/dma/ioat/meson.build b/drivers/dma/ioat/meson.build new file mode 100644 index 0000000000..d67fac96fb --- /dev/null +++ b/drivers/dma/ioat/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021 Intel Corporation + +build = dpdk_conf.has('RTE_ARCH_X86') +reason = 'only supported on x86' +sources = files('ioat_dmadev.c') +deps += ['bus_pci', 'dmadev'] diff --git a/drivers/dma/ioat/version.map b/drivers/dma/ioat/version.map new file mode 100644 index 0000000000..c2e0723b4c --- /dev/null +++ b/drivers/dma/ioat/version.map @@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +}; diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build index 0b01b6a8ab..f8f5154a40 100644 --- a/drivers/dma/meson.build +++ b/drivers/dma/meson.build @@ -7,6 +7,7 @@ endif drivers = [ 'idxd', + 'ioat', 'skeleton', ] std_deps = ['dmadev'] -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH 2/8] dma/ioat: create dmadev instances on PCI probe 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 1/8] dma/ioat: add device probe and removal functions Conor Walsh @ 2021-08-27 17:25 ` Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 3/8] dma/ioat: add datapath structures Conor Walsh ` (11 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-08-27 17:25 UTC (permalink / raw) To: dev; +Cc: bruce.richardson, fengchengwen, jerinj, kevin.laatz, Conor Walsh When a suitable device is found during the PCI probe, create a dmadev instance for each channel. Internal structures and HW definitions required for device creation are also included. Signed-off-by: Conor Walsh <conor.walsh@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 115 ++++++++++++++++++++++++++++++- drivers/dma/ioat/ioat_hw_defs.h | 47 +++++++++++++ drivers/dma/ioat/ioat_internal.h | 24 +++++++ 3 files changed, 184 insertions(+), 2 deletions(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index f3491d45b1..a56383dbb4 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -4,6 +4,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> +#include <rte_malloc.h> #include "ioat_internal.h" @@ -14,6 +15,116 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Create a DMA device. */ +static int +ioat_dmadev_create(const char *name, struct rte_pci_device *dev) +{ + static const struct rte_dmadev_ops ioat_dmadev_ops = { }; + + struct rte_dmadev *dmadev = NULL; + struct ioat_dmadev *ioat = NULL; + int ret = 0, retry = 0; + + if (!name) { + IOAT_PMD_ERR("Invalid name of the device!"); + ret = -EINVAL; + goto cleanup; + } + + /* Allocate device structure. */ + dmadev = rte_dmadev_pmd_allocate(name); + if (dmadev == NULL) { + IOAT_PMD_ERR("Unable to allocate dma device"); + ret = -ENOMEM; + goto cleanup; + } + + dmadev->device = &dev->device; + + dmadev->data->dev_private = rte_malloc_socket(NULL, sizeof(*ioat), + 0, dmadev->device->numa_node); + dmadev->dev_private = dmadev->data->dev_private; + + dmadev->dev_ops = &ioat_dmadev_ops; + + ioat = dmadev->data->dev_private; + ioat->dmadev = dmadev; + ioat->regs = dev->mem_resource[0].addr; + ioat->doorbell = &ioat->regs->dmacount; + ioat->qcfg.nb_desc = 0; + ioat->desc_ring = NULL; + + /* Do device initialization - reset and set error behaviour. */ + if (ioat->regs->chancnt != 1) + IOAT_PMD_WARN("%s: Channel count == %d\n", __func__, + ioat->regs->chancnt); + + /* Locked by someone else. */ + if (ioat->regs->chanctrl & IOAT_CHANCTRL_CHANNEL_IN_USE) { + IOAT_PMD_WARN("%s: Channel appears locked\n", __func__); + ioat->regs->chanctrl = 0; + } + + /* clear any previous errors */ + if (ioat->regs->chanerr != 0) { + uint32_t val = ioat->regs->chanerr; + ioat->regs->chanerr = val; + } + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + rte_delay_ms(1); + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + rte_delay_ms(1); + while (ioat->regs->chancmd & IOAT_CHANCMD_RESET) { + ioat->regs->chainaddr = 0; + rte_delay_ms(1); + if (++retry >= 200) { + IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=%#"PRIx8 + ", CHANSTS=%#"PRIx64", CHANERR=%#"PRIx32"\n", + __func__, + ioat->regs->chancmd, + ioat->regs->chansts, + ioat->regs->chanerr); + ret = -EIO; + } + } + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + + return 0; + +cleanup: + if (dmadev) + rte_dmadev_pmd_release(dmadev); + + return ret; +} + +/* Destroy a DMA device. */ +static int +ioat_dmadev_destroy(const char *name) +{ + struct rte_dmadev *dev; + int ret; + + if (!name) { + IOAT_PMD_ERR("Invalid device name"); + return -EINVAL; + } + + dev = rte_dmadev_get_device_by_name(name); + if (dev == NULL) { + IOAT_PMD_ERR("Invalid device name (%s)", name); + return -EINVAL; + } + + ret = rte_dmadev_pmd_release(dev); + if (ret) + IOAT_PMD_DEBUG("Device cleanup failed"); + + return 0; +} + /* Probe DMA device. */ static int ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) @@ -24,7 +135,7 @@ ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); dev->device.driver = &drv->driver; - return 0; + return ioat_dmadev_create(name, dev); } /* Remove DMA device. */ @@ -38,7 +149,7 @@ ioat_dmadev_remove(struct rte_pci_device *dev) IOAT_PMD_INFO("Closing %s on NUMA node %d", name, dev->device.numa_node); - return 0; + return ioat_dmadev_destroy(name); } static const struct rte_pci_id pci_id_ioat_map[] = { diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index eeabba41ef..c3a592c306 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -11,6 +11,10 @@ extern "C" { #include <stdint.h> +#define IOAT_PCI_CHANERR_INT_OFFSET 0x180 + +#define IOAT_INTRCTRL_MASTER_INT_EN 0x01 + #define IOAT_VER_3_0 0x30 #define IOAT_VER_3_3 0x33 @@ -28,6 +32,49 @@ extern "C" { #define IOAT_DEVICE_ID_BDXF 0x6f2F #define IOAT_DEVICE_ID_ICX 0x0b00 +#define IOAT_COMP_UPDATE_SHIFT 3 +#define IOAT_CMD_OP_SHIFT 24 + +/* DMA Channel Registers */ +#define IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK 0xF000 +#define IOAT_CHANCTRL_COMPL_DCA_EN 0x0200 +#define IOAT_CHANCTRL_CHANNEL_IN_USE 0x0100 +#define IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL 0x0020 +#define IOAT_CHANCTRL_ERR_INT_EN 0x0010 +#define IOAT_CHANCTRL_ANY_ERR_ABORT_EN 0x0008 +#define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 +#define IOAT_CHANCTRL_INT_REARM 0x0001 + +struct ioat_registers { + uint8_t chancnt; + uint8_t xfercap; + uint8_t genctrl; + uint8_t intrctrl; + uint32_t attnstatus; + uint8_t cbver; /* 0x08 */ + uint8_t reserved4[0x3]; /* 0x09 */ + uint16_t intrdelay; /* 0x0C */ + uint16_t cs_status; /* 0x0E */ + uint32_t dmacapability; /* 0x10 */ + uint8_t reserved5[0x6C]; /* 0x14 */ + uint16_t chanctrl; /* 0x80 */ + uint8_t reserved6[0x2]; /* 0x82 */ + uint8_t chancmd; /* 0x84 */ + uint8_t reserved3[1]; /* 0x85 */ + uint16_t dmacount; /* 0x86 */ + uint64_t chansts; /* 0x88 */ + uint64_t chainaddr; /* 0x90 */ + uint64_t chancmp; /* 0x98 */ + uint8_t reserved2[0x8]; /* 0xA0 */ + uint32_t chanerr; /* 0xA8 */ + uint32_t chanerrmask; /* 0xAC */ +} __rte_packed; + +#define IOAT_CHANCMD_RESET 0x20 +#define IOAT_CHANCMD_SUSPEND 0x04 + +#define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ + #ifdef __cplusplus } #endif diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h index f1ec12a919..133fceae86 100644 --- a/drivers/dma/ioat/ioat_internal.h +++ b/drivers/dma/ioat/ioat_internal.h @@ -7,6 +7,30 @@ #include "ioat_hw_defs.h" +struct ioat_dmadev { + struct rte_dmadev *dmadev; + struct rte_dmadev_vchan_conf qcfg; + struct rte_dmadev_stats stats; + + volatile uint16_t *doorbell __rte_cache_aligned; + phys_addr_t status_addr; + phys_addr_t ring_addr; + + struct ioat_dma_hw_desc *desc_ring; + + unsigned short next_read; + unsigned short next_write; + unsigned short last_write; /* Used to compute submitted count. */ + unsigned short offset; /* Used after a device recovery when counts -> 0. */ + unsigned int failure; /* Used to store chanerr for error handling. */ + + /* To report completions, the device will write status back here. */ + volatile uint64_t status __rte_cache_aligned; + + /* Pointer to the register bar. */ + volatile struct ioat_registers *regs; +}; + extern int ioat_pmd_logtype; #define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH 3/8] dma/ioat: add datapath structures 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 1/8] dma/ioat: add device probe and removal functions Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 2/8] dma/ioat: create dmadev instances on PCI probe Conor Walsh @ 2021-08-27 17:25 ` Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 4/8] dma/ioat: add configuration functions Conor Walsh ` (10 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-08-27 17:25 UTC (permalink / raw) To: dev; +Cc: bruce.richardson, fengchengwen, jerinj, kevin.laatz, Conor Walsh Add data structures required for the data path of IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 61 ++++++++- drivers/dma/ioat/ioat_hw_defs.h | 214 ++++++++++++++++++++++++++++++++ 2 files changed, 274 insertions(+), 1 deletion(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index a56383dbb4..c4d7ad4236 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -15,11 +15,70 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Dump DMA device info. */ +static int +ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint64_t chansts_masked = ioat->regs->chansts & IOAT_CHANSTS_STATUS; + uint32_t chanerr = ioat->regs->chanerr; + uint64_t mask = (ioat->qcfg.nb_desc - 1); + fprintf(f, "========= IOAT =========\n"); + fprintf(f, " Channel status: %s [%#lx]\n", + chansts_readable[chansts_masked], chansts_masked); + fprintf(f, " ChainADDR: %#lx\n", ioat->regs->chainaddr); + if (chanerr == 0) { + fprintf(f, " No Channel Errors\n"); + } else { + fprintf(f, " ChanERR: %#x\n", chanerr); + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + fprintf(f, " Invalid Source Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + fprintf(f, " Invalid Destination Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + fprintf(f, " Invalid Descriptor Length\n"); + if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + fprintf(f, " Descriptor Read Error\n"); + if ((chanerr & ~(IOAT_CHANERR_INVALID_SRC_ADDR_MASK | + IOAT_CHANERR_INVALID_DST_ADDR_MASK | + IOAT_CHANERR_INVALID_LENGTH_MASK | + IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK)) != 0) + fprintf(f, " Unknown Error(s)\n"); + } + fprintf(f, "== Private Data ==\n"); + fprintf(f, " Config: { ring_size: %u }\n", ioat->qcfg.nb_desc); + fprintf(f, " Status: %#lx\n", ioat->status); + fprintf(f, " Status IOVA: %#lx\n", ioat->status_addr); + fprintf(f, " Status ADDR: %p\n", &ioat->status); + fprintf(f, " Ring IOVA: %#lx\n", ioat->ring_addr); + fprintf(f, " Ring ADDR: %#lx\n", ioat->desc_ring[0].next-64); + fprintf(f, " Next write: %u\n", ioat->next_write); + fprintf(f, " Next read: %u\n", ioat->next_read); + struct ioat_dma_hw_desc *desc_ring = &ioat->desc_ring[(ioat->next_write - 1) & mask]; + fprintf(f, " Last Descriptor Written {\n"); + fprintf(f, " Size: %u\n", desc_ring->size); + fprintf(f, " Control: %#x\n", desc_ring->u.control_raw); + fprintf(f, " Src: %#lx\n", desc_ring->src_addr); + fprintf(f, " Dest: %#lx\n", desc_ring->dest_addr); + fprintf(f, " Next: %#lx\n", desc_ring->next); + fprintf(f, " }\n"); + fprintf(f, " Next Descriptor {\n"); + fprintf(f, " Size: %u\n", ioat->desc_ring[ioat->next_read & mask].size); + fprintf(f, " Src: %#lx\n", ioat->desc_ring[ioat->next_read & mask].src_addr); + fprintf(f, " Dest: %#lx\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); + fprintf(f, " Next: %#lx\n", ioat->desc_ring[ioat->next_read & mask].next); + fprintf(f, " }\n"); + + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { - static const struct rte_dmadev_ops ioat_dmadev_ops = { }; + static const struct rte_dmadev_ops ioat_dmadev_ops = { + .dev_dump = ioat_dev_dump, + }; struct rte_dmadev *dmadev = NULL; struct ioat_dmadev *ioat = NULL; diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index c3a592c306..b4109620f8 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -45,6 +45,14 @@ extern "C" { #define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 #define IOAT_CHANCTRL_INT_REARM 0x0001 +/* DMA Channel Capabilities */ +#define IOAT_DMACAP_PB (1 << 0) +#define IOAT_DMACAP_DCA (1 << 4) +#define IOAT_DMACAP_BFILL (1 << 6) +#define IOAT_DMACAP_XOR (1 << 8) +#define IOAT_DMACAP_PQ (1 << 9) +#define IOAT_DMACAP_DMA_DIF (1 << 10) + struct ioat_registers { uint8_t chancnt; uint8_t xfercap; @@ -73,8 +81,214 @@ struct ioat_registers { #define IOAT_CHANCMD_RESET 0x20 #define IOAT_CHANCMD_SUSPEND 0x04 +#define IOAT_CHANSTS_STATUS 0x7ULL +#define IOAT_CHANSTS_ACTIVE 0x0 +#define IOAT_CHANSTS_IDLE 0x1 +#define IOAT_CHANSTS_SUSPENDED 0x2 +#define IOAT_CHANSTS_HALTED 0x3 +#define IOAT_CHANSTS_ARMED 0x4 + +#define IOAT_CHANERR_INVALID_SRC_ADDR_MASK (1 << 0) +#define IOAT_CHANERR_INVALID_DST_ADDR_MASK (1 << 1) +#define IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK (1 << 8) +#define IOAT_CHANERR_INVALID_LENGTH_MASK (1 << 10) + +const char *chansts_readable[] = { + "ACTIVE", /* 0x0 */ + "IDLE", /* 0x1 */ + "SUSPENDED", /* 0x2 */ + "HALTED", /* 0x3 */ + "ARMED" /* 0x4 */ +}; + +#define IOAT_CHANSTS_UNAFFILIATED_ERROR 0x8ULL +#define IOAT_CHANSTS_SOFT_ERROR 0x10ULL + +#define IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK (~0x3FULL) + #define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ +struct ioat_dma_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t null: 1; + uint32_t src_page_break: 1; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_COPY 0x00 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t reserved2; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_fill_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t reserved: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t reserved2: 2; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t reserved3: 15; +#define IOAT_OP_FILL 0x01 + uint32_t op: 8; + } control; + } u; + uint64_t src_data; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t next_dest_addr; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_xor_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_XOR 0x87 +#define IOAT_OP_XOR_VAL 0x88 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint64_t src_addr4; + uint64_t src_addr5; +}; + +struct ioat_xor_ext_hw_desc { + uint64_t src_addr6; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t next; + uint64_t reserved[4]; +}; + +struct ioat_pq_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 11; +#define IOAT_OP_PQ 0x89 +#define IOAT_OP_PQ_VAL 0x8a + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint8_t coef[8]; + uint64_t q_addr; +}; + +struct ioat_pq_ext_hw_desc { + uint64_t src_addr4; + uint64_t src_addr5; + uint64_t src_addr6; + uint64_t next; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t reserved[2]; +}; + +struct ioat_pq_update_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_cnt: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 3; + uint32_t coef: 8; +#define IOAT_OP_PQ_UP 0x8b + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t p_src; + uint64_t q_src; + uint64_t q_addr; +}; + +union ioat_hw_desc { + struct ioat_dma_hw_desc dma; + struct ioat_fill_hw_desc fill; + struct ioat_xor_hw_desc xor_desc; + struct ioat_xor_ext_hw_desc xor_ext; + struct ioat_pq_hw_desc pq; + struct ioat_pq_ext_hw_desc pq_ext; + struct ioat_pq_update_hw_desc pq_update; +}; + +#define GENSTS_DEV_STATE_MASK 0x03 +#define CMDSTATUS_ACTIVE_SHIFT 31 +#define CMDSTATUS_ACTIVE_MASK (1 << 31) +#define CMDSTATUS_ERR_MASK 0xFF + #ifdef __cplusplus } #endif -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH 4/8] dma/ioat: add configuration functions 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh ` (2 preceding siblings ...) 2021-08-27 17:25 ` [dpdk-dev] [PATCH 3/8] dma/ioat: add datapath structures Conor Walsh @ 2021-08-27 17:25 ` Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 5/8] dma/ioat: add start and stop functions Conor Walsh ` (9 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-08-27 17:25 UTC (permalink / raw) To: dev; +Cc: bruce.richardson, fengchengwen, jerinj, kevin.laatz, Conor Walsh Add functions for device configuration. The info_get and close functions are included here also. info_get can be useful for checking successful configuration and close is used by the dmadev api when releasing a configured device. Signed-off-by: Conor Walsh <conor.walsh@intel.com> --- doc/guides/dmadevs/ioat.rst | 24 ++++++++++ drivers/dma/ioat/ioat_dmadev.c | 80 ++++++++++++++++++++++++++++++++++ 2 files changed, 104 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 45a2e65d70..b6d88fe966 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -62,3 +62,27 @@ For example:: Once probed successfully, the device will appear as a ``dmadev``, that is a "DMA device type" inside DPDK, and can be accessed using APIs from the ``rte_dmadev`` library. + +Device Configuration +~~~~~~~~~~~~~~~~~~~~~ + +Configuring an IOAT dmadev device is done using the ``rte_dmadev_configure()`` +and ``rte_dmadev_vchan_setup()`` APIs. ``rte_dmadev_configure()`` uses the structure +``rte_dmadev_conf`` to configure an IOAT device, within this struct the number +of virtual DMA channels for the device is set and silent mode can be enabled. +Each IOAT device can only use 1 virtual DMA channel and silent mode is not +supported so these will always be set to ``1`` and ``false``. +``rte_dmadev_vchan_setup()`` uses the structure ``rte_dmadev_vchan_conf`` to setup +a virtual DMA channel within this struct the transfer direction and ring size +are set. Generally for an onboard Intel\ |reg| IOAT device the transfer direction +will be set to ``RTE_DMA_DIR_MEM_TO_MEM`` to copy from memory to memory (more info +available in the dmadev API guide). The ring size must be a power of two, between +64 and 4096. + +The following code shows how the device is configured in ``test_dmadev.c``: + +.. literalinclude:: ../../../app/test/test_dmadev.c + :language: c + :start-after: Setup of the dmadev device. 8< + :end-before: >8 End of setup of the dmadev device. + :dedent: 1 diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index c4d7ad4236..db862b72c0 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -12,9 +12,85 @@ static struct rte_pci_driver ioat_pmd_drv; RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); +#define DESC_SZ sizeof(struct ioat_dma_hw_desc) + #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Configure a device. */ +static int +ioat_dev_configure(struct rte_dmadev *dev __rte_unused, const struct rte_dmadev_conf *dev_conf) +{ + if (dev_conf->nb_vchans != 1) + return -EINVAL; + return 0; +} + +/* Setup a virtual channel for IOAT, only 1 vchan is supported. */ +static int +ioat_vchan_setup(struct rte_dmadev *dev, uint16_t vchan __rte_unused, + const struct rte_dmadev_vchan_conf *qconf) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint16_t max_desc = qconf->nb_desc; + int i; + + ioat->qcfg = *qconf; + + if (!rte_is_power_of_2(max_desc)) { + max_desc = rte_align32pow2(max_desc); + IOAT_PMD_DEBUG("DMA dev %u using %u descriptors", dev->data->dev_id, max_desc); + ioat->qcfg.nb_desc = max_desc; + } + + /* In case we are reconfiguring a device, free any existing memory. */ + rte_free(ioat->desc_ring); + + ioat->desc_ring = rte_zmalloc(NULL, sizeof(*ioat->desc_ring) * max_desc, 0); + if (ioat->desc_ring == NULL) + return -ENOMEM; + + ioat->ring_addr = rte_mem_virt2iova(ioat->desc_ring); + + ioat->status_addr = rte_mem_virt2iova(ioat) + offsetof(struct ioat_dmadev, status); + + ioat->offset = 0; + + /* Configure descriptor ring - each one points to next. */ + for (i = 0; i < ioat->qcfg.nb_desc; i++) { + ioat->desc_ring[i].next = ioat->ring_addr + + (((i + 1) % ioat->qcfg.nb_desc) * DESC_SZ); + } + + return 0; +} + +/* Get device information of a device. */ +static int +ioat_dev_info_get(const struct rte_dmadev *dev, struct rte_dmadev_info *info, uint32_t size) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + if (size < sizeof(*info)) + return -EINVAL; + info->device = dev->device; + info->dev_capa = RTE_DMADEV_CAPA_MEM_TO_MEM | + RTE_DMADEV_CAPA_OPS_COPY | RTE_DMADEV_CAPA_OPS_FILL, + info->max_vchans = 1; + info->min_desc = 32; + info->max_desc = 4096; + info->nb_vchans = (ioat->desc_ring == NULL); + return 0; +} + +/* Close a configured device. */ +static int +ioat_dev_close(struct rte_dmadev *dev) +{ + RTE_SET_USED(dev); + return 0; +} + /* Dump DMA device info. */ static int ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) @@ -77,7 +153,11 @@ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { static const struct rte_dmadev_ops ioat_dmadev_ops = { + .dev_close = ioat_dev_close, + .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, + .dev_info_get = ioat_dev_info_get, + .vchan_setup = ioat_vchan_setup, }; struct rte_dmadev *dmadev = NULL; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH 5/8] dma/ioat: add start and stop functions 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh ` (3 preceding siblings ...) 2021-08-27 17:25 ` [dpdk-dev] [PATCH 4/8] dma/ioat: add configuration functions Conor Walsh @ 2021-08-27 17:25 ` Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 6/8] dma/ioat: add data path job submission functions Conor Walsh ` (8 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-08-27 17:25 UTC (permalink / raw) To: dev; +Cc: bruce.richardson, fengchengwen, jerinj, kevin.laatz, Conor Walsh Add start, stop and recover functions for IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> --- doc/guides/dmadevs/ioat.rst | 3 ++ drivers/dma/ioat/ioat_dmadev.c | 82 ++++++++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index b6d88fe966..f7742642b5 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -86,3 +86,6 @@ The following code shows how the device is configured in ``test_dmadev.c``: :start-after: Setup of the dmadev device. 8< :end-before: >8 End of setup of the dmadev device. :dedent: 1 + +Once configured, the device can then be made ready for use by calling the +``rte_dmadev_start()`` API. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index db862b72c0..6cf368c8a5 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -65,6 +65,86 @@ ioat_vchan_setup(struct rte_dmadev *dev, uint16_t vchan __rte_unused, return 0; } +/* Recover IOAT device. */ +static inline int +__ioat_recover(struct ioat_dmadev *ioat) +{ + uint32_t chanerr, retry = 0; + uint16_t mask = ioat->qcfg.nb_desc - 1; + + /* Clear any channel errors. Reading and writing to chanerr does this. */ + chanerr = ioat->regs->chanerr; + ioat->regs->chanerr = chanerr; + + /* Reset Channel. */ + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + + /* Write new chain address to trigger state change. */ + ioat->regs->chainaddr = ioat->desc_ring[(ioat->next_read - 1) & mask].next; + /* Ensure channel control and status addr are correct. */ + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + ioat->regs->chancmp = ioat->status_addr; + + /* Allow HW time to move to the ARMED state. */ + do { + rte_pause(); + retry++; + } while (ioat->regs->chansts != IOAT_CHANSTS_ARMED && retry < 200); + + /* Exit as failure if device is still HALTED. */ + if (ioat->regs->chansts != IOAT_CHANSTS_ARMED) + return -1; + + /* Store next write as offset as recover will move HW and SW ring out of sync. */ + ioat->offset = ioat->next_read; + + /* Prime status register with previous address. */ + ioat->status = ioat->desc_ring[(ioat->next_read - 2) & mask].next; + + return 0; +} + +/* Start a configured device. */ +static int +ioat_dev_start(struct rte_dmadev *dev) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + if (ioat->qcfg.nb_desc == 0 || ioat->desc_ring == NULL) + return -EBUSY; + + /* Inform hardware of where the descriptor ring is. */ + ioat->regs->chainaddr = ioat->ring_addr; + /* Inform hardware of where to write the status/completions. */ + ioat->regs->chancmp = ioat->status_addr; + + /* Prime the status register to be set to the last element. */ + ioat->status = ioat->ring_addr + ((ioat->qcfg.nb_desc - 1) * DESC_SZ); + + printf("IOAT.status: %s [%#lx]\n", + chansts_readable[ioat->status & IOAT_CHANSTS_STATUS], + ioat->status); + + if ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_HALTED) { + IOAT_PMD_WARN("Device HALTED on start, attempting to recover\n"); + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device couldn't be recovered"); + return -1; + } + } + + return 0; +} + +/* Stop a configured device. */ +static int +ioat_dev_stop(struct rte_dmadev *dev) +{ + RTE_SET_USED(dev); + return 0; +} + /* Get device information of a device. */ static int ioat_dev_info_get(const struct rte_dmadev *dev, struct rte_dmadev_info *info, uint32_t size) @@ -157,6 +237,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, .dev_info_get = ioat_dev_info_get, + .dev_start = ioat_dev_start, + .dev_stop = ioat_dev_stop, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH 6/8] dma/ioat: add data path job submission functions 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh ` (4 preceding siblings ...) 2021-08-27 17:25 ` [dpdk-dev] [PATCH 5/8] dma/ioat: add start and stop functions Conor Walsh @ 2021-08-27 17:25 ` Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 7/8] dma/ioat: add data path completion functions Conor Walsh ` (7 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-08-27 17:25 UTC (permalink / raw) To: dev; +Cc: bruce.richardson, fengchengwen, jerinj, kevin.laatz, Conor Walsh Add data path functions for enqueuing and submitting operations to IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> --- doc/guides/dmadevs/ioat.rst | 54 ++++++++++++++++++++ drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index f7742642b5..16acfda3e3 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -89,3 +89,57 @@ The following code shows how the device is configured in ``test_dmadev.c``: Once configured, the device can then be made ready for use by calling the ``rte_dmadev_start()`` API. + +Performing Data Copies +~~~~~~~~~~~~~~~~~~~~~~~ + +To perform data copies using IOAT dmadev devices, the functions +``rte_dmadev_copy()`` and ``rte_dmadev_submit()`` should be used. Alternatively +``rte_dmadev_copy()`` can be called with the ``RTE_DMA_OP_FLAG_SUBMIT`` flag +set. + +The ``rte_dmadev_copy()`` function enqueues a single copy to the +device ring for copying at a later point. The parameters to the function +include the device ID of the desired device, the virtual DMA channel required +(always 0 for IOAT), the IOVA addresses of both the source and destination +buffers, the length of the data to be copied and any operation flags. The +function will return the index of the enqueued job which can be use to +track that operation. + +While the ``rte_dmadev_copy()`` function enqueues a copy operation on the device +ring, the copy will not actually be performed until after the application calls +the ``rte_dmadev_submit()`` function. This function informs the device hardware +of the elements enqueued on the ring, and the device will begin to process them. +It is expected that, for efficiency reasons, a burst of operations will be +enqueued to the device via multiple enqueue calls between calls to the +``rte_dmadev_submit()`` function. If desired you can pass the +``RTE_DMA_OP_FLAG_SUBMIT`` flag when calling ``rte_dmadev_copy()`` and this will +tell the device to perform the enqueued operation and any unperformed operations +before it. The ``RTE_DMA_OP_FLAG_SUBMIT`` flag can be passed instead of calling +the ``rte_dmadev_submit()`` function for example on the last enqueue of the burst. + +The following code from demonstrates how to enqueue a burst of copies to the +device and start the hardware processing of them: + +.. code-block:: C + + for (i = 0; i < BURST_SIZE; i++) { + if (rte_dmadev_copy(dev_id, vchan, rte_mbuf_data_iova(srcs[i]), + rte_mbuf_data_iova(dsts[i]), COPY_LEN, 0) < 0) { + PRINT_ERR("Error with rte_dmadev_copy for buffer %u\n", i); + return -1; + } + } + if (rte_dmadev_submit(dev_id, vchan) < 0) { + PRINT_ERR("Error with performing operations\n", i); + return -1; + } + +Filling an Area of Memory +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The driver also has support for the ``fill`` operation, where an area +of memory is overwritten, or filled, with a short pattern of data. +Fill operations can be performed in much the same was as copy operations +described above, just using the ``rte_dmadev_fill()`` function rather +than the ``rte_dmadev_copy()`` function. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 6cf368c8a5..52c8db9f66 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -5,6 +5,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> #include <rte_malloc.h> +#include <rte_prefetch.h> #include "ioat_internal.h" @@ -17,6 +18,12 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* IOAT operations. */ +enum rte_ioat_ops { + ioat_op_copy = 0, /* Standard DMA Operation */ + ioat_op_fill /* Block Fill */ +}; + /* Configure a device. */ static int ioat_dev_configure(struct rte_dmadev *dev __rte_unused, const struct rte_dmadev_conf *dev_conf) @@ -171,6 +178,87 @@ ioat_dev_close(struct rte_dmadev *dev) return 0; } +/* Trigger hardware to begin performing enqueued operations. */ +static inline void +__submit(struct ioat_dmadev *ioat) +{ + *ioat->doorbell = ioat->next_write - ioat->offset; + + ioat->last_write = ioat->next_write; +} + +/* External submit function wrapper. */ +static int +ioat_submit(struct rte_dmadev *dev, uint16_t qid __rte_unused) +{ + struct ioat_dmadev *ioat = (struct ioat_dmadev *)dev->dev_private; + + __submit(ioat); + + return 0; +} + +/* Write descriptor for enqueue. */ +static inline int +__write_desc(struct rte_dmadev *dev, uint32_t op, uint64_t src, phys_addr_t dst, + unsigned int length, uint64_t flags) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint16_t ret; + const unsigned short mask = ioat->qcfg.nb_desc - 1; + const unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + const unsigned short space = mask + read - write; + struct ioat_dma_hw_desc *desc; + + if (space == 0) + return -ENOSPC; + + ioat->next_write = write + 1; + write &= mask; + + desc = &ioat->desc_ring[write]; + desc->size = length; + desc->u.control_raw = (uint32_t)((op << IOAT_CMD_OP_SHIFT) | + (1 << IOAT_COMP_UPDATE_SHIFT)); + + /* In IOAT the fence ensures that all operations including the current one + * are completed before moving on, DMAdev assumes that the fence ensures + * all operations before the current one are completed before starting + * the current one, so in IOAT we set the fence for the previous descriptor. + */ + if (flags & RTE_DMA_OP_FLAG_FENCE) + ioat->desc_ring[(write - 1) & mask].u.control.fence = 1; + + desc->src_addr = src; + desc->dest_addr = dst; + + rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]); + + ret = (uint16_t)(ioat->next_write - 1); + + if (flags & RTE_DMA_OP_FLAG_SUBMIT) + __submit(ioat); + + return ret; +} + +/* Enqueue a fill operation onto the ioat device. */ +static int +ioat_enqueue_fill(struct rte_dmadev *dev, uint16_t qid __rte_unused, uint64_t pattern, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev, ioat_op_fill, pattern, dst, length, flags); +} + +/* Enqueue a copy operation onto the ioat device. */ +static int +ioat_enqueue_copy(struct rte_dmadev *dev, uint16_t qid __rte_unused, rte_iova_t src, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev, ioat_op_copy, src, dst, length, flags); +} + /* Dump DMA device info. */ static int ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) @@ -268,6 +356,10 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->copy = ioat_enqueue_copy; + dmadev->fill = ioat_enqueue_fill; + dmadev->submit = ioat_submit; + ioat = dmadev->data->dev_private; ioat->dmadev = dmadev; ioat->regs = dev->mem_resource[0].addr; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH 7/8] dma/ioat: add data path completion functions 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh ` (5 preceding siblings ...) 2021-08-27 17:25 ` [dpdk-dev] [PATCH 6/8] dma/ioat: add data path job submission functions Conor Walsh @ 2021-08-27 17:25 ` Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 8/8] dma/ioat: add statistics Conor Walsh ` (6 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-08-27 17:25 UTC (permalink / raw) To: dev; +Cc: bruce.richardson, fengchengwen, jerinj, kevin.laatz, Conor Walsh Add the data path functions for gathering completed operations from IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 48 ++++++++++- drivers/dma/ioat/ioat_dmadev.c | 141 +++++++++++++++++++++++++++++++++ 2 files changed, 188 insertions(+), 1 deletion(-) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 16acfda3e3..42bbc0dee6 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -96,7 +96,9 @@ Performing Data Copies To perform data copies using IOAT dmadev devices, the functions ``rte_dmadev_copy()`` and ``rte_dmadev_submit()`` should be used. Alternatively ``rte_dmadev_copy()`` can be called with the ``RTE_DMA_OP_FLAG_SUBMIT`` flag -set. +set. Once copies have been completed, the completion will be reported back when +the application calls ``rte_dmadev_completed()`` or +``rte_dmadev_completed_status()``. The ``rte_dmadev_copy()`` function enqueues a single copy to the device ring for copying at a later point. The parameters to the function @@ -135,6 +137,50 @@ device and start the hardware processing of them: return -1; } +To retrieve information about completed copies, the API +``rte_dmadev_completed()`` or ``rte_dmadev_completed_status ()``should be used. +``rte_dmadev_completed()`` will return to the application how many operations +have successfully completed upto a supplied maximum, it can also pass back the +index of the last completed operation and if a completion within the requested +batch has failed. If ``rte_dmadev_completed()`` passes back ``has_error`` as +``true`` then ``rte_dmadev_completed_status()`` should be called to retrieve +the detailed status of each completion in that batch. Once an error has occurred +``rte_dmadev_completed()`` will not process any more completions until +``rte_dmadev_completed_status()`` has been called to get the error information. +The supplied ``status`` array will contain either ``RTE_DMA_STATUS_SUCCESSFUL`` +if the operation was successful or a DMA error code if the operation failed. +The status codes supported by IOAT are: + +* ``RTE_DMA_STATUS_SUCCESSFUL``: The operation was successful. +* ``RTE_DMA_STATUS_INVALID_SRC_ADDR``: The operation failed due to an invalid source address. +* ``RTE_DMA_STATUS_INVALID_DST_ADDR``: The operation failed due to an invalid destination address. +* ``RTE_DMA_STATUS_INVALID_LENGTH``: The operation failed due to an invalid descriptor length. +* ``RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR``: The device could not read the descriptor. +* ``RTE_DMA_STATUS_ERROR_UNKNOWN``: The operation failed due to an unspecified error. + +``rte_dmadev_completed_status()`` can be used on it's own instead of with +``rte_dmadev_completed()`` but calling ``rte_dmadev_completed_status()`` may be +slower than ``rte_dmadev_completed()``. + +The following code shows how to retrieve the number of successfully completed +copies within a burst and then using ``rte_dmadev_completed_status()`` to check +which operation failed and reset the device: + +.. code-block:: C + + enum rte_dma_status_code status[COMP_BURST_SZ]; + uint16_t count, idx, status_count; + bool error = 0; + + count = rte_dmadev_completed(dev_id, vchan, COMP_BURST_SZ, &idx, &error); + + if (error){ + status_count = rte_dmadev_completed_status(dev_id, vchan, COMP_BURST_SZ, &idx, status); + } + +After this snippet the application would use the status array to find which copy +failed and handle it appropriately. + Filling an Area of Memory ~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 52c8db9f66..f765adcca6 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -6,6 +6,7 @@ #include <rte_dmadev_pmd.h> #include <rte_malloc.h> #include <rte_prefetch.h> +#include <rte_errno.h> #include "ioat_internal.h" @@ -316,6 +317,144 @@ ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) return 0; } +/* Returns the index of the last completed operation. */ +static inline uint16_t +__get_last_completed(struct ioat_dmadev *ioat, int *error) +{ + /* Status register contains the address of the completed operation */ + uint64_t status = ioat->status; + + /* lower 3 bits indicate "transfer status" : active, idle, halted. + * We can ignore bit 0. + */ + *error = status & IOAT_CHANSTS_STATUS; + + /* If we are just after recovering from an error the address returned by + * status will be 0, in this case we return the offset - 1 as the last + * completed. If not return the status value minus the chainaddr which + * gives us an offset into the ring. Right shifting by 6 (divide by 64) + * gives the index of the completion from the HW point of view and adding + * the offset translates the ring index from HW to SW point of view. + */ + if ((status & ~IOAT_CHANSTS_STATUS) == 0) + return ioat->offset - 1; + + return (status - ioat->ring_addr) >> 6; +} + +/* Translates IOAT ChanERRs to DMA error codes. */ +static inline enum rte_dma_status_code +__translate_status_ioat_to_dma(uint32_t chanerr) +{ + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_SRC_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_DST_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + return RTE_DMA_STATUS_INVALID_LENGTH; + else if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + return RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR; + else + return RTE_DMA_STATUS_ERROR_UNKNOWN; +} + +/* Returns details of operations that have been completed. */ +static uint16_t +ioat_completed(struct rte_dmadev *dev, uint16_t qid __rte_unused, const uint16_t max_ops, + uint16_t *last_idx, bool *has_error) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short last_completed, count; + int error, fails = 0; + + /* Do not do any work if there is an uncleared error. */ + if (ioat->failure != 0) { + *has_error = true; + *last_idx = ioat->next_read - 2; + return 0; + } + + last_completed = __get_last_completed(ioat, &error); + count = (last_completed + 1 - read) & mask; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || error != IOAT_CHANSTS_HALTED) { + ioat->next_read = read + count; + *last_idx = ioat->next_read - 1; + } else { + *has_error = true; + rte_errno = EIO; + ioat->failure = ioat->regs->chanerr; + ioat->next_read = read + count + 1; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + ioat_dev_dump(dev, stdout); + return 0; + } + __submit(ioat); + fails++; + *last_idx = ioat->next_read - 2; + } + + return count; +} + +/* Returns detailed status information about operations that have been completed. */ +static uint16_t +ioat_completed_status(struct rte_dmadev *dev, uint16_t qid __rte_unused, + uint16_t max_ops, uint16_t *last_idx, enum rte_dma_status_code *status) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short count, last_completed; + uint64_t fails = 0; + int error, i; + + last_completed = __get_last_completed(ioat, &error); + count = (last_completed + 1 - read) & mask; + + for (i = 0; i < RTE_MIN(count + 1, max_ops); i++) + status[i] = RTE_DMA_STATUS_SUCCESSFUL; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || error != IOAT_CHANSTS_HALTED) + ioat->next_read = read + count; + else { + rte_errno = EIO; + status[count] = __translate_status_ioat_to_dma(ioat->regs->chanerr); + count++; + ioat->next_read = read + count; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + ioat_dev_dump(dev, stdout); + return 0; + } + __submit(ioat); + fails++; + } + + if (ioat->failure > 0) { + status[0] = __translate_status_ioat_to_dma(ioat->failure); + count = RTE_MIN(count + 1, max_ops); + ioat->failure = 0; + } + + *last_idx = ioat->next_read - 1; + + return count; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -356,6 +495,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->completed = ioat_completed; + dmadev->completed_status = ioat_completed_status; dmadev->copy = ioat_enqueue_copy; dmadev->fill = ioat_enqueue_fill; dmadev->submit = ioat_submit; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH 8/8] dma/ioat: add statistics 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh ` (6 preceding siblings ...) 2021-08-27 17:25 ` [dpdk-dev] [PATCH 7/8] dma/ioat: add data path completion functions Conor Walsh @ 2021-08-27 17:25 ` Conor Walsh 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices Conor Walsh ` (5 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-08-27 17:25 UTC (permalink / raw) To: dev; +Cc: bruce.richardson, fengchengwen, jerinj, kevin.laatz, Conor Walsh Add statistic tracking for operations in IOAT. Signed-off-by: Conor Walsh <conor.walsh@intel.com> --- doc/guides/dmadevs/ioat.rst | 23 +++++++++++++++++++ drivers/dma/ioat/ioat_dmadev.c | 40 ++++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 42bbc0dee6..da2e073bbe 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -189,3 +189,26 @@ of memory is overwritten, or filled, with a short pattern of data. Fill operations can be performed in much the same was as copy operations described above, just using the ``rte_dmadev_fill()`` function rather than the ``rte_dmadev_copy()`` function. + +Querying Device Statistics +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The statistics from the IOAT dmadev device can be got via the +``rte_dmadev_stats_get()`` API in the ``rte_dmadev`` library. + +The statistics returned for each IOAT device are: + +* ``submitted``: The number of operations submitted to the device. +* ``completed``: The number of operations successfully completed by the device. +* ``errors``: The number of operations that failed. + +The stats function can be used as follows: + +.. code-block:: C + + struct rte_dmadev_stats stats; + if (rte_dmadev_stats_get(dmadev_id, vchan, &stats) >= 0) { + printf("Total submitted ops: %lu", stats.submitted); + printf("Total completed ops: %lu", stats.completed); + printf("Total failed ops: %lu", stats.errors); + } diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index f765adcca6..f6916f35e9 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -185,6 +185,8 @@ __submit(struct ioat_dmadev *ioat) { *ioat->doorbell = ioat->next_write - ioat->offset; + ioat->stats.submitted += (uint16_t)(ioat->next_write - ioat->last_write); + ioat->last_write = ioat->next_write; } @@ -313,6 +315,10 @@ ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) fprintf(f, " Dest: %#lx\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); fprintf(f, " Next: %#lx\n", ioat->desc_ring[ioat->next_read & mask].next); fprintf(f, " }\n"); + fprintf(f, " Key Stats { submitted: %"PRIx64", comp: %"PRIx64", failed: %"PRIx64" }\n", + ioat->stats.submitted, + ioat->stats.completed, + ioat->stats.errors); return 0; } @@ -402,6 +408,9 @@ ioat_completed(struct rte_dmadev *dev, uint16_t qid __rte_unused, const uint16_t *last_idx = ioat->next_read - 2; } + ioat->stats.completed += (count - fails); + ioat->stats.errors += fails; + return count; } @@ -452,9 +461,38 @@ ioat_completed_status(struct rte_dmadev *dev, uint16_t qid __rte_unused, *last_idx = ioat->next_read - 1; + ioat->stats.completed += (count - (unsigned short)fails); + ioat->stats.errors += fails; + return count; } +/* Retrieve the generic stats of a DMA device. */ +static int +ioat_stats_get(const struct rte_dmadev *dev, uint16_t vchan __rte_unused, + struct rte_dmadev_stats *rte_stats, uint32_t size) +{ + struct rte_dmadev_stats *stats = (&((struct ioat_dmadev *)dev->dev_private)->stats); + + if (size < sizeof(rte_stats)) + return -EINVAL; + if (rte_stats == NULL) + return -EINVAL; + + *rte_stats = *stats; + return 0; +} + +/* Reset the generic stat counters for the DMA device. */ +static int +ioat_stats_reset(struct rte_dmadev *dev, uint16_t vchan __rte_unused) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + memset(&ioat->stats, 0, sizeof(ioat->stats)); + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -466,6 +504,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_info_get = ioat_dev_info_get, .dev_start = ioat_dev_start, .dev_stop = ioat_dev_stop, + .stats_get = ioat_stats_get, + .stats_reset = ioat_stats_reset, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh ` (7 preceding siblings ...) 2021-08-27 17:25 ` [dpdk-dev] [PATCH 8/8] dma/ioat: add statistics Conor Walsh @ 2021-09-03 11:17 ` Conor Walsh 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 01/10] dma/ioat: add device probe and removal functions Conor Walsh ` (10 more replies) 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (4 subsequent siblings) 13 siblings, 11 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-03 11:17 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh This patchset adds a dmadev driver and associated documentation to support Intel QuickData Technology devices, part of the Intel I/O Acceleration Technology (Intel I/OAT). This driver is intended to ultimately replace the current IOAT part of the IOAT rawdev driver NOTE: This patchset has several dependencies: - v19 of the dmadev set [1] - v2 of the dmadev test suite [2] - v2 of the IDXD driver [3] [1] http://patches.dpdk.org/project/dpdk/list/?series=18629 [2] http://patches.dpdk.org/project/dpdk/list/?series=18607 [3] http://patches.dpdk.org/project/dpdk/list/?series=18658 --- v2: - Rebased on the above patchsets. - Added support for the vchan idle function. - Stop function now suspends IOAT channel to allow for reconfig. - dmadev_autotest can now be run multiple times using the IOAT driver without errors. - Added devbind updates for DMA devices - Removed some logically dead code found by coverity in the create function. Conor Walsh (10): dma/ioat: add device probe and removal functions dma/ioat: create dmadev instances on PCI probe dma/ioat: add datapath structures dma/ioat: add configuration functions dma/ioat: add start and stop functions dma/ioat: add data path job submission functions dma/ioat: add data path completion functions dma/ioat: add statistics dma/ioat: add support for vchan idle function devbind: move ioat device ID for ICX to dmadev category MAINTAINERS | 6 + doc/guides/dmadevs/index.rst | 1 + doc/guides/dmadevs/ioat.rst | 214 ++++++++ doc/guides/rel_notes/release_21_11.rst | 7 +- drivers/dma/ioat/ioat_dmadev.c | 694 +++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 294 +++++++++++ drivers/dma/ioat/ioat_internal.h | 44 ++ drivers/dma/ioat/meson.build | 7 + drivers/dma/ioat/version.map | 3 + drivers/dma/meson.build | 1 + usertools/dpdk-devbind.py | 5 +- 11 files changed, 1270 insertions(+), 6 deletions(-) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v2 01/10] dma/ioat: add device probe and removal functions 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices Conor Walsh @ 2021-09-03 11:17 ` Conor Walsh 2021-09-07 10:10 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 02/10] dma/ioat: create dmadev instances on PCI probe Conor Walsh ` (9 subsequent siblings) 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-03 11:17 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add the basic device probe/remove skeleton code and initial documentation for new IOAT DMA driver. Maintainers update is also included in this patch. Signed-off-by: Conor Walsh <conor.walsh@intel.com> --- MAINTAINERS | 6 +++ doc/guides/dmadevs/index.rst | 1 + doc/guides/dmadevs/ioat.rst | 64 ++++++++++++++++++++++++ doc/guides/rel_notes/release_21_11.rst | 7 +-- drivers/dma/ioat/ioat_dmadev.c | 69 ++++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 35 +++++++++++++ drivers/dma/ioat/ioat_internal.h | 20 ++++++++ drivers/dma/ioat/meson.build | 7 +++ drivers/dma/ioat/version.map | 3 ++ drivers/dma/meson.build | 1 + 10 files changed, 210 insertions(+), 3 deletions(-) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map diff --git a/MAINTAINERS b/MAINTAINERS index b4c614a229..00b319c811 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1208,6 +1208,12 @@ M: Kevin Laatz <kevin.laatz@intel.com> F: drivers/dma/idxd/ F: doc/guides/dmadevs/idxd.rst +Intel IOAT - EXPERIMENTAL +M: Bruce Richardson <bruce.richardson@intel.com> +M: Conor Walsh <conor.walsh@intel.com> +F: drivers/dma/ioat/ +F: doc/guides/dmadevs/ioat.rst + RegEx Drivers ------------- diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst index b30004fd65..0b01493c5a 100644 --- a/doc/guides/dmadevs/index.rst +++ b/doc/guides/dmadevs/index.rst @@ -12,3 +12,4 @@ application through DMAdev API. :numbered: idxd + ioat diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst new file mode 100644 index 0000000000..45a2e65d70 --- /dev/null +++ b/doc/guides/dmadevs/ioat.rst @@ -0,0 +1,64 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2021 Intel Corporation. + +.. include:: <isonum.txt> + +IOAT DMA Device Driver +======================= + +The ``ioat`` dmadev driver provides a poll-mode driver (PMD) for Intel\ +|reg| QuickData Technology which is part of part of Intel\ |reg| I/O +Acceleration Technology (`Intel I/OAT +<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_). +This PMD, when used on supported hardware, allows data copies, for example, +cloning packet data, to be accelerated by IOAT hardware rather than having to +be done by software, freeing up CPU cycles for other tasks. + +Hardware Requirements +---------------------- + +The ``dpdk-devbind.py`` script, included with DPDK, can be used to show the +presence of supported hardware. Running ``dpdk-devbind.py --status-dev dma`` +will show all the DMA devices on the system, IOAT devices are included in this +list. For Intel\ |reg| IOAT devices, the hardware will often be listed as +"Crystal Beach DMA", or "CBDMA" or on some newer systems '0b00' due to the +absence of pci-id database entries for them at this point. + +Compilation +------------ + +For builds using ``meson`` and ``ninja``, the driver will be built when the +target platform is x86-based. No additional compilation steps are necessary. + +Device Setup +------------- + +Intel\ |reg| IOAT devices will need to be bound to a suitable DPDK-supported +user-space IO driver such as ``vfio-pci`` in order to be used by DPDK. + +The ``dpdk-devbind.py`` script can be used to view the state of the devices using:: + + $ dpdk-devbind.py --status-dev dma + +The ``dpdk-devbind.py`` script can also be used to bind devices to a suitable driver. +For example:: + + $ dpdk-devbind.py -b vfio-pci 00:01.0 00:01.1 + +Device Probing and Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For devices bound to a suitable DPDK-supported driver (``vfio-pci``), the HW +devices will be found as part of the device scan done at application +initialization time without the need to pass parameters to the application. + +If the application does not require all the devices available an allowlist can +be used in the same way that other DPDK devices use them. + +For example:: + + $ dpdk-test -a <b:d:f> + +Once probed successfully, the device will appear as a ``dmadev``, that is a +"DMA device type" inside DPDK, and can be accessed using APIs from the +``rte_dmadev`` library. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index f7b0fcf3fd..6ce33e2787 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -60,10 +60,11 @@ New Features The dmadev library provides a DMA device framework for management and provision of hardware and software DMA devices. -* **Added IDXD dmadev driver implementation.** +* **Added Intel dmadev driver implementations.** - The IDXD dmadev driver provide device drivers for the Intel DSA devices. - This device driver can be used through the generic dmadev API. + The IDXD and IOAT dmadev drivers provide device drivers for Intel DSA + and IOAT devices. These device drivers can be used through the generic + dmadev API. Removed Items diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c new file mode 100644 index 0000000000..f3491d45b1 --- /dev/null +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#include <rte_bus_pci.h> +#include <rte_dmadev_pmd.h> + +#include "ioat_internal.h" + +static struct rte_pci_driver ioat_pmd_drv; + +RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); + +#define IOAT_PMD_NAME dmadev_ioat +#define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) + +/* Probe DMA device. */ +static int +ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); + + dev->device.driver = &drv->driver; + return 0; +} + +/* Remove DMA device. */ +static int +ioat_dmadev_remove(struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + + IOAT_PMD_INFO("Closing %s on NUMA node %d", + name, dev->device.numa_node); + + return 0; +} + +static const struct rte_pci_id pci_id_ioat_map[] = { + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_SKX) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX0) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX1) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX2) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX3) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX4) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX5) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX6) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX7) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXE) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXF) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_ICX) }, + { .vendor_id = 0, /* sentinel */ }, +}; + +static struct rte_pci_driver ioat_pmd_drv = { + .id_table = pci_id_ioat_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC, + .probe = ioat_dmadev_probe, + .remove = ioat_dmadev_remove, +}; + +RTE_PMD_REGISTER_PCI(IOAT_PMD_NAME, ioat_pmd_drv); +RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_NAME, pci_id_ioat_map); +RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_NAME, "* igb_uio | uio_pci_generic | vfio-pci"); diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h new file mode 100644 index 0000000000..eeabba41ef --- /dev/null +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#ifndef IOAT_HW_DEFS_H +#define IOAT_HW_DEFS_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> + +#define IOAT_VER_3_0 0x30 +#define IOAT_VER_3_3 0x33 + +#define IOAT_VENDOR_ID 0x8086 +#define IOAT_DEVICE_ID_SKX 0x2021 +#define IOAT_DEVICE_ID_BDX0 0x6f20 +#define IOAT_DEVICE_ID_BDX1 0x6f21 +#define IOAT_DEVICE_ID_BDX2 0x6f22 +#define IOAT_DEVICE_ID_BDX3 0x6f23 +#define IOAT_DEVICE_ID_BDX4 0x6f24 +#define IOAT_DEVICE_ID_BDX5 0x6f25 +#define IOAT_DEVICE_ID_BDX6 0x6f26 +#define IOAT_DEVICE_ID_BDX7 0x6f27 +#define IOAT_DEVICE_ID_BDXE 0x6f2E +#define IOAT_DEVICE_ID_BDXF 0x6f2F +#define IOAT_DEVICE_ID_ICX 0x0b00 + +#ifdef __cplusplus +} +#endif + +#endif /* IOAT_HW_DEFS_H */ diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h new file mode 100644 index 0000000000..f1ec12a919 --- /dev/null +++ b/drivers/dma/ioat/ioat_internal.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021 Intel Corporation + */ + +#ifndef _IOAT_INTERNAL_H_ +#define _IOAT_INTERNAL_H_ + +#include "ioat_hw_defs.h" + +extern int ioat_pmd_logtype; + +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + ioat_pmd_logtype, "IOAT: %s(): " fmt "\n", __func__, ##args) + +#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) +#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) +#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) +#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) + +#endif /* _IOAT_INTERNAL_H_ */ diff --git a/drivers/dma/ioat/meson.build b/drivers/dma/ioat/meson.build new file mode 100644 index 0000000000..d67fac96fb --- /dev/null +++ b/drivers/dma/ioat/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021 Intel Corporation + +build = dpdk_conf.has('RTE_ARCH_X86') +reason = 'only supported on x86' +sources = files('ioat_dmadev.c') +deps += ['bus_pci', 'dmadev'] diff --git a/drivers/dma/ioat/version.map b/drivers/dma/ioat/version.map new file mode 100644 index 0000000000..c2e0723b4c --- /dev/null +++ b/drivers/dma/ioat/version.map @@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +}; diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build index 0b01b6a8ab..f8f5154a40 100644 --- a/drivers/dma/meson.build +++ b/drivers/dma/meson.build @@ -7,6 +7,7 @@ endif drivers = [ 'idxd', + 'ioat', 'skeleton', ] std_deps = ['dmadev'] -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v2 01/10] dma/ioat: add device probe and removal functions 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 01/10] dma/ioat: add device probe and removal functions Conor Walsh @ 2021-09-07 10:10 ` Kevin Laatz 0 siblings, 0 replies; 132+ messages in thread From: Kevin Laatz @ 2021-09-07 10:10 UTC (permalink / raw) To: Conor Walsh, bruce.richardson, fengchengwen, jerinj; +Cc: dev On 03/09/2021 12:17, Conor Walsh wrote: > Add the basic device probe/remove skeleton code and initial documentation > for new IOAT DMA driver. Maintainers update is also included in this > patch. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > --- > MAINTAINERS | 6 +++ > doc/guides/dmadevs/index.rst | 1 + > doc/guides/dmadevs/ioat.rst | 64 ++++++++++++++++++++++++ > doc/guides/rel_notes/release_21_11.rst | 7 +-- > drivers/dma/ioat/ioat_dmadev.c | 69 ++++++++++++++++++++++++++ > drivers/dma/ioat/ioat_hw_defs.h | 35 +++++++++++++ > drivers/dma/ioat/ioat_internal.h | 20 ++++++++ > drivers/dma/ioat/meson.build | 7 +++ > drivers/dma/ioat/version.map | 3 ++ > drivers/dma/meson.build | 1 + > 10 files changed, 210 insertions(+), 3 deletions(-) > create mode 100644 doc/guides/dmadevs/ioat.rst > create mode 100644 drivers/dma/ioat/ioat_dmadev.c > create mode 100644 drivers/dma/ioat/ioat_hw_defs.h > create mode 100644 drivers/dma/ioat/ioat_internal.h > create mode 100644 drivers/dma/ioat/meson.build > create mode 100644 drivers/dma/ioat/version.map > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v2 02/10] dma/ioat: create dmadev instances on PCI probe 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 01/10] dma/ioat: add device probe and removal functions Conor Walsh @ 2021-09-03 11:17 ` Conor Walsh 2021-09-07 10:11 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 03/10] dma/ioat: add datapath structures Conor Walsh ` (8 subsequent siblings) 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-03 11:17 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh When a suitable device is found during the PCI probe, create a dmadev instance for each channel. Internal structures and HW definitions required for device creation are also included. Signed-off-by: Conor Walsh <conor.walsh@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 108 ++++++++++++++++++++++++++++++- drivers/dma/ioat/ioat_hw_defs.h | 45 +++++++++++++ drivers/dma/ioat/ioat_internal.h | 24 +++++++ 3 files changed, 175 insertions(+), 2 deletions(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index f3491d45b1..aa03dd1cd2 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -4,6 +4,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> +#include <rte_malloc.h> #include "ioat_internal.h" @@ -14,6 +15,109 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Create a DMA device. */ +static int +ioat_dmadev_create(const char *name, struct rte_pci_device *dev) +{ + static const struct rte_dmadev_ops ioat_dmadev_ops = { }; + + struct rte_dmadev *dmadev = NULL; + struct ioat_dmadev *ioat = NULL; + int retry = 0; + + if (!name) { + IOAT_PMD_ERR("Invalid name of the device!"); + return -EINVAL; + } + + /* Allocate device structure. */ + dmadev = rte_dmadev_pmd_allocate(name); + if (dmadev == NULL) { + IOAT_PMD_ERR("Unable to allocate dma device"); + return -ENOMEM; + } + + dmadev->device = &dev->device; + + dmadev->data->dev_private = rte_malloc_socket(NULL, sizeof(*ioat), + 0, dmadev->device->numa_node); + dmadev->dev_private = dmadev->data->dev_private; + + dmadev->dev_ops = &ioat_dmadev_ops; + + ioat = dmadev->data->dev_private; + ioat->dmadev = dmadev; + ioat->regs = dev->mem_resource[0].addr; + ioat->doorbell = &ioat->regs->dmacount; + ioat->qcfg.nb_desc = 0; + ioat->desc_ring = NULL; + + /* Do device initialization - reset and set error behaviour. */ + if (ioat->regs->chancnt != 1) + IOAT_PMD_WARN("%s: Channel count == %d\n", __func__, + ioat->regs->chancnt); + + /* Locked by someone else. */ + if (ioat->regs->chanctrl & IOAT_CHANCTRL_CHANNEL_IN_USE) { + IOAT_PMD_WARN("%s: Channel appears locked\n", __func__); + ioat->regs->chanctrl = 0; + } + + /* clear any previous errors */ + if (ioat->regs->chanerr != 0) { + uint32_t val = ioat->regs->chanerr; + ioat->regs->chanerr = val; + } + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + rte_delay_ms(1); + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + rte_delay_ms(1); + while (ioat->regs->chancmd & IOAT_CHANCMD_RESET) { + ioat->regs->chainaddr = 0; + rte_delay_ms(1); + if (++retry >= 200) { + IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=%#"PRIx8 + ", CHANSTS=%#"PRIx64", CHANERR=%#"PRIx32"\n", + __func__, + ioat->regs->chancmd, + ioat->regs->chansts, + ioat->regs->chanerr); + return -EIO; + } + } + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + + return 0; + +} + +/* Destroy a DMA device. */ +static int +ioat_dmadev_destroy(const char *name) +{ + struct rte_dmadev *dev; + int ret; + + if (!name) { + IOAT_PMD_ERR("Invalid device name"); + return -EINVAL; + } + + dev = rte_dmadev_get_device_by_name(name); + if (dev == NULL) { + IOAT_PMD_ERR("Invalid device name (%s)", name); + return -EINVAL; + } + + ret = rte_dmadev_pmd_release(dev); + if (ret) + IOAT_PMD_DEBUG("Device cleanup failed"); + + return 0; +} + /* Probe DMA device. */ static int ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) @@ -24,7 +128,7 @@ ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); dev->device.driver = &drv->driver; - return 0; + return ioat_dmadev_create(name, dev); } /* Remove DMA device. */ @@ -38,7 +142,7 @@ ioat_dmadev_remove(struct rte_pci_device *dev) IOAT_PMD_INFO("Closing %s on NUMA node %d", name, dev->device.numa_node); - return 0; + return ioat_dmadev_destroy(name); } static const struct rte_pci_id pci_id_ioat_map[] = { diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index eeabba41ef..73bdf548b3 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -11,6 +11,8 @@ extern "C" { #include <stdint.h> +#define IOAT_PCI_CHANERR_INT_OFFSET 0x180 + #define IOAT_VER_3_0 0x30 #define IOAT_VER_3_3 0x33 @@ -28,6 +30,49 @@ extern "C" { #define IOAT_DEVICE_ID_BDXF 0x6f2F #define IOAT_DEVICE_ID_ICX 0x0b00 +#define IOAT_COMP_UPDATE_SHIFT 3 +#define IOAT_CMD_OP_SHIFT 24 + +/* DMA Channel Registers */ +#define IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK 0xF000 +#define IOAT_CHANCTRL_COMPL_DCA_EN 0x0200 +#define IOAT_CHANCTRL_CHANNEL_IN_USE 0x0100 +#define IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL 0x0020 +#define IOAT_CHANCTRL_ERR_INT_EN 0x0010 +#define IOAT_CHANCTRL_ANY_ERR_ABORT_EN 0x0008 +#define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 +#define IOAT_CHANCTRL_INT_REARM 0x0001 + +struct ioat_registers { + uint8_t chancnt; + uint8_t xfercap; + uint8_t genctrl; + uint8_t intrctrl; + uint32_t attnstatus; + uint8_t cbver; /* 0x08 */ + uint8_t reserved4[0x3]; /* 0x09 */ + uint16_t intrdelay; /* 0x0C */ + uint16_t cs_status; /* 0x0E */ + uint32_t dmacapability; /* 0x10 */ + uint8_t reserved5[0x6C]; /* 0x14 */ + uint16_t chanctrl; /* 0x80 */ + uint8_t reserved6[0x2]; /* 0x82 */ + uint8_t chancmd; /* 0x84 */ + uint8_t reserved3[1]; /* 0x85 */ + uint16_t dmacount; /* 0x86 */ + uint64_t chansts; /* 0x88 */ + uint64_t chainaddr; /* 0x90 */ + uint64_t chancmp; /* 0x98 */ + uint8_t reserved2[0x8]; /* 0xA0 */ + uint32_t chanerr; /* 0xA8 */ + uint32_t chanerrmask; /* 0xAC */ +} __rte_packed; + +#define IOAT_CHANCMD_RESET 0x20 +#define IOAT_CHANCMD_SUSPEND 0x04 + +#define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ + #ifdef __cplusplus } #endif diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h index f1ec12a919..133fceae86 100644 --- a/drivers/dma/ioat/ioat_internal.h +++ b/drivers/dma/ioat/ioat_internal.h @@ -7,6 +7,30 @@ #include "ioat_hw_defs.h" +struct ioat_dmadev { + struct rte_dmadev *dmadev; + struct rte_dmadev_vchan_conf qcfg; + struct rte_dmadev_stats stats; + + volatile uint16_t *doorbell __rte_cache_aligned; + phys_addr_t status_addr; + phys_addr_t ring_addr; + + struct ioat_dma_hw_desc *desc_ring; + + unsigned short next_read; + unsigned short next_write; + unsigned short last_write; /* Used to compute submitted count. */ + unsigned short offset; /* Used after a device recovery when counts -> 0. */ + unsigned int failure; /* Used to store chanerr for error handling. */ + + /* To report completions, the device will write status back here. */ + volatile uint64_t status __rte_cache_aligned; + + /* Pointer to the register bar. */ + volatile struct ioat_registers *regs; +}; + extern int ioat_pmd_logtype; #define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v2 02/10] dma/ioat: create dmadev instances on PCI probe 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 02/10] dma/ioat: create dmadev instances on PCI probe Conor Walsh @ 2021-09-07 10:11 ` Kevin Laatz 0 siblings, 0 replies; 132+ messages in thread From: Kevin Laatz @ 2021-09-07 10:11 UTC (permalink / raw) To: Conor Walsh, bruce.richardson, fengchengwen, jerinj; +Cc: dev On 03/09/2021 12:17, Conor Walsh wrote: > When a suitable device is found during the PCI probe, create a dmadev > instance for each channel. Internal structures and HW definitions required > for device creation are also included. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > --- > drivers/dma/ioat/ioat_dmadev.c | 108 ++++++++++++++++++++++++++++++- > drivers/dma/ioat/ioat_hw_defs.h | 45 +++++++++++++ > drivers/dma/ioat/ioat_internal.h | 24 +++++++ > 3 files changed, 175 insertions(+), 2 deletions(-) Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v2 03/10] dma/ioat: add datapath structures 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 01/10] dma/ioat: add device probe and removal functions Conor Walsh 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 02/10] dma/ioat: create dmadev instances on PCI probe Conor Walsh @ 2021-09-03 11:17 ` Conor Walsh 2021-09-07 10:11 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 04/10] dma/ioat: add configuration functions Conor Walsh ` (7 subsequent siblings) 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-03 11:17 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add data structures required for the data path of IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 61 ++++++++- drivers/dma/ioat/ioat_hw_defs.h | 214 ++++++++++++++++++++++++++++++++ 2 files changed, 274 insertions(+), 1 deletion(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index aa03dd1cd2..d65881d852 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -15,11 +15,70 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Dump DMA device info. */ +static int +ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint64_t chansts_masked = ioat->regs->chansts & IOAT_CHANSTS_STATUS; + uint32_t chanerr = ioat->regs->chanerr; + uint64_t mask = (ioat->qcfg.nb_desc - 1); + fprintf(f, "========= IOAT =========\n"); + fprintf(f, " Channel status: %s [%#lx]\n", + chansts_readable[chansts_masked], chansts_masked); + fprintf(f, " ChainADDR: %#lx\n", ioat->regs->chainaddr); + if (chanerr == 0) { + fprintf(f, " No Channel Errors\n"); + } else { + fprintf(f, " ChanERR: %#x\n", chanerr); + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + fprintf(f, " Invalid Source Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + fprintf(f, " Invalid Destination Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + fprintf(f, " Invalid Descriptor Length\n"); + if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + fprintf(f, " Descriptor Read Error\n"); + if ((chanerr & ~(IOAT_CHANERR_INVALID_SRC_ADDR_MASK | + IOAT_CHANERR_INVALID_DST_ADDR_MASK | + IOAT_CHANERR_INVALID_LENGTH_MASK | + IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK)) != 0) + fprintf(f, " Unknown Error(s)\n"); + } + fprintf(f, "== Private Data ==\n"); + fprintf(f, " Config: { ring_size: %u }\n", ioat->qcfg.nb_desc); + fprintf(f, " Status: %#lx\n", ioat->status); + fprintf(f, " Status IOVA: %#lx\n", ioat->status_addr); + fprintf(f, " Status ADDR: %p\n", &ioat->status); + fprintf(f, " Ring IOVA: %#lx\n", ioat->ring_addr); + fprintf(f, " Ring ADDR: %#lx\n", ioat->desc_ring[0].next-64); + fprintf(f, " Next write: %u\n", ioat->next_write); + fprintf(f, " Next read: %u\n", ioat->next_read); + struct ioat_dma_hw_desc *desc_ring = &ioat->desc_ring[(ioat->next_write - 1) & mask]; + fprintf(f, " Last Descriptor Written {\n"); + fprintf(f, " Size: %u\n", desc_ring->size); + fprintf(f, " Control: %#x\n", desc_ring->u.control_raw); + fprintf(f, " Src: %#lx\n", desc_ring->src_addr); + fprintf(f, " Dest: %#lx\n", desc_ring->dest_addr); + fprintf(f, " Next: %#lx\n", desc_ring->next); + fprintf(f, " }\n"); + fprintf(f, " Next Descriptor {\n"); + fprintf(f, " Size: %u\n", ioat->desc_ring[ioat->next_read & mask].size); + fprintf(f, " Src: %#lx\n", ioat->desc_ring[ioat->next_read & mask].src_addr); + fprintf(f, " Dest: %#lx\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); + fprintf(f, " Next: %#lx\n", ioat->desc_ring[ioat->next_read & mask].next); + fprintf(f, " }\n"); + + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { - static const struct rte_dmadev_ops ioat_dmadev_ops = { }; + static const struct rte_dmadev_ops ioat_dmadev_ops = { + .dev_dump = ioat_dev_dump, + }; struct rte_dmadev *dmadev = NULL; struct ioat_dmadev *ioat = NULL; diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index 73bdf548b3..1f47209e60 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -43,6 +43,14 @@ extern "C" { #define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 #define IOAT_CHANCTRL_INT_REARM 0x0001 +/* DMA Channel Capabilities */ +#define IOAT_DMACAP_PB (1 << 0) +#define IOAT_DMACAP_DCA (1 << 4) +#define IOAT_DMACAP_BFILL (1 << 6) +#define IOAT_DMACAP_XOR (1 << 8) +#define IOAT_DMACAP_PQ (1 << 9) +#define IOAT_DMACAP_DMA_DIF (1 << 10) + struct ioat_registers { uint8_t chancnt; uint8_t xfercap; @@ -71,8 +79,214 @@ struct ioat_registers { #define IOAT_CHANCMD_RESET 0x20 #define IOAT_CHANCMD_SUSPEND 0x04 +#define IOAT_CHANSTS_STATUS 0x7ULL +#define IOAT_CHANSTS_ACTIVE 0x0 +#define IOAT_CHANSTS_IDLE 0x1 +#define IOAT_CHANSTS_SUSPENDED 0x2 +#define IOAT_CHANSTS_HALTED 0x3 +#define IOAT_CHANSTS_ARMED 0x4 + +#define IOAT_CHANERR_INVALID_SRC_ADDR_MASK (1 << 0) +#define IOAT_CHANERR_INVALID_DST_ADDR_MASK (1 << 1) +#define IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK (1 << 8) +#define IOAT_CHANERR_INVALID_LENGTH_MASK (1 << 10) + +const char *chansts_readable[] = { + "ACTIVE", /* 0x0 */ + "IDLE", /* 0x1 */ + "SUSPENDED", /* 0x2 */ + "HALTED", /* 0x3 */ + "ARMED" /* 0x4 */ +}; + +#define IOAT_CHANSTS_UNAFFILIATED_ERROR 0x8ULL +#define IOAT_CHANSTS_SOFT_ERROR 0x10ULL + +#define IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK (~0x3FULL) + #define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ +struct ioat_dma_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t null: 1; + uint32_t src_page_break: 1; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_COPY 0x00 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t reserved2; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_fill_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t reserved: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t reserved2: 2; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t reserved3: 15; +#define IOAT_OP_FILL 0x01 + uint32_t op: 8; + } control; + } u; + uint64_t src_data; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t next_dest_addr; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_xor_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_XOR 0x87 +#define IOAT_OP_XOR_VAL 0x88 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint64_t src_addr4; + uint64_t src_addr5; +}; + +struct ioat_xor_ext_hw_desc { + uint64_t src_addr6; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t next; + uint64_t reserved[4]; +}; + +struct ioat_pq_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 11; +#define IOAT_OP_PQ 0x89 +#define IOAT_OP_PQ_VAL 0x8a + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint8_t coef[8]; + uint64_t q_addr; +}; + +struct ioat_pq_ext_hw_desc { + uint64_t src_addr4; + uint64_t src_addr5; + uint64_t src_addr6; + uint64_t next; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t reserved[2]; +}; + +struct ioat_pq_update_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_cnt: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 3; + uint32_t coef: 8; +#define IOAT_OP_PQ_UP 0x8b + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t p_src; + uint64_t q_src; + uint64_t q_addr; +}; + +union ioat_hw_desc { + struct ioat_dma_hw_desc dma; + struct ioat_fill_hw_desc fill; + struct ioat_xor_hw_desc xor_desc; + struct ioat_xor_ext_hw_desc xor_ext; + struct ioat_pq_hw_desc pq; + struct ioat_pq_ext_hw_desc pq_ext; + struct ioat_pq_update_hw_desc pq_update; +}; + +#define GENSTS_DEV_STATE_MASK 0x03 +#define CMDSTATUS_ACTIVE_SHIFT 31 +#define CMDSTATUS_ACTIVE_MASK (1 << 31) +#define CMDSTATUS_ERR_MASK 0xFF + #ifdef __cplusplus } #endif -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v2 03/10] dma/ioat: add datapath structures 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 03/10] dma/ioat: add datapath structures Conor Walsh @ 2021-09-07 10:11 ` Kevin Laatz 0 siblings, 0 replies; 132+ messages in thread From: Kevin Laatz @ 2021-09-07 10:11 UTC (permalink / raw) To: Conor Walsh, bruce.richardson, fengchengwen, jerinj; +Cc: dev On 03/09/2021 12:17, Conor Walsh wrote: > Add data structures required for the data path of IOAT devices. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> > --- > drivers/dma/ioat/ioat_dmadev.c | 61 ++++++++- > drivers/dma/ioat/ioat_hw_defs.h | 214 ++++++++++++++++++++++++++++++++ > 2 files changed, 274 insertions(+), 1 deletion(-) > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v2 04/10] dma/ioat: add configuration functions 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices Conor Walsh ` (2 preceding siblings ...) 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 03/10] dma/ioat: add datapath structures Conor Walsh @ 2021-09-03 11:17 ` Conor Walsh 2021-09-07 10:11 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 05/10] dma/ioat: add start and stop functions Conor Walsh ` (6 subsequent siblings) 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-03 11:17 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add functions for device configuration. The info_get and close functions are included here also. info_get can be useful for checking successful configuration and close is used by the dmadev api when releasing a configured device. Signed-off-by: Conor Walsh <conor.walsh@intel.com> --- doc/guides/dmadevs/ioat.rst | 24 ++++++++++ drivers/dma/ioat/ioat_dmadev.c | 88 ++++++++++++++++++++++++++++++++++ 2 files changed, 112 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 45a2e65d70..b6d88fe966 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -62,3 +62,27 @@ For example:: Once probed successfully, the device will appear as a ``dmadev``, that is a "DMA device type" inside DPDK, and can be accessed using APIs from the ``rte_dmadev`` library. + +Device Configuration +~~~~~~~~~~~~~~~~~~~~~ + +Configuring an IOAT dmadev device is done using the ``rte_dmadev_configure()`` +and ``rte_dmadev_vchan_setup()`` APIs. ``rte_dmadev_configure()`` uses the structure +``rte_dmadev_conf`` to configure an IOAT device, within this struct the number +of virtual DMA channels for the device is set and silent mode can be enabled. +Each IOAT device can only use 1 virtual DMA channel and silent mode is not +supported so these will always be set to ``1`` and ``false``. +``rte_dmadev_vchan_setup()`` uses the structure ``rte_dmadev_vchan_conf`` to setup +a virtual DMA channel within this struct the transfer direction and ring size +are set. Generally for an onboard Intel\ |reg| IOAT device the transfer direction +will be set to ``RTE_DMA_DIR_MEM_TO_MEM`` to copy from memory to memory (more info +available in the dmadev API guide). The ring size must be a power of two, between +64 and 4096. + +The following code shows how the device is configured in ``test_dmadev.c``: + +.. literalinclude:: ../../../app/test/test_dmadev.c + :language: c + :start-after: Setup of the dmadev device. 8< + :end-before: >8 End of setup of the dmadev device. + :dedent: 1 diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index d65881d852..94f9139e0d 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -12,9 +12,93 @@ static struct rte_pci_driver ioat_pmd_drv; RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); +#define DESC_SZ sizeof(struct ioat_dma_hw_desc) + #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Configure a device. */ +static int +ioat_dev_configure(struct rte_dmadev *dev __rte_unused, const struct rte_dmadev_conf *dev_conf) +{ + if (dev_conf->nb_vchans != 1) + return -EINVAL; + return 0; +} + +/* Setup a virtual channel for IOAT, only 1 vchan is supported. */ +static int +ioat_vchan_setup(struct rte_dmadev *dev, uint16_t vchan __rte_unused, + const struct rte_dmadev_vchan_conf *qconf) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint16_t max_desc = qconf->nb_desc; + int i; + + ioat->qcfg = *qconf; + + if (!rte_is_power_of_2(max_desc)) { + max_desc = rte_align32pow2(max_desc); + IOAT_PMD_DEBUG("DMA dev %u using %u descriptors", dev->data->dev_id, max_desc); + ioat->qcfg.nb_desc = max_desc; + } + + /* In case we are reconfiguring a device, free any existing memory. */ + rte_free(ioat->desc_ring); + + ioat->desc_ring = rte_zmalloc(NULL, sizeof(*ioat->desc_ring) * max_desc, 0); + if (ioat->desc_ring == NULL) + return -ENOMEM; + + ioat->ring_addr = rte_mem_virt2iova(ioat->desc_ring); + + ioat->status_addr = rte_mem_virt2iova(ioat) + offsetof(struct ioat_dmadev, status); + + /* Ensure all counters are reset, if reconfiguring/restarting device. */ + ioat->next_read = 0; + ioat->next_write = 0; + ioat->last_write = 0; + ioat->offset = 0; + ioat->failure = 0; + + /* Reset Stats. */ + memset(&ioat->stats, 0, sizeof(ioat->stats)); + + /* Configure descriptor ring - each one points to next. */ + for (i = 0; i < ioat->qcfg.nb_desc; i++) { + ioat->desc_ring[i].next = ioat->ring_addr + + (((i + 1) % ioat->qcfg.nb_desc) * DESC_SZ); + } + + return 0; +} + +/* Get device information of a device. */ +static int +ioat_dev_info_get(const struct rte_dmadev *dev, struct rte_dmadev_info *info, uint32_t size) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + if (size < sizeof(*info)) + return -EINVAL; + info->device = dev->device; + info->dev_capa = RTE_DMADEV_CAPA_MEM_TO_MEM | + RTE_DMADEV_CAPA_OPS_COPY | RTE_DMADEV_CAPA_OPS_FILL, + info->max_vchans = 1; + info->min_desc = 32; + info->max_desc = 4096; + info->nb_vchans = (ioat->desc_ring == NULL); + return 0; +} + +/* Close a configured device. */ +static int +ioat_dev_close(struct rte_dmadev *dev) +{ + RTE_SET_USED(dev); + return 0; +} + /* Dump DMA device info. */ static int ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) @@ -77,7 +161,11 @@ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { static const struct rte_dmadev_ops ioat_dmadev_ops = { + .dev_close = ioat_dev_close, + .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, + .dev_info_get = ioat_dev_info_get, + .vchan_setup = ioat_vchan_setup, }; struct rte_dmadev *dmadev = NULL; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v2 04/10] dma/ioat: add configuration functions 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 04/10] dma/ioat: add configuration functions Conor Walsh @ 2021-09-07 10:11 ` Kevin Laatz 0 siblings, 0 replies; 132+ messages in thread From: Kevin Laatz @ 2021-09-07 10:11 UTC (permalink / raw) To: Conor Walsh, bruce.richardson, fengchengwen, jerinj; +Cc: dev On 03/09/2021 12:17, Conor Walsh wrote: > Add functions for device configuration. The info_get and close functions > are included here also. info_get can be useful for checking successful > configuration and close is used by the dmadev api when releasing a > configured device. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > --- > doc/guides/dmadevs/ioat.rst | 24 ++++++++++ > drivers/dma/ioat/ioat_dmadev.c | 88 ++++++++++++++++++++++++++++++++++ > 2 files changed, 112 insertions(+) > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v2 05/10] dma/ioat: add start and stop functions 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices Conor Walsh ` (3 preceding siblings ...) 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 04/10] dma/ioat: add configuration functions Conor Walsh @ 2021-09-03 11:17 ` Conor Walsh 2021-09-07 10:11 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 06/10] dma/ioat: add data path job submission functions Conor Walsh ` (5 subsequent siblings) 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-03 11:17 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add start, stop and recover functions for IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> --- doc/guides/dmadevs/ioat.rst | 3 ++ drivers/dma/ioat/ioat_dmadev.c | 87 ++++++++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index b6d88fe966..f7742642b5 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -86,3 +86,6 @@ The following code shows how the device is configured in ``test_dmadev.c``: :start-after: Setup of the dmadev device. 8< :end-before: >8 End of setup of the dmadev device. :dedent: 1 + +Once configured, the device can then be made ready for use by calling the +``rte_dmadev_start()`` API. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 94f9139e0d..9f9feecd49 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -73,6 +73,91 @@ ioat_vchan_setup(struct rte_dmadev *dev, uint16_t vchan __rte_unused, return 0; } +/* Recover IOAT device. */ +static inline int +__ioat_recover(struct ioat_dmadev *ioat) +{ + uint32_t chanerr, retry = 0; + uint16_t mask = ioat->qcfg.nb_desc - 1; + + /* Clear any channel errors. Reading and writing to chanerr does this. */ + chanerr = ioat->regs->chanerr; + ioat->regs->chanerr = chanerr; + + /* Reset Channel. */ + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + + /* Write new chain address to trigger state change. */ + ioat->regs->chainaddr = ioat->desc_ring[(ioat->next_read - 1) & mask].next; + /* Ensure channel control and status addr are correct. */ + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + ioat->regs->chancmp = ioat->status_addr; + + /* Allow HW time to move to the ARMED state. */ + do { + rte_pause(); + retry++; + } while (ioat->regs->chansts != IOAT_CHANSTS_ARMED && retry < 200); + + /* Exit as failure if device is still HALTED. */ + if (ioat->regs->chansts != IOAT_CHANSTS_ARMED) + return -1; + + /* Store next write as offset as recover will move HW and SW ring out of sync. */ + ioat->offset = ioat->next_read; + + /* Prime status register with previous address. */ + ioat->status = ioat->desc_ring[(ioat->next_read - 2) & mask].next; + + return 0; +} + +/* Start a configured device. */ +static int +ioat_dev_start(struct rte_dmadev *dev) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + if (ioat->qcfg.nb_desc == 0 || ioat->desc_ring == NULL) + return -EBUSY; + + /* Inform hardware of where the descriptor ring is. */ + ioat->regs->chainaddr = ioat->ring_addr; + /* Inform hardware of where to write the status/completions. */ + ioat->regs->chancmp = ioat->status_addr; + + /* Prime the status register to be set to the last element. */ + ioat->status = ioat->ring_addr + ((ioat->qcfg.nb_desc - 1) * DESC_SZ); + + printf("IOAT.status: %s [%#lx]\n", + chansts_readable[ioat->status & IOAT_CHANSTS_STATUS], + ioat->status); + + if ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_HALTED) { + IOAT_PMD_WARN("Device HALTED on start, attempting to recover\n"); + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device couldn't be recovered"); + return -1; + } + } + + return 0; +} + +/* Stop a configured device. */ +static int +ioat_dev_stop(struct rte_dmadev *dev) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + /* Allow the device time to suspend itself. */ + rte_delay_ms(1); + + return 0; +} + /* Get device information of a device. */ static int ioat_dev_info_get(const struct rte_dmadev *dev, struct rte_dmadev_info *info, uint32_t size) @@ -165,6 +250,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, .dev_info_get = ioat_dev_info_get, + .dev_start = ioat_dev_start, + .dev_stop = ioat_dev_stop, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v2 05/10] dma/ioat: add start and stop functions 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 05/10] dma/ioat: add start and stop functions Conor Walsh @ 2021-09-07 10:11 ` Kevin Laatz 2021-09-07 10:35 ` Walsh, Conor 0 siblings, 1 reply; 132+ messages in thread From: Kevin Laatz @ 2021-09-07 10:11 UTC (permalink / raw) To: Conor Walsh, bruce.richardson, fengchengwen, jerinj; +Cc: dev On 03/09/2021 12:17, Conor Walsh wrote: > Add start, stop and recover functions for IOAT devices. > > Signed-off-by: Conor Walsh<conor.walsh@intel.com> > Signed-off-by: Bruce Richardson<bruce.richardson@intel.com> > --- > doc/guides/dmadevs/ioat.rst | 3 ++ > drivers/dma/ioat/ioat_dmadev.c | 87 ++++++++++++++++++++++++++++++++++ > 2 files changed, 90 insertions(+) > > diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst > index b6d88fe966..f7742642b5 100644 > --- a/doc/guides/dmadevs/ioat.rst > +++ b/doc/guides/dmadevs/ioat.rst > @@ -86,3 +86,6 @@ The following code shows how the device is configured in ``test_dmadev.c``: > :start-after: Setup of the dmadev device. 8< > :end-before: >8 End of setup of the dmadev device. > :dedent: 1 > + > +Once configured, the device can then be made ready for use by calling the > +``rte_dmadev_start()`` API. > diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c > index 94f9139e0d..9f9feecd49 100644 > --- a/drivers/dma/ioat/ioat_dmadev.c > +++ b/drivers/dma/ioat/ioat_dmadev.c > @@ -73,6 +73,91 @@ ioat_vchan_setup(struct rte_dmadev *dev, uint16_t vchan __rte_unused, > return 0; > } > > +/* Recover IOAT device. */ > +static inline int > +__ioat_recover(struct ioat_dmadev *ioat) > +{ > + uint32_t chanerr, retry = 0; > + uint16_t mask = ioat->qcfg.nb_desc - 1; > + > + /* Clear any channel errors. Reading and writing to chanerr does this. */ > + chanerr = ioat->regs->chanerr; > + ioat->regs->chanerr = chanerr; > + > + /* Reset Channel. */ > + ioat->regs->chancmd = IOAT_CHANCMD_RESET; > + > + /* Write new chain address to trigger state change. */ > + ioat->regs->chainaddr = ioat->desc_ring[(ioat->next_read - 1) & mask].next; > + /* Ensure channel control and status addr are correct. */ > + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | > + IOAT_CHANCTRL_ERR_COMPLETION_EN; > + ioat->regs->chancmp = ioat->status_addr; > + > + /* Allow HW time to move to the ARMED state. */ > + do { > + rte_pause(); > + retry++; > + } while (ioat->regs->chansts != IOAT_CHANSTS_ARMED && retry < 200); > + > + /* Exit as failure if device is still HALTED. */ > + if (ioat->regs->chansts != IOAT_CHANSTS_ARMED) > + return -1; > + > + /* Store next write as offset as recover will move HW and SW ring out of sync. */ > + ioat->offset = ioat->next_read; > + > + /* Prime status register with previous address. */ > + ioat->status = ioat->desc_ring[(ioat->next_read - 2) & mask].next; > + > + return 0; > +} > + > +/* Start a configured device. */ > +static int > +ioat_dev_start(struct rte_dmadev *dev) > +{ > + struct ioat_dmadev *ioat = dev->dev_private; > + > + if (ioat->qcfg.nb_desc == 0 || ioat->desc_ring == NULL) > + return -EBUSY; > + > + /* Inform hardware of where the descriptor ring is. */ > + ioat->regs->chainaddr = ioat->ring_addr; > + /* Inform hardware of where to write the status/completions. */ > + ioat->regs->chancmp = ioat->status_addr; > + > + /* Prime the status register to be set to the last element. */ > + ioat->status = ioat->ring_addr + ((ioat->qcfg.nb_desc - 1) * DESC_SZ); > + > + printf("IOAT.status: %s [%#lx]\n", > + chansts_readable[ioat->status & IOAT_CHANSTS_STATUS], > + ioat->status); > + > + if ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_HALTED) { > + IOAT_PMD_WARN("Device HALTED on start, attempting to recover\n"); > + if (__ioat_recover(ioat) != 0) { > + IOAT_PMD_ERR("Device couldn't be recovered"); > + return -1; > + } > + } > + > + return 0; > +} > + > +/* Stop a configured device. */ > +static int > +ioat_dev_stop(struct rte_dmadev *dev) > +{ > + struct ioat_dmadev *ioat = dev->dev_private; > + > + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; > + /* Allow the device time to suspend itself. */ > + rte_delay_ms(1); > + > + return 0; It be more beneficial to check if the device has actually suspended before returning. Similar to recover, a timeout could be set by which the device is expected to be suspended. If the device is still not suspended by then, return error. IMO this would be more useful to an application that always returning 0, since the device may still be active. With the above addressed, Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v2 05/10] dma/ioat: add start and stop functions 2021-09-07 10:11 ` Kevin Laatz @ 2021-09-07 10:35 ` Walsh, Conor 0 siblings, 0 replies; 132+ messages in thread From: Walsh, Conor @ 2021-09-07 10:35 UTC (permalink / raw) To: Laatz, Kevin, Richardson, Bruce, fengchengwen, jerinj; +Cc: dev <snip> > > +/* Stop a configured device. */ > > +static int > > +ioat_dev_stop(struct rte_dmadev *dev) > > +{ > > + struct ioat_dmadev *ioat = dev->dev_private; > > + > > + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; > > + /* Allow the device time to suspend itself. */ > > + rte_delay_ms(1); > > + > > + return 0; > > It be more beneficial to check if the device has actually suspended > before returning. Similar to recover, a timeout could be set by which > the device is expected to be suspended. If the device is still not > suspended by then, return error. > > IMO this would be more useful to an application that always returning 0, > since the device may still be active. > > > With the above addressed, > > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Thanks for the review Kevin, I agree this is a better solution than just delaying and returning 0. I will implement this and include it as part of v3. /Conor. ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v2 06/10] dma/ioat: add data path job submission functions 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices Conor Walsh ` (4 preceding siblings ...) 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 05/10] dma/ioat: add start and stop functions Conor Walsh @ 2021-09-03 11:17 ` Conor Walsh 2021-09-07 10:12 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 07/10] dma/ioat: add data path completion functions Conor Walsh ` (4 subsequent siblings) 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-03 11:17 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add data path functions for enqueuing and submitting operations to IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> --- doc/guides/dmadevs/ioat.rst | 54 ++++++++++++++++++++ drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index f7742642b5..16acfda3e3 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -89,3 +89,57 @@ The following code shows how the device is configured in ``test_dmadev.c``: Once configured, the device can then be made ready for use by calling the ``rte_dmadev_start()`` API. + +Performing Data Copies +~~~~~~~~~~~~~~~~~~~~~~~ + +To perform data copies using IOAT dmadev devices, the functions +``rte_dmadev_copy()`` and ``rte_dmadev_submit()`` should be used. Alternatively +``rte_dmadev_copy()`` can be called with the ``RTE_DMA_OP_FLAG_SUBMIT`` flag +set. + +The ``rte_dmadev_copy()`` function enqueues a single copy to the +device ring for copying at a later point. The parameters to the function +include the device ID of the desired device, the virtual DMA channel required +(always 0 for IOAT), the IOVA addresses of both the source and destination +buffers, the length of the data to be copied and any operation flags. The +function will return the index of the enqueued job which can be use to +track that operation. + +While the ``rte_dmadev_copy()`` function enqueues a copy operation on the device +ring, the copy will not actually be performed until after the application calls +the ``rte_dmadev_submit()`` function. This function informs the device hardware +of the elements enqueued on the ring, and the device will begin to process them. +It is expected that, for efficiency reasons, a burst of operations will be +enqueued to the device via multiple enqueue calls between calls to the +``rte_dmadev_submit()`` function. If desired you can pass the +``RTE_DMA_OP_FLAG_SUBMIT`` flag when calling ``rte_dmadev_copy()`` and this will +tell the device to perform the enqueued operation and any unperformed operations +before it. The ``RTE_DMA_OP_FLAG_SUBMIT`` flag can be passed instead of calling +the ``rte_dmadev_submit()`` function for example on the last enqueue of the burst. + +The following code from demonstrates how to enqueue a burst of copies to the +device and start the hardware processing of them: + +.. code-block:: C + + for (i = 0; i < BURST_SIZE; i++) { + if (rte_dmadev_copy(dev_id, vchan, rte_mbuf_data_iova(srcs[i]), + rte_mbuf_data_iova(dsts[i]), COPY_LEN, 0) < 0) { + PRINT_ERR("Error with rte_dmadev_copy for buffer %u\n", i); + return -1; + } + } + if (rte_dmadev_submit(dev_id, vchan) < 0) { + PRINT_ERR("Error with performing operations\n", i); + return -1; + } + +Filling an Area of Memory +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The driver also has support for the ``fill`` operation, where an area +of memory is overwritten, or filled, with a short pattern of data. +Fill operations can be performed in much the same was as copy operations +described above, just using the ``rte_dmadev_fill()`` function rather +than the ``rte_dmadev_copy()`` function. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 9f9feecd49..7ffe4e4fe9 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -5,6 +5,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> #include <rte_malloc.h> +#include <rte_prefetch.h> #include "ioat_internal.h" @@ -17,6 +18,12 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* IOAT operations. */ +enum rte_ioat_ops { + ioat_op_copy = 0, /* Standard DMA Operation */ + ioat_op_fill /* Block Fill */ +}; + /* Configure a device. */ static int ioat_dev_configure(struct rte_dmadev *dev __rte_unused, const struct rte_dmadev_conf *dev_conf) @@ -184,6 +191,87 @@ ioat_dev_close(struct rte_dmadev *dev) return 0; } +/* Trigger hardware to begin performing enqueued operations. */ +static inline void +__submit(struct ioat_dmadev *ioat) +{ + *ioat->doorbell = ioat->next_write - ioat->offset; + + ioat->last_write = ioat->next_write; +} + +/* External submit function wrapper. */ +static int +ioat_submit(struct rte_dmadev *dev, uint16_t qid __rte_unused) +{ + struct ioat_dmadev *ioat = (struct ioat_dmadev *)dev->dev_private; + + __submit(ioat); + + return 0; +} + +/* Write descriptor for enqueue. */ +static inline int +__write_desc(struct rte_dmadev *dev, uint32_t op, uint64_t src, phys_addr_t dst, + unsigned int length, uint64_t flags) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint16_t ret; + const unsigned short mask = ioat->qcfg.nb_desc - 1; + const unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + const unsigned short space = mask + read - write; + struct ioat_dma_hw_desc *desc; + + if (space == 0) + return -ENOSPC; + + ioat->next_write = write + 1; + write &= mask; + + desc = &ioat->desc_ring[write]; + desc->size = length; + desc->u.control_raw = (uint32_t)((op << IOAT_CMD_OP_SHIFT) | + (1 << IOAT_COMP_UPDATE_SHIFT)); + + /* In IOAT the fence ensures that all operations including the current one + * are completed before moving on, DMAdev assumes that the fence ensures + * all operations before the current one are completed before starting + * the current one, so in IOAT we set the fence for the previous descriptor. + */ + if (flags & RTE_DMA_OP_FLAG_FENCE) + ioat->desc_ring[(write - 1) & mask].u.control.fence = 1; + + desc->src_addr = src; + desc->dest_addr = dst; + + rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]); + + ret = (uint16_t)(ioat->next_write - 1); + + if (flags & RTE_DMA_OP_FLAG_SUBMIT) + __submit(ioat); + + return ret; +} + +/* Enqueue a fill operation onto the ioat device. */ +static int +ioat_enqueue_fill(struct rte_dmadev *dev, uint16_t qid __rte_unused, uint64_t pattern, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev, ioat_op_fill, pattern, dst, length, flags); +} + +/* Enqueue a copy operation onto the ioat device. */ +static int +ioat_enqueue_copy(struct rte_dmadev *dev, uint16_t qid __rte_unused, rte_iova_t src, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev, ioat_op_copy, src, dst, length, flags); +} + /* Dump DMA device info. */ static int ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) @@ -279,6 +367,10 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->copy = ioat_enqueue_copy; + dmadev->fill = ioat_enqueue_fill; + dmadev->submit = ioat_submit; + ioat = dmadev->data->dev_private; ioat->dmadev = dmadev; ioat->regs = dev->mem_resource[0].addr; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v2 06/10] dma/ioat: add data path job submission functions 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 06/10] dma/ioat: add data path job submission functions Conor Walsh @ 2021-09-07 10:12 ` Kevin Laatz 0 siblings, 0 replies; 132+ messages in thread From: Kevin Laatz @ 2021-09-07 10:12 UTC (permalink / raw) To: Conor Walsh, bruce.richardson, fengchengwen, jerinj; +Cc: dev On 03/09/2021 12:17, Conor Walsh wrote: > Add data path functions for enqueuing and submitting operations to > IOAT devices. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > --- > doc/guides/dmadevs/ioat.rst | 54 ++++++++++++++++++++ > drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ > 2 files changed, 146 insertions(+) > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v2 07/10] dma/ioat: add data path completion functions 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices Conor Walsh ` (5 preceding siblings ...) 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 06/10] dma/ioat: add data path job submission functions Conor Walsh @ 2021-09-03 11:17 ` Conor Walsh 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 08/10] dma/ioat: add statistics Conor Walsh ` (3 subsequent siblings) 10 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-03 11:17 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add the data path functions for gathering completed operations from IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 48 ++++++++++- drivers/dma/ioat/ioat_dmadev.c | 141 +++++++++++++++++++++++++++++++++ 2 files changed, 188 insertions(+), 1 deletion(-) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 16acfda3e3..9ea5708090 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -96,7 +96,9 @@ Performing Data Copies To perform data copies using IOAT dmadev devices, the functions ``rte_dmadev_copy()`` and ``rte_dmadev_submit()`` should be used. Alternatively ``rte_dmadev_copy()`` can be called with the ``RTE_DMA_OP_FLAG_SUBMIT`` flag -set. +set. Once copies have been completed, the completion will be reported back when +the application calls ``rte_dmadev_completed()`` or +``rte_dmadev_completed_status()``. The ``rte_dmadev_copy()`` function enqueues a single copy to the device ring for copying at a later point. The parameters to the function @@ -135,6 +137,50 @@ device and start the hardware processing of them: return -1; } +To retrieve information about completed copies, the API +``rte_dmadev_completed()`` or ``rte_dmadev_completed_status ()``should be used. +``rte_dmadev_completed()`` will return to the application how many operations +have successfully completed up to a supplied maximum, it can also pass back the +index of the last completed operation and if a completion within the requested +batch has failed. If ``rte_dmadev_completed()`` passes back ``has_error`` as +``true`` then ``rte_dmadev_completed_status()`` should be called to retrieve +the detailed status of each completion in that batch. Once an error has occurred +``rte_dmadev_completed()`` will not process any more completions until +``rte_dmadev_completed_status()`` has been called to get the error information. +The supplied ``status`` array will contain either ``RTE_DMA_STATUS_SUCCESSFUL`` +if the operation was successful or a DMA error code if the operation failed. +The status codes supported by IOAT are: + +* ``RTE_DMA_STATUS_SUCCESSFUL``: The operation was successful. +* ``RTE_DMA_STATUS_INVALID_SRC_ADDR``: The operation failed due to an invalid source address. +* ``RTE_DMA_STATUS_INVALID_DST_ADDR``: The operation failed due to an invalid destination address. +* ``RTE_DMA_STATUS_INVALID_LENGTH``: The operation failed due to an invalid descriptor length. +* ``RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR``: The device could not read the descriptor. +* ``RTE_DMA_STATUS_ERROR_UNKNOWN``: The operation failed due to an unspecified error. + +``rte_dmadev_completed_status()`` can be used on it's own instead of with +``rte_dmadev_completed()`` but calling ``rte_dmadev_completed_status()`` may be +slower than ``rte_dmadev_completed()``. + +The following code shows how to retrieve the number of successfully completed +copies within a burst and then using ``rte_dmadev_completed_status()`` to check +which operation failed and reset the device: + +.. code-block:: C + + enum rte_dma_status_code status[COMP_BURST_SZ]; + uint16_t count, idx, status_count; + bool error = 0; + + count = rte_dmadev_completed(dev_id, vchan, COMP_BURST_SZ, &idx, &error); + + if (error){ + status_count = rte_dmadev_completed_status(dev_id, vchan, COMP_BURST_SZ, &idx, status); + } + +After this snippet the application would use the status array to find which copy +failed and handle it appropriately. + Filling an Area of Memory ~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 7ffe4e4fe9..715beb3b7e 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -6,6 +6,7 @@ #include <rte_dmadev_pmd.h> #include <rte_malloc.h> #include <rte_prefetch.h> +#include <rte_errno.h> #include "ioat_internal.h" @@ -329,6 +330,144 @@ ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) return 0; } +/* Returns the index of the last completed operation. */ +static inline uint16_t +__get_last_completed(const struct ioat_dmadev *ioat, int *state) +{ + /* Status register contains the address of the completed operation */ + uint64_t status = ioat->status; + + /* lower 3 bits indicate "transfer status" : active, idle, halted. + * We can ignore bit 0. + */ + *state = status & IOAT_CHANSTS_STATUS; + + /* If we are just after recovering from an error the address returned by + * status will be 0, in this case we return the offset - 1 as the last + * completed. If not return the status value minus the chainaddr which + * gives us an offset into the ring. Right shifting by 6 (divide by 64) + * gives the index of the completion from the HW point of view and adding + * the offset translates the ring index from HW to SW point of view. + */ + if ((status & ~IOAT_CHANSTS_STATUS) == 0) + return ioat->offset - 1; + + return (status - ioat->ring_addr) >> 6; +} + +/* Translates IOAT ChanERRs to DMA error codes. */ +static inline enum rte_dma_status_code +__translate_status_ioat_to_dma(uint32_t chanerr) +{ + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_SRC_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_DST_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + return RTE_DMA_STATUS_INVALID_LENGTH; + else if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + return RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR; + else + return RTE_DMA_STATUS_ERROR_UNKNOWN; +} + +/* Returns details of operations that have been completed. */ +static uint16_t +ioat_completed(struct rte_dmadev *dev, uint16_t qid __rte_unused, const uint16_t max_ops, + uint16_t *last_idx, bool *has_error) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short last_completed, count; + int state, fails = 0; + + /* Do not do any work if there is an uncleared error. */ + if (ioat->failure != 0) { + *has_error = true; + *last_idx = ioat->next_read - 2; + return 0; + } + + last_completed = __get_last_completed(ioat, &state); + count = (last_completed + 1 - read) & mask; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || state != IOAT_CHANSTS_HALTED) { + ioat->next_read = read + count; + *last_idx = ioat->next_read - 1; + } else { + *has_error = true; + rte_errno = EIO; + ioat->failure = ioat->regs->chanerr; + ioat->next_read = read + count + 1; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + ioat_dev_dump(dev, stdout); + return 0; + } + __submit(ioat); + fails++; + *last_idx = ioat->next_read - 2; + } + + return count; +} + +/* Returns detailed status information about operations that have been completed. */ +static uint16_t +ioat_completed_status(struct rte_dmadev *dev, uint16_t qid __rte_unused, + uint16_t max_ops, uint16_t *last_idx, enum rte_dma_status_code *status) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short count, last_completed; + uint64_t fails = 0; + int state, i; + + last_completed = __get_last_completed(ioat, &state); + count = (last_completed + 1 - read) & mask; + + for (i = 0; i < RTE_MIN(count + 1, max_ops); i++) + status[i] = RTE_DMA_STATUS_SUCCESSFUL; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || state != IOAT_CHANSTS_HALTED) + ioat->next_read = read + count; + else { + rte_errno = EIO; + status[count] = __translate_status_ioat_to_dma(ioat->regs->chanerr); + count++; + ioat->next_read = read + count; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + ioat_dev_dump(dev, stdout); + return 0; + } + __submit(ioat); + fails++; + } + + if (ioat->failure > 0) { + status[0] = __translate_status_ioat_to_dma(ioat->failure); + count = RTE_MIN(count + 1, max_ops); + ioat->failure = 0; + } + + *last_idx = ioat->next_read - 1; + + return count; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -367,6 +506,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->completed = ioat_completed; + dmadev->completed_status = ioat_completed_status; dmadev->copy = ioat_enqueue_copy; dmadev->fill = ioat_enqueue_fill; dmadev->submit = ioat_submit; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v2 08/10] dma/ioat: add statistics 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices Conor Walsh ` (6 preceding siblings ...) 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 07/10] dma/ioat: add data path completion functions Conor Walsh @ 2021-09-03 11:17 ` Conor Walsh 2021-09-07 10:12 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 09/10] dma/ioat: add support for vchan idle function Conor Walsh ` (2 subsequent siblings) 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-03 11:17 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add statistic tracking for operations in IOAT. Signed-off-by: Conor Walsh <conor.walsh@intel.com> --- doc/guides/dmadevs/ioat.rst | 23 +++++++++++++++++++ drivers/dma/ioat/ioat_dmadev.c | 40 ++++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 9ea5708090..620b676718 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -189,3 +189,26 @@ of memory is overwritten, or filled, with a short pattern of data. Fill operations can be performed in much the same was as copy operations described above, just using the ``rte_dmadev_fill()`` function rather than the ``rte_dmadev_copy()`` function. + +Querying Device Statistics +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The statistics from the IOAT dmadev device can be got via the +``rte_dmadev_stats_get()`` API in the ``rte_dmadev`` library. + +The statistics returned for each IOAT device are: + +* ``submitted``: The number of operations submitted to the device. +* ``completed``: The number of operations successfully completed by the device. +* ``errors``: The number of operations that failed. + +The stats function can be used as follows: + +.. code-block:: C + + struct rte_dmadev_stats stats; + if (rte_dmadev_stats_get(dmadev_id, vchan, &stats) >= 0) { + printf("Total submitted ops: %lu", stats.submitted); + printf("Total completed ops: %lu", stats.completed); + printf("Total failed ops: %lu", stats.errors); + } diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 715beb3b7e..743fb2f558 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -198,6 +198,8 @@ __submit(struct ioat_dmadev *ioat) { *ioat->doorbell = ioat->next_write - ioat->offset; + ioat->stats.submitted += (uint16_t)(ioat->next_write - ioat->last_write); + ioat->last_write = ioat->next_write; } @@ -326,6 +328,10 @@ ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) fprintf(f, " Dest: %#lx\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); fprintf(f, " Next: %#lx\n", ioat->desc_ring[ioat->next_read & mask].next); fprintf(f, " }\n"); + fprintf(f, " Key Stats { submitted: %"PRIu64", comp: %"PRIu64", failed: %"PRIu64" }\n", + ioat->stats.submitted, + ioat->stats.completed, + ioat->stats.errors); return 0; } @@ -415,6 +421,9 @@ ioat_completed(struct rte_dmadev *dev, uint16_t qid __rte_unused, const uint16_t *last_idx = ioat->next_read - 2; } + ioat->stats.completed += count; + ioat->stats.errors += fails; + return count; } @@ -465,9 +474,38 @@ ioat_completed_status(struct rte_dmadev *dev, uint16_t qid __rte_unused, *last_idx = ioat->next_read - 1; + ioat->stats.completed += count; + ioat->stats.errors += fails; + return count; } +/* Retrieve the generic stats of a DMA device. */ +static int +ioat_stats_get(const struct rte_dmadev *dev, uint16_t vchan __rte_unused, + struct rte_dmadev_stats *rte_stats, uint32_t size) +{ + struct rte_dmadev_stats *stats = (&((struct ioat_dmadev *)dev->dev_private)->stats); + + if (size < sizeof(rte_stats)) + return -EINVAL; + if (rte_stats == NULL) + return -EINVAL; + + *rte_stats = *stats; + return 0; +} + +/* Reset the generic stat counters for the DMA device. */ +static int +ioat_stats_reset(struct rte_dmadev *dev, uint16_t vchan __rte_unused) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + memset(&ioat->stats, 0, sizeof(ioat->stats)); + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -479,6 +517,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_info_get = ioat_dev_info_get, .dev_start = ioat_dev_start, .dev_stop = ioat_dev_stop, + .stats_get = ioat_stats_get, + .stats_reset = ioat_stats_reset, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v2 08/10] dma/ioat: add statistics 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 08/10] dma/ioat: add statistics Conor Walsh @ 2021-09-07 10:12 ` Kevin Laatz 0 siblings, 0 replies; 132+ messages in thread From: Kevin Laatz @ 2021-09-07 10:12 UTC (permalink / raw) To: Conor Walsh, bruce.richardson, fengchengwen, jerinj; +Cc: dev On 03/09/2021 12:17, Conor Walsh wrote: > Add statistic tracking for operations in IOAT. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > --- > doc/guides/dmadevs/ioat.rst | 23 +++++++++++++++++++ > drivers/dma/ioat/ioat_dmadev.c | 40 ++++++++++++++++++++++++++++++++++ > 2 files changed, 63 insertions(+) > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v2 09/10] dma/ioat: add support for vchan idle function 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices Conor Walsh ` (7 preceding siblings ...) 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 08/10] dma/ioat: add statistics Conor Walsh @ 2021-09-03 11:17 ` Conor Walsh 2021-09-07 10:12 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 10/10] devbind: move ioat device ID for ICX to dmadev category Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices Conor Walsh 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-03 11:17 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add support for the rte_dmadev_vchan_idle API call. Signed-off-by: Conor Walsh <conor.walsh@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 743fb2f558..a09e60b27e 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -506,6 +506,19 @@ ioat_stats_reset(struct rte_dmadev *dev, uint16_t vchan __rte_unused) return 0; } +/* Check if the IOAT device is idle. */ +static int +ioat_vchan_idle(const struct rte_dmadev *dev, uint16_t vchan __rte_unused) +{ + int state = 0; + const struct ioat_dmadev *ioat = dev->dev_private; + const uint16_t mask = ioat->qcfg.nb_desc - 1; + const uint16_t last = __get_last_completed(ioat, &state); + + return (last == ((ioat->next_write - 1) & mask) || state == IOAT_CHANSTS_HALTED + || state == IOAT_CHANSTS_SUSPENDED); +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -519,6 +532,7 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_stop = ioat_dev_stop, .stats_get = ioat_stats_get, .stats_reset = ioat_stats_reset, + .vchan_idle = ioat_vchan_idle, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v2 09/10] dma/ioat: add support for vchan idle function 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 09/10] dma/ioat: add support for vchan idle function Conor Walsh @ 2021-09-07 10:12 ` Kevin Laatz 0 siblings, 0 replies; 132+ messages in thread From: Kevin Laatz @ 2021-09-07 10:12 UTC (permalink / raw) To: Conor Walsh, bruce.richardson, fengchengwen, jerinj; +Cc: dev On 03/09/2021 12:17, Conor Walsh wrote: > Add support for the rte_dmadev_vchan_idle API call. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > --- > drivers/dma/ioat/ioat_dmadev.c | 14 ++++++++++++++ > 1 file changed, 14 insertions(+) > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v2 10/10] devbind: move ioat device ID for ICX to dmadev category 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices Conor Walsh ` (8 preceding siblings ...) 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 09/10] dma/ioat: add support for vchan idle function Conor Walsh @ 2021-09-03 11:17 ` Conor Walsh 2021-09-07 10:10 ` Kevin Laatz 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices Conor Walsh 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-03 11:17 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Move Intel IOAT devices on Ice Lake systems from Misc to DMA devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> --- usertools/dpdk-devbind.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index 98b698ccc0..afebc8cb62 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -69,14 +69,13 @@ network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] -dma_devices = [intel_idxd_spr] +dma_devices = [intel_idxd_spr, intel_ioat_icx] eventdev_devices = [cavium_sso, cavium_tim, intel_dlb, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] regex_devices = [octeontx2_ree] misc_devices = [cnxk_bphy, cnxk_bphy_cgx, intel_ioat_bdw, intel_ioat_skx, - intel_ioat_icx, intel_ntb_skx, intel_ntb_icx, - octeontx2_dma] + intel_ntb_skx, intel_ntb_icx, octeontx2_dma] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v2 10/10] devbind: move ioat device ID for ICX to dmadev category 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 10/10] devbind: move ioat device ID for ICX to dmadev category Conor Walsh @ 2021-09-07 10:10 ` Kevin Laatz 0 siblings, 0 replies; 132+ messages in thread From: Kevin Laatz @ 2021-09-07 10:10 UTC (permalink / raw) To: Conor Walsh, bruce.richardson, fengchengwen, jerinj; +Cc: dev On 03/09/2021 12:17, Conor Walsh wrote: > Move Intel IOAT devices on Ice Lake systems from Misc to DMA devices. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > --- > usertools/dpdk-devbind.py | 5 ++--- > 1 file changed, 2 insertions(+), 3 deletions(-) > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices Conor Walsh ` (9 preceding siblings ...) 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 10/10] devbind: move ioat device ID for ICX to dmadev category Conor Walsh @ 2021-09-08 10:39 ` Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 01/11] dma/ioat: add device probe and removal functions Conor Walsh ` (10 more replies) 10 siblings, 11 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-08 10:39 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh This patchset adds a dmadev driver and associated documentation to support Intel QuickData Technology devices, part of the Intel I/O Acceleration Technology (Intel I/OAT). This driver is intended to ultimately replace the current IOAT part of the IOAT rawdev driver. This patchset passes all the driver tests added in the dmadev test suite. NOTE: This patchset has several dependencies: - v21 of the dmadev set [1] - v3 of the dmadev test suite [2] - v3 of the IDXD driver [3] [1] http://patches.dpdk.org/project/dpdk/list/?series=18738 [2] http://patches.dpdk.org/project/dpdk/list/?series=18744 [3] http://patches.dpdk.org/project/dpdk/list/?series=18762 --- v3: - Added burst capacity function. - Stop function now waits for suspend rather than just using a sleep. - Changed from vchan idle to vchan status function. - Other minor changes to update from dmadev v19 to v21. v2: - Rebased on the above patchsets. - Added support for the vchan idle function. - Stop function now suspends IOAT channel to allow for reconfig. - dmadev_autotest can now be run multiple times using the IOAT driver without errors. - Added devbind updates for DMA devices - Removed some logically dead code found by coverity in the create function. Conor Walsh (11): dma/ioat: add device probe and removal functions dma/ioat: create dmadev instances on PCI probe dma/ioat: add datapath structures dma/ioat: add configuration functions dma/ioat: add start and stop functions dma/ioat: add data path job submission functions dma/ioat: add data path completion functions dma/ioat: add statistics dma/ioat: add support for vchan status function dma/ioat: add burst capacity function devbind: move ioat device ID for ICX to dmadev category MAINTAINERS | 6 + doc/guides/dmadevs/index.rst | 1 + doc/guides/dmadevs/ioat.rst | 214 ++++++++ doc/guides/rel_notes/release_21_11.rst | 7 +- drivers/dma/ioat/ioat_dmadev.c | 728 +++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 294 ++++++++++ drivers/dma/ioat/ioat_internal.h | 44 ++ drivers/dma/ioat/meson.build | 7 + drivers/dma/ioat/version.map | 3 + drivers/dma/meson.build | 1 + usertools/dpdk-devbind.py | 5 +- 11 files changed, 1304 insertions(+), 6 deletions(-) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v3 01/11] dma/ioat: add device probe and removal functions 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices Conor Walsh @ 2021-09-08 10:39 ` Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 02/11] dma/ioat: create dmadev instances on PCI probe Conor Walsh ` (9 subsequent siblings) 10 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-08 10:39 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add the basic device probe/remove skeleton code and initial documentation for new IOAT DMA driver. Maintainers update is also included in this patch. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- MAINTAINERS | 6 +++ doc/guides/dmadevs/index.rst | 1 + doc/guides/dmadevs/ioat.rst | 64 ++++++++++++++++++++++++ doc/guides/rel_notes/release_21_11.rst | 7 +-- drivers/dma/ioat/ioat_dmadev.c | 69 ++++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 35 +++++++++++++ drivers/dma/ioat/ioat_internal.h | 20 ++++++++ drivers/dma/ioat/meson.build | 7 +++ drivers/dma/ioat/version.map | 3 ++ drivers/dma/meson.build | 1 + 10 files changed, 210 insertions(+), 3 deletions(-) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map diff --git a/MAINTAINERS b/MAINTAINERS index b4c614a229..00b319c811 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1208,6 +1208,12 @@ M: Kevin Laatz <kevin.laatz@intel.com> F: drivers/dma/idxd/ F: doc/guides/dmadevs/idxd.rst +Intel IOAT - EXPERIMENTAL +M: Bruce Richardson <bruce.richardson@intel.com> +M: Conor Walsh <conor.walsh@intel.com> +F: drivers/dma/ioat/ +F: doc/guides/dmadevs/ioat.rst + RegEx Drivers ------------- diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst index b30004fd65..0b01493c5a 100644 --- a/doc/guides/dmadevs/index.rst +++ b/doc/guides/dmadevs/index.rst @@ -12,3 +12,4 @@ application through DMAdev API. :numbered: idxd + ioat diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst new file mode 100644 index 0000000000..45a2e65d70 --- /dev/null +++ b/doc/guides/dmadevs/ioat.rst @@ -0,0 +1,64 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2021 Intel Corporation. + +.. include:: <isonum.txt> + +IOAT DMA Device Driver +======================= + +The ``ioat`` dmadev driver provides a poll-mode driver (PMD) for Intel\ +|reg| QuickData Technology which is part of part of Intel\ |reg| I/O +Acceleration Technology (`Intel I/OAT +<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_). +This PMD, when used on supported hardware, allows data copies, for example, +cloning packet data, to be accelerated by IOAT hardware rather than having to +be done by software, freeing up CPU cycles for other tasks. + +Hardware Requirements +---------------------- + +The ``dpdk-devbind.py`` script, included with DPDK, can be used to show the +presence of supported hardware. Running ``dpdk-devbind.py --status-dev dma`` +will show all the DMA devices on the system, IOAT devices are included in this +list. For Intel\ |reg| IOAT devices, the hardware will often be listed as +"Crystal Beach DMA", or "CBDMA" or on some newer systems '0b00' due to the +absence of pci-id database entries for them at this point. + +Compilation +------------ + +For builds using ``meson`` and ``ninja``, the driver will be built when the +target platform is x86-based. No additional compilation steps are necessary. + +Device Setup +------------- + +Intel\ |reg| IOAT devices will need to be bound to a suitable DPDK-supported +user-space IO driver such as ``vfio-pci`` in order to be used by DPDK. + +The ``dpdk-devbind.py`` script can be used to view the state of the devices using:: + + $ dpdk-devbind.py --status-dev dma + +The ``dpdk-devbind.py`` script can also be used to bind devices to a suitable driver. +For example:: + + $ dpdk-devbind.py -b vfio-pci 00:01.0 00:01.1 + +Device Probing and Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For devices bound to a suitable DPDK-supported driver (``vfio-pci``), the HW +devices will be found as part of the device scan done at application +initialization time without the need to pass parameters to the application. + +If the application does not require all the devices available an allowlist can +be used in the same way that other DPDK devices use them. + +For example:: + + $ dpdk-test -a <b:d:f> + +Once probed successfully, the device will appear as a ``dmadev``, that is a +"DMA device type" inside DPDK, and can be accessed using APIs from the +``rte_dmadev`` library. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 8526646b13..fcc9cbd841 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -67,10 +67,11 @@ New Features The dmadev library provides a DMA device framework for management and provision of hardware and software DMA devices. -* **Added IDXD dmadev driver implementation.** +* **Added Intel dmadev driver implementations.** - The IDXD dmadev driver provide device drivers for the Intel DSA devices. - This device driver can be used through the generic dmadev API. + The IDXD and IOAT dmadev drivers provide device drivers for Intel DSA + and IOAT devices. These device drivers can be used through the generic + dmadev API. Removed Items diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c new file mode 100644 index 0000000000..f3491d45b1 --- /dev/null +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#include <rte_bus_pci.h> +#include <rte_dmadev_pmd.h> + +#include "ioat_internal.h" + +static struct rte_pci_driver ioat_pmd_drv; + +RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); + +#define IOAT_PMD_NAME dmadev_ioat +#define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) + +/* Probe DMA device. */ +static int +ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); + + dev->device.driver = &drv->driver; + return 0; +} + +/* Remove DMA device. */ +static int +ioat_dmadev_remove(struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + + IOAT_PMD_INFO("Closing %s on NUMA node %d", + name, dev->device.numa_node); + + return 0; +} + +static const struct rte_pci_id pci_id_ioat_map[] = { + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_SKX) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX0) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX1) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX2) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX3) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX4) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX5) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX6) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX7) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXE) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXF) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_ICX) }, + { .vendor_id = 0, /* sentinel */ }, +}; + +static struct rte_pci_driver ioat_pmd_drv = { + .id_table = pci_id_ioat_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC, + .probe = ioat_dmadev_probe, + .remove = ioat_dmadev_remove, +}; + +RTE_PMD_REGISTER_PCI(IOAT_PMD_NAME, ioat_pmd_drv); +RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_NAME, pci_id_ioat_map); +RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_NAME, "* igb_uio | uio_pci_generic | vfio-pci"); diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h new file mode 100644 index 0000000000..eeabba41ef --- /dev/null +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#ifndef IOAT_HW_DEFS_H +#define IOAT_HW_DEFS_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> + +#define IOAT_VER_3_0 0x30 +#define IOAT_VER_3_3 0x33 + +#define IOAT_VENDOR_ID 0x8086 +#define IOAT_DEVICE_ID_SKX 0x2021 +#define IOAT_DEVICE_ID_BDX0 0x6f20 +#define IOAT_DEVICE_ID_BDX1 0x6f21 +#define IOAT_DEVICE_ID_BDX2 0x6f22 +#define IOAT_DEVICE_ID_BDX3 0x6f23 +#define IOAT_DEVICE_ID_BDX4 0x6f24 +#define IOAT_DEVICE_ID_BDX5 0x6f25 +#define IOAT_DEVICE_ID_BDX6 0x6f26 +#define IOAT_DEVICE_ID_BDX7 0x6f27 +#define IOAT_DEVICE_ID_BDXE 0x6f2E +#define IOAT_DEVICE_ID_BDXF 0x6f2F +#define IOAT_DEVICE_ID_ICX 0x0b00 + +#ifdef __cplusplus +} +#endif + +#endif /* IOAT_HW_DEFS_H */ diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h new file mode 100644 index 0000000000..f1ec12a919 --- /dev/null +++ b/drivers/dma/ioat/ioat_internal.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021 Intel Corporation + */ + +#ifndef _IOAT_INTERNAL_H_ +#define _IOAT_INTERNAL_H_ + +#include "ioat_hw_defs.h" + +extern int ioat_pmd_logtype; + +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + ioat_pmd_logtype, "IOAT: %s(): " fmt "\n", __func__, ##args) + +#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) +#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) +#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) +#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) + +#endif /* _IOAT_INTERNAL_H_ */ diff --git a/drivers/dma/ioat/meson.build b/drivers/dma/ioat/meson.build new file mode 100644 index 0000000000..d67fac96fb --- /dev/null +++ b/drivers/dma/ioat/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021 Intel Corporation + +build = dpdk_conf.has('RTE_ARCH_X86') +reason = 'only supported on x86' +sources = files('ioat_dmadev.c') +deps += ['bus_pci', 'dmadev'] diff --git a/drivers/dma/ioat/version.map b/drivers/dma/ioat/version.map new file mode 100644 index 0000000000..c2e0723b4c --- /dev/null +++ b/drivers/dma/ioat/version.map @@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +}; diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build index 0b01b6a8ab..f8f5154a40 100644 --- a/drivers/dma/meson.build +++ b/drivers/dma/meson.build @@ -7,6 +7,7 @@ endif drivers = [ 'idxd', + 'ioat', 'skeleton', ] std_deps = ['dmadev'] -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v3 02/11] dma/ioat: create dmadev instances on PCI probe 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 01/11] dma/ioat: add device probe and removal functions Conor Walsh @ 2021-09-08 10:39 ` Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 03/11] dma/ioat: add datapath structures Conor Walsh ` (8 subsequent siblings) 10 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-08 10:39 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh When a suitable device is found during the PCI probe, create a dmadev instance for each channel. Internal structures and HW definitions required for device creation are also included. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 108 ++++++++++++++++++++++++++++++- drivers/dma/ioat/ioat_hw_defs.h | 45 +++++++++++++ drivers/dma/ioat/ioat_internal.h | 24 +++++++ 3 files changed, 175 insertions(+), 2 deletions(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index f3491d45b1..aa03dd1cd2 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -4,6 +4,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> +#include <rte_malloc.h> #include "ioat_internal.h" @@ -14,6 +15,109 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Create a DMA device. */ +static int +ioat_dmadev_create(const char *name, struct rte_pci_device *dev) +{ + static const struct rte_dmadev_ops ioat_dmadev_ops = { }; + + struct rte_dmadev *dmadev = NULL; + struct ioat_dmadev *ioat = NULL; + int retry = 0; + + if (!name) { + IOAT_PMD_ERR("Invalid name of the device!"); + return -EINVAL; + } + + /* Allocate device structure. */ + dmadev = rte_dmadev_pmd_allocate(name); + if (dmadev == NULL) { + IOAT_PMD_ERR("Unable to allocate dma device"); + return -ENOMEM; + } + + dmadev->device = &dev->device; + + dmadev->data->dev_private = rte_malloc_socket(NULL, sizeof(*ioat), + 0, dmadev->device->numa_node); + dmadev->dev_private = dmadev->data->dev_private; + + dmadev->dev_ops = &ioat_dmadev_ops; + + ioat = dmadev->data->dev_private; + ioat->dmadev = dmadev; + ioat->regs = dev->mem_resource[0].addr; + ioat->doorbell = &ioat->regs->dmacount; + ioat->qcfg.nb_desc = 0; + ioat->desc_ring = NULL; + + /* Do device initialization - reset and set error behaviour. */ + if (ioat->regs->chancnt != 1) + IOAT_PMD_WARN("%s: Channel count == %d\n", __func__, + ioat->regs->chancnt); + + /* Locked by someone else. */ + if (ioat->regs->chanctrl & IOAT_CHANCTRL_CHANNEL_IN_USE) { + IOAT_PMD_WARN("%s: Channel appears locked\n", __func__); + ioat->regs->chanctrl = 0; + } + + /* clear any previous errors */ + if (ioat->regs->chanerr != 0) { + uint32_t val = ioat->regs->chanerr; + ioat->regs->chanerr = val; + } + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + rte_delay_ms(1); + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + rte_delay_ms(1); + while (ioat->regs->chancmd & IOAT_CHANCMD_RESET) { + ioat->regs->chainaddr = 0; + rte_delay_ms(1); + if (++retry >= 200) { + IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=%#"PRIx8 + ", CHANSTS=%#"PRIx64", CHANERR=%#"PRIx32"\n", + __func__, + ioat->regs->chancmd, + ioat->regs->chansts, + ioat->regs->chanerr); + return -EIO; + } + } + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + + return 0; + +} + +/* Destroy a DMA device. */ +static int +ioat_dmadev_destroy(const char *name) +{ + struct rte_dmadev *dev; + int ret; + + if (!name) { + IOAT_PMD_ERR("Invalid device name"); + return -EINVAL; + } + + dev = rte_dmadev_get_device_by_name(name); + if (dev == NULL) { + IOAT_PMD_ERR("Invalid device name (%s)", name); + return -EINVAL; + } + + ret = rte_dmadev_pmd_release(dev); + if (ret) + IOAT_PMD_DEBUG("Device cleanup failed"); + + return 0; +} + /* Probe DMA device. */ static int ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) @@ -24,7 +128,7 @@ ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); dev->device.driver = &drv->driver; - return 0; + return ioat_dmadev_create(name, dev); } /* Remove DMA device. */ @@ -38,7 +142,7 @@ ioat_dmadev_remove(struct rte_pci_device *dev) IOAT_PMD_INFO("Closing %s on NUMA node %d", name, dev->device.numa_node); - return 0; + return ioat_dmadev_destroy(name); } static const struct rte_pci_id pci_id_ioat_map[] = { diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index eeabba41ef..73bdf548b3 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -11,6 +11,8 @@ extern "C" { #include <stdint.h> +#define IOAT_PCI_CHANERR_INT_OFFSET 0x180 + #define IOAT_VER_3_0 0x30 #define IOAT_VER_3_3 0x33 @@ -28,6 +30,49 @@ extern "C" { #define IOAT_DEVICE_ID_BDXF 0x6f2F #define IOAT_DEVICE_ID_ICX 0x0b00 +#define IOAT_COMP_UPDATE_SHIFT 3 +#define IOAT_CMD_OP_SHIFT 24 + +/* DMA Channel Registers */ +#define IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK 0xF000 +#define IOAT_CHANCTRL_COMPL_DCA_EN 0x0200 +#define IOAT_CHANCTRL_CHANNEL_IN_USE 0x0100 +#define IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL 0x0020 +#define IOAT_CHANCTRL_ERR_INT_EN 0x0010 +#define IOAT_CHANCTRL_ANY_ERR_ABORT_EN 0x0008 +#define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 +#define IOAT_CHANCTRL_INT_REARM 0x0001 + +struct ioat_registers { + uint8_t chancnt; + uint8_t xfercap; + uint8_t genctrl; + uint8_t intrctrl; + uint32_t attnstatus; + uint8_t cbver; /* 0x08 */ + uint8_t reserved4[0x3]; /* 0x09 */ + uint16_t intrdelay; /* 0x0C */ + uint16_t cs_status; /* 0x0E */ + uint32_t dmacapability; /* 0x10 */ + uint8_t reserved5[0x6C]; /* 0x14 */ + uint16_t chanctrl; /* 0x80 */ + uint8_t reserved6[0x2]; /* 0x82 */ + uint8_t chancmd; /* 0x84 */ + uint8_t reserved3[1]; /* 0x85 */ + uint16_t dmacount; /* 0x86 */ + uint64_t chansts; /* 0x88 */ + uint64_t chainaddr; /* 0x90 */ + uint64_t chancmp; /* 0x98 */ + uint8_t reserved2[0x8]; /* 0xA0 */ + uint32_t chanerr; /* 0xA8 */ + uint32_t chanerrmask; /* 0xAC */ +} __rte_packed; + +#define IOAT_CHANCMD_RESET 0x20 +#define IOAT_CHANCMD_SUSPEND 0x04 + +#define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ + #ifdef __cplusplus } #endif diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h index f1ec12a919..133fceae86 100644 --- a/drivers/dma/ioat/ioat_internal.h +++ b/drivers/dma/ioat/ioat_internal.h @@ -7,6 +7,30 @@ #include "ioat_hw_defs.h" +struct ioat_dmadev { + struct rte_dmadev *dmadev; + struct rte_dmadev_vchan_conf qcfg; + struct rte_dmadev_stats stats; + + volatile uint16_t *doorbell __rte_cache_aligned; + phys_addr_t status_addr; + phys_addr_t ring_addr; + + struct ioat_dma_hw_desc *desc_ring; + + unsigned short next_read; + unsigned short next_write; + unsigned short last_write; /* Used to compute submitted count. */ + unsigned short offset; /* Used after a device recovery when counts -> 0. */ + unsigned int failure; /* Used to store chanerr for error handling. */ + + /* To report completions, the device will write status back here. */ + volatile uint64_t status __rte_cache_aligned; + + /* Pointer to the register bar. */ + volatile struct ioat_registers *regs; +}; + extern int ioat_pmd_logtype; #define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v3 03/11] dma/ioat: add datapath structures 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 01/11] dma/ioat: add device probe and removal functions Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 02/11] dma/ioat: create dmadev instances on PCI probe Conor Walsh @ 2021-09-08 10:39 ` Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 04/11] dma/ioat: add configuration functions Conor Walsh ` (7 subsequent siblings) 10 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-08 10:39 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add data structures required for the data path of IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 61 ++++++++- drivers/dma/ioat/ioat_hw_defs.h | 214 ++++++++++++++++++++++++++++++++ 2 files changed, 274 insertions(+), 1 deletion(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index aa03dd1cd2..d65881d852 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -15,11 +15,70 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Dump DMA device info. */ +static int +ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint64_t chansts_masked = ioat->regs->chansts & IOAT_CHANSTS_STATUS; + uint32_t chanerr = ioat->regs->chanerr; + uint64_t mask = (ioat->qcfg.nb_desc - 1); + fprintf(f, "========= IOAT =========\n"); + fprintf(f, " Channel status: %s [%#lx]\n", + chansts_readable[chansts_masked], chansts_masked); + fprintf(f, " ChainADDR: %#lx\n", ioat->regs->chainaddr); + if (chanerr == 0) { + fprintf(f, " No Channel Errors\n"); + } else { + fprintf(f, " ChanERR: %#x\n", chanerr); + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + fprintf(f, " Invalid Source Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + fprintf(f, " Invalid Destination Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + fprintf(f, " Invalid Descriptor Length\n"); + if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + fprintf(f, " Descriptor Read Error\n"); + if ((chanerr & ~(IOAT_CHANERR_INVALID_SRC_ADDR_MASK | + IOAT_CHANERR_INVALID_DST_ADDR_MASK | + IOAT_CHANERR_INVALID_LENGTH_MASK | + IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK)) != 0) + fprintf(f, " Unknown Error(s)\n"); + } + fprintf(f, "== Private Data ==\n"); + fprintf(f, " Config: { ring_size: %u }\n", ioat->qcfg.nb_desc); + fprintf(f, " Status: %#lx\n", ioat->status); + fprintf(f, " Status IOVA: %#lx\n", ioat->status_addr); + fprintf(f, " Status ADDR: %p\n", &ioat->status); + fprintf(f, " Ring IOVA: %#lx\n", ioat->ring_addr); + fprintf(f, " Ring ADDR: %#lx\n", ioat->desc_ring[0].next-64); + fprintf(f, " Next write: %u\n", ioat->next_write); + fprintf(f, " Next read: %u\n", ioat->next_read); + struct ioat_dma_hw_desc *desc_ring = &ioat->desc_ring[(ioat->next_write - 1) & mask]; + fprintf(f, " Last Descriptor Written {\n"); + fprintf(f, " Size: %u\n", desc_ring->size); + fprintf(f, " Control: %#x\n", desc_ring->u.control_raw); + fprintf(f, " Src: %#lx\n", desc_ring->src_addr); + fprintf(f, " Dest: %#lx\n", desc_ring->dest_addr); + fprintf(f, " Next: %#lx\n", desc_ring->next); + fprintf(f, " }\n"); + fprintf(f, " Next Descriptor {\n"); + fprintf(f, " Size: %u\n", ioat->desc_ring[ioat->next_read & mask].size); + fprintf(f, " Src: %#lx\n", ioat->desc_ring[ioat->next_read & mask].src_addr); + fprintf(f, " Dest: %#lx\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); + fprintf(f, " Next: %#lx\n", ioat->desc_ring[ioat->next_read & mask].next); + fprintf(f, " }\n"); + + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { - static const struct rte_dmadev_ops ioat_dmadev_ops = { }; + static const struct rte_dmadev_ops ioat_dmadev_ops = { + .dev_dump = ioat_dev_dump, + }; struct rte_dmadev *dmadev = NULL; struct ioat_dmadev *ioat = NULL; diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index 73bdf548b3..1f47209e60 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -43,6 +43,14 @@ extern "C" { #define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 #define IOAT_CHANCTRL_INT_REARM 0x0001 +/* DMA Channel Capabilities */ +#define IOAT_DMACAP_PB (1 << 0) +#define IOAT_DMACAP_DCA (1 << 4) +#define IOAT_DMACAP_BFILL (1 << 6) +#define IOAT_DMACAP_XOR (1 << 8) +#define IOAT_DMACAP_PQ (1 << 9) +#define IOAT_DMACAP_DMA_DIF (1 << 10) + struct ioat_registers { uint8_t chancnt; uint8_t xfercap; @@ -71,8 +79,214 @@ struct ioat_registers { #define IOAT_CHANCMD_RESET 0x20 #define IOAT_CHANCMD_SUSPEND 0x04 +#define IOAT_CHANSTS_STATUS 0x7ULL +#define IOAT_CHANSTS_ACTIVE 0x0 +#define IOAT_CHANSTS_IDLE 0x1 +#define IOAT_CHANSTS_SUSPENDED 0x2 +#define IOAT_CHANSTS_HALTED 0x3 +#define IOAT_CHANSTS_ARMED 0x4 + +#define IOAT_CHANERR_INVALID_SRC_ADDR_MASK (1 << 0) +#define IOAT_CHANERR_INVALID_DST_ADDR_MASK (1 << 1) +#define IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK (1 << 8) +#define IOAT_CHANERR_INVALID_LENGTH_MASK (1 << 10) + +const char *chansts_readable[] = { + "ACTIVE", /* 0x0 */ + "IDLE", /* 0x1 */ + "SUSPENDED", /* 0x2 */ + "HALTED", /* 0x3 */ + "ARMED" /* 0x4 */ +}; + +#define IOAT_CHANSTS_UNAFFILIATED_ERROR 0x8ULL +#define IOAT_CHANSTS_SOFT_ERROR 0x10ULL + +#define IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK (~0x3FULL) + #define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ +struct ioat_dma_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t null: 1; + uint32_t src_page_break: 1; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_COPY 0x00 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t reserved2; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_fill_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t reserved: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t reserved2: 2; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t reserved3: 15; +#define IOAT_OP_FILL 0x01 + uint32_t op: 8; + } control; + } u; + uint64_t src_data; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t next_dest_addr; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_xor_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_XOR 0x87 +#define IOAT_OP_XOR_VAL 0x88 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint64_t src_addr4; + uint64_t src_addr5; +}; + +struct ioat_xor_ext_hw_desc { + uint64_t src_addr6; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t next; + uint64_t reserved[4]; +}; + +struct ioat_pq_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 11; +#define IOAT_OP_PQ 0x89 +#define IOAT_OP_PQ_VAL 0x8a + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint8_t coef[8]; + uint64_t q_addr; +}; + +struct ioat_pq_ext_hw_desc { + uint64_t src_addr4; + uint64_t src_addr5; + uint64_t src_addr6; + uint64_t next; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t reserved[2]; +}; + +struct ioat_pq_update_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_cnt: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 3; + uint32_t coef: 8; +#define IOAT_OP_PQ_UP 0x8b + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t p_src; + uint64_t q_src; + uint64_t q_addr; +}; + +union ioat_hw_desc { + struct ioat_dma_hw_desc dma; + struct ioat_fill_hw_desc fill; + struct ioat_xor_hw_desc xor_desc; + struct ioat_xor_ext_hw_desc xor_ext; + struct ioat_pq_hw_desc pq; + struct ioat_pq_ext_hw_desc pq_ext; + struct ioat_pq_update_hw_desc pq_update; +}; + +#define GENSTS_DEV_STATE_MASK 0x03 +#define CMDSTATUS_ACTIVE_SHIFT 31 +#define CMDSTATUS_ACTIVE_MASK (1 << 31) +#define CMDSTATUS_ERR_MASK 0xFF + #ifdef __cplusplus } #endif -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v3 04/11] dma/ioat: add configuration functions 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (2 preceding siblings ...) 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 03/11] dma/ioat: add datapath structures Conor Walsh @ 2021-09-08 10:39 ` Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 05/11] dma/ioat: add start and stop functions Conor Walsh ` (6 subsequent siblings) 10 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-08 10:39 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add functions for device configuration. The info_get and close functions are included here also. info_get can be useful for checking successful configuration and close is used by the dmadev api when releasing a configured device. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 24 +++++++++ drivers/dma/ioat/ioat_dmadev.c | 96 ++++++++++++++++++++++++++++++++++ 2 files changed, 120 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 45a2e65d70..b6d88fe966 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -62,3 +62,27 @@ For example:: Once probed successfully, the device will appear as a ``dmadev``, that is a "DMA device type" inside DPDK, and can be accessed using APIs from the ``rte_dmadev`` library. + +Device Configuration +~~~~~~~~~~~~~~~~~~~~~ + +Configuring an IOAT dmadev device is done using the ``rte_dmadev_configure()`` +and ``rte_dmadev_vchan_setup()`` APIs. ``rte_dmadev_configure()`` uses the structure +``rte_dmadev_conf`` to configure an IOAT device, within this struct the number +of virtual DMA channels for the device is set and silent mode can be enabled. +Each IOAT device can only use 1 virtual DMA channel and silent mode is not +supported so these will always be set to ``1`` and ``false``. +``rte_dmadev_vchan_setup()`` uses the structure ``rte_dmadev_vchan_conf`` to setup +a virtual DMA channel within this struct the transfer direction and ring size +are set. Generally for an onboard Intel\ |reg| IOAT device the transfer direction +will be set to ``RTE_DMA_DIR_MEM_TO_MEM`` to copy from memory to memory (more info +available in the dmadev API guide). The ring size must be a power of two, between +64 and 4096. + +The following code shows how the device is configured in ``test_dmadev.c``: + +.. literalinclude:: ../../../app/test/test_dmadev.c + :language: c + :start-after: Setup of the dmadev device. 8< + :end-before: >8 End of setup of the dmadev device. + :dedent: 1 diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index d65881d852..e23cbb2068 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -12,9 +12,101 @@ static struct rte_pci_driver ioat_pmd_drv; RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); +#define DESC_SZ sizeof(struct ioat_dma_hw_desc) + #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Configure a device. */ +static int +ioat_dev_configure(struct rte_dmadev *dev __rte_unused, const struct rte_dmadev_conf *dev_conf, + uint32_t conf_sz) +{ + if (sizeof(struct rte_dmadev_conf) != conf_sz) + return -EINVAL; + + if (dev_conf->nb_vchans != 1) + return -EINVAL; + + return 0; +} + +/* Setup a virtual channel for IOAT, only 1 vchan is supported. */ +static int +ioat_vchan_setup(struct rte_dmadev *dev, uint16_t vchan __rte_unused, + const struct rte_dmadev_vchan_conf *qconf, uint32_t qconf_sz) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint16_t max_desc = qconf->nb_desc; + int i; + + if (sizeof(struct rte_dmadev_vchan_conf) != qconf_sz) + return -EINVAL; + + ioat->qcfg = *qconf; + + if (!rte_is_power_of_2(max_desc)) { + max_desc = rte_align32pow2(max_desc); + IOAT_PMD_DEBUG("DMA dev %u using %u descriptors", dev->data->dev_id, max_desc); + ioat->qcfg.nb_desc = max_desc; + } + + /* In case we are reconfiguring a device, free any existing memory. */ + rte_free(ioat->desc_ring); + + ioat->desc_ring = rte_zmalloc(NULL, sizeof(*ioat->desc_ring) * max_desc, 0); + if (ioat->desc_ring == NULL) + return -ENOMEM; + + ioat->ring_addr = rte_mem_virt2iova(ioat->desc_ring); + + ioat->status_addr = rte_mem_virt2iova(ioat) + offsetof(struct ioat_dmadev, status); + + /* Ensure all counters are reset, if reconfiguring/restarting device. */ + ioat->next_read = 0; + ioat->next_write = 0; + ioat->last_write = 0; + ioat->offset = 0; + ioat->failure = 0; + + /* Reset Stats. */ + memset(&ioat->stats, 0, sizeof(ioat->stats)); + + /* Configure descriptor ring - each one points to next. */ + for (i = 0; i < ioat->qcfg.nb_desc; i++) { + ioat->desc_ring[i].next = ioat->ring_addr + + (((i + 1) % ioat->qcfg.nb_desc) * DESC_SZ); + } + + return 0; +} + +/* Get device information of a device. */ +static int +ioat_dev_info_get(const struct rte_dmadev *dev, struct rte_dmadev_info *info, uint32_t size) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + if (size < sizeof(*info)) + return -EINVAL; + info->device = dev->device; + info->dev_capa = RTE_DMADEV_CAPA_MEM_TO_MEM | + RTE_DMADEV_CAPA_OPS_COPY | RTE_DMADEV_CAPA_OPS_FILL, + info->max_vchans = 1; + info->min_desc = 32; + info->max_desc = 4096; + info->nb_vchans = (ioat->desc_ring == NULL); + return 0; +} + +/* Close a configured device. */ +static int +ioat_dev_close(struct rte_dmadev *dev) +{ + RTE_SET_USED(dev); + return 0; +} + /* Dump DMA device info. */ static int ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) @@ -77,7 +169,11 @@ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { static const struct rte_dmadev_ops ioat_dmadev_ops = { + .dev_close = ioat_dev_close, + .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, + .dev_info_get = ioat_dev_info_get, + .vchan_setup = ioat_vchan_setup, }; struct rte_dmadev *dmadev = NULL; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v3 05/11] dma/ioat: add start and stop functions 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (3 preceding siblings ...) 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 04/11] dma/ioat: add configuration functions Conor Walsh @ 2021-09-08 10:39 ` Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 06/11] dma/ioat: add data path job submission functions Conor Walsh ` (5 subsequent siblings) 10 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-08 10:39 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add start, stop and recover functions for IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 3 ++ drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index b6d88fe966..f7742642b5 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -86,3 +86,6 @@ The following code shows how the device is configured in ``test_dmadev.c``: :start-after: Setup of the dmadev device. 8< :end-before: >8 End of setup of the dmadev device. :dedent: 1 + +Once configured, the device can then be made ready for use by calling the +``rte_dmadev_start()`` API. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index e23cbb2068..76ef977e93 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -81,6 +81,96 @@ ioat_vchan_setup(struct rte_dmadev *dev, uint16_t vchan __rte_unused, return 0; } +/* Recover IOAT device. */ +static inline int +__ioat_recover(struct ioat_dmadev *ioat) +{ + uint32_t chanerr, retry = 0; + uint16_t mask = ioat->qcfg.nb_desc - 1; + + /* Clear any channel errors. Reading and writing to chanerr does this. */ + chanerr = ioat->regs->chanerr; + ioat->regs->chanerr = chanerr; + + /* Reset Channel. */ + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + + /* Write new chain address to trigger state change. */ + ioat->regs->chainaddr = ioat->desc_ring[(ioat->next_read - 1) & mask].next; + /* Ensure channel control and status addr are correct. */ + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + ioat->regs->chancmp = ioat->status_addr; + + /* Allow HW time to move to the ARMED state. */ + do { + rte_pause(); + retry++; + } while (ioat->regs->chansts != IOAT_CHANSTS_ARMED && retry < 200); + + /* Exit as failure if device is still HALTED. */ + if (ioat->regs->chansts != IOAT_CHANSTS_ARMED) + return -1; + + /* Store next write as offset as recover will move HW and SW ring out of sync. */ + ioat->offset = ioat->next_read; + + /* Prime status register with previous address. */ + ioat->status = ioat->desc_ring[(ioat->next_read - 2) & mask].next; + + return 0; +} + +/* Start a configured device. */ +static int +ioat_dev_start(struct rte_dmadev *dev) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + if (ioat->qcfg.nb_desc == 0 || ioat->desc_ring == NULL) + return -EBUSY; + + /* Inform hardware of where the descriptor ring is. */ + ioat->regs->chainaddr = ioat->ring_addr; + /* Inform hardware of where to write the status/completions. */ + ioat->regs->chancmp = ioat->status_addr; + + /* Prime the status register to be set to the last element. */ + ioat->status = ioat->ring_addr + ((ioat->qcfg.nb_desc - 1) * DESC_SZ); + + printf("IOAT.status: %s [%#lx]\n", + chansts_readable[ioat->status & IOAT_CHANSTS_STATUS], + ioat->status); + + if ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_HALTED) { + IOAT_PMD_WARN("Device HALTED on start, attempting to recover\n"); + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device couldn't be recovered"); + return -1; + } + } + + return 0; +} + +/* Stop a configured device. */ +static int +ioat_dev_stop(struct rte_dmadev *dev) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint32_t retry = 0; + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + + do { + rte_pause(); + retry++; + } while ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) != IOAT_CHANSTS_SUSPENDED + && retry < 200); + + return ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_SUSPENDED) ? 0 : -1; +} + /* Get device information of a device. */ static int ioat_dev_info_get(const struct rte_dmadev *dev, struct rte_dmadev_info *info, uint32_t size) @@ -173,6 +263,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, .dev_info_get = ioat_dev_info_get, + .dev_start = ioat_dev_start, + .dev_stop = ioat_dev_stop, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v3 06/11] dma/ioat: add data path job submission functions 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (4 preceding siblings ...) 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 05/11] dma/ioat: add start and stop functions Conor Walsh @ 2021-09-08 10:39 ` Conor Walsh 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 07/11] dma/ioat: add data path completion functions Conor Walsh ` (4 subsequent siblings) 10 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-08 10:39 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add data path functions for enqueuing and submitting operations to IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 54 ++++++++++++++++++++ drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index f7742642b5..16acfda3e3 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -89,3 +89,57 @@ The following code shows how the device is configured in ``test_dmadev.c``: Once configured, the device can then be made ready for use by calling the ``rte_dmadev_start()`` API. + +Performing Data Copies +~~~~~~~~~~~~~~~~~~~~~~~ + +To perform data copies using IOAT dmadev devices, the functions +``rte_dmadev_copy()`` and ``rte_dmadev_submit()`` should be used. Alternatively +``rte_dmadev_copy()`` can be called with the ``RTE_DMA_OP_FLAG_SUBMIT`` flag +set. + +The ``rte_dmadev_copy()`` function enqueues a single copy to the +device ring for copying at a later point. The parameters to the function +include the device ID of the desired device, the virtual DMA channel required +(always 0 for IOAT), the IOVA addresses of both the source and destination +buffers, the length of the data to be copied and any operation flags. The +function will return the index of the enqueued job which can be use to +track that operation. + +While the ``rte_dmadev_copy()`` function enqueues a copy operation on the device +ring, the copy will not actually be performed until after the application calls +the ``rte_dmadev_submit()`` function. This function informs the device hardware +of the elements enqueued on the ring, and the device will begin to process them. +It is expected that, for efficiency reasons, a burst of operations will be +enqueued to the device via multiple enqueue calls between calls to the +``rte_dmadev_submit()`` function. If desired you can pass the +``RTE_DMA_OP_FLAG_SUBMIT`` flag when calling ``rte_dmadev_copy()`` and this will +tell the device to perform the enqueued operation and any unperformed operations +before it. The ``RTE_DMA_OP_FLAG_SUBMIT`` flag can be passed instead of calling +the ``rte_dmadev_submit()`` function for example on the last enqueue of the burst. + +The following code from demonstrates how to enqueue a burst of copies to the +device and start the hardware processing of them: + +.. code-block:: C + + for (i = 0; i < BURST_SIZE; i++) { + if (rte_dmadev_copy(dev_id, vchan, rte_mbuf_data_iova(srcs[i]), + rte_mbuf_data_iova(dsts[i]), COPY_LEN, 0) < 0) { + PRINT_ERR("Error with rte_dmadev_copy for buffer %u\n", i); + return -1; + } + } + if (rte_dmadev_submit(dev_id, vchan) < 0) { + PRINT_ERR("Error with performing operations\n", i); + return -1; + } + +Filling an Area of Memory +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The driver also has support for the ``fill`` operation, where an area +of memory is overwritten, or filled, with a short pattern of data. +Fill operations can be performed in much the same was as copy operations +described above, just using the ``rte_dmadev_fill()`` function rather +than the ``rte_dmadev_copy()`` function. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 76ef977e93..41edfcfe86 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -5,6 +5,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> #include <rte_malloc.h> +#include <rte_prefetch.h> #include "ioat_internal.h" @@ -17,6 +18,12 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* IOAT operations. */ +enum rte_ioat_ops { + ioat_op_copy = 0, /* Standard DMA Operation */ + ioat_op_fill /* Block Fill */ +}; + /* Configure a device. */ static int ioat_dev_configure(struct rte_dmadev *dev __rte_unused, const struct rte_dmadev_conf *dev_conf, @@ -197,6 +204,87 @@ ioat_dev_close(struct rte_dmadev *dev) return 0; } +/* Trigger hardware to begin performing enqueued operations. */ +static inline void +__submit(struct ioat_dmadev *ioat) +{ + *ioat->doorbell = ioat->next_write - ioat->offset; + + ioat->last_write = ioat->next_write; +} + +/* External submit function wrapper. */ +static int +ioat_submit(struct rte_dmadev *dev, uint16_t qid __rte_unused) +{ + struct ioat_dmadev *ioat = (struct ioat_dmadev *)dev->dev_private; + + __submit(ioat); + + return 0; +} + +/* Write descriptor for enqueue. */ +static inline int +__write_desc(struct rte_dmadev *dev, uint32_t op, uint64_t src, phys_addr_t dst, + unsigned int length, uint64_t flags) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint16_t ret; + const unsigned short mask = ioat->qcfg.nb_desc - 1; + const unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + const unsigned short space = mask + read - write; + struct ioat_dma_hw_desc *desc; + + if (space == 0) + return -ENOSPC; + + ioat->next_write = write + 1; + write &= mask; + + desc = &ioat->desc_ring[write]; + desc->size = length; + desc->u.control_raw = (uint32_t)((op << IOAT_CMD_OP_SHIFT) | + (1 << IOAT_COMP_UPDATE_SHIFT)); + + /* In IOAT the fence ensures that all operations including the current one + * are completed before moving on, DMAdev assumes that the fence ensures + * all operations before the current one are completed before starting + * the current one, so in IOAT we set the fence for the previous descriptor. + */ + if (flags & RTE_DMA_OP_FLAG_FENCE) + ioat->desc_ring[(write - 1) & mask].u.control.fence = 1; + + desc->src_addr = src; + desc->dest_addr = dst; + + rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]); + + ret = (uint16_t)(ioat->next_write - 1); + + if (flags & RTE_DMA_OP_FLAG_SUBMIT) + __submit(ioat); + + return ret; +} + +/* Enqueue a fill operation onto the ioat device. */ +static int +ioat_enqueue_fill(struct rte_dmadev *dev, uint16_t qid __rte_unused, uint64_t pattern, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev, ioat_op_fill, pattern, dst, length, flags); +} + +/* Enqueue a copy operation onto the ioat device. */ +static int +ioat_enqueue_copy(struct rte_dmadev *dev, uint16_t qid __rte_unused, rte_iova_t src, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev, ioat_op_copy, src, dst, length, flags); +} + /* Dump DMA device info. */ static int ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) @@ -292,6 +380,10 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->copy = ioat_enqueue_copy; + dmadev->fill = ioat_enqueue_fill; + dmadev->submit = ioat_submit; + ioat = dmadev->data->dev_private; ioat->dmadev = dmadev; ioat->regs = dev->mem_resource[0].addr; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v3 07/11] dma/ioat: add data path completion functions 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (5 preceding siblings ...) 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 06/11] dma/ioat: add data path job submission functions Conor Walsh @ 2021-09-08 10:40 ` Conor Walsh 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 08/11] dma/ioat: add statistics Conor Walsh ` (3 subsequent siblings) 10 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-08 10:40 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add the data path functions for gathering completed operations from IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 48 ++++++++++- drivers/dma/ioat/ioat_dmadev.c | 141 +++++++++++++++++++++++++++++++++ 2 files changed, 188 insertions(+), 1 deletion(-) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 16acfda3e3..9ea5708090 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -96,7 +96,9 @@ Performing Data Copies To perform data copies using IOAT dmadev devices, the functions ``rte_dmadev_copy()`` and ``rte_dmadev_submit()`` should be used. Alternatively ``rte_dmadev_copy()`` can be called with the ``RTE_DMA_OP_FLAG_SUBMIT`` flag -set. +set. Once copies have been completed, the completion will be reported back when +the application calls ``rte_dmadev_completed()`` or +``rte_dmadev_completed_status()``. The ``rte_dmadev_copy()`` function enqueues a single copy to the device ring for copying at a later point. The parameters to the function @@ -135,6 +137,50 @@ device and start the hardware processing of them: return -1; } +To retrieve information about completed copies, the API +``rte_dmadev_completed()`` or ``rte_dmadev_completed_status ()``should be used. +``rte_dmadev_completed()`` will return to the application how many operations +have successfully completed up to a supplied maximum, it can also pass back the +index of the last completed operation and if a completion within the requested +batch has failed. If ``rte_dmadev_completed()`` passes back ``has_error`` as +``true`` then ``rte_dmadev_completed_status()`` should be called to retrieve +the detailed status of each completion in that batch. Once an error has occurred +``rte_dmadev_completed()`` will not process any more completions until +``rte_dmadev_completed_status()`` has been called to get the error information. +The supplied ``status`` array will contain either ``RTE_DMA_STATUS_SUCCESSFUL`` +if the operation was successful or a DMA error code if the operation failed. +The status codes supported by IOAT are: + +* ``RTE_DMA_STATUS_SUCCESSFUL``: The operation was successful. +* ``RTE_DMA_STATUS_INVALID_SRC_ADDR``: The operation failed due to an invalid source address. +* ``RTE_DMA_STATUS_INVALID_DST_ADDR``: The operation failed due to an invalid destination address. +* ``RTE_DMA_STATUS_INVALID_LENGTH``: The operation failed due to an invalid descriptor length. +* ``RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR``: The device could not read the descriptor. +* ``RTE_DMA_STATUS_ERROR_UNKNOWN``: The operation failed due to an unspecified error. + +``rte_dmadev_completed_status()`` can be used on it's own instead of with +``rte_dmadev_completed()`` but calling ``rte_dmadev_completed_status()`` may be +slower than ``rte_dmadev_completed()``. + +The following code shows how to retrieve the number of successfully completed +copies within a burst and then using ``rte_dmadev_completed_status()`` to check +which operation failed and reset the device: + +.. code-block:: C + + enum rte_dma_status_code status[COMP_BURST_SZ]; + uint16_t count, idx, status_count; + bool error = 0; + + count = rte_dmadev_completed(dev_id, vchan, COMP_BURST_SZ, &idx, &error); + + if (error){ + status_count = rte_dmadev_completed_status(dev_id, vchan, COMP_BURST_SZ, &idx, status); + } + +After this snippet the application would use the status array to find which copy +failed and handle it appropriately. + Filling an Area of Memory ~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 41edfcfe86..74ee21222a 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -6,6 +6,7 @@ #include <rte_dmadev_pmd.h> #include <rte_malloc.h> #include <rte_prefetch.h> +#include <rte_errno.h> #include "ioat_internal.h" @@ -342,6 +343,144 @@ ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) return 0; } +/* Returns the index of the last completed operation. */ +static inline uint16_t +__get_last_completed(const struct ioat_dmadev *ioat, int *state) +{ + /* Status register contains the address of the completed operation */ + uint64_t status = ioat->status; + + /* lower 3 bits indicate "transfer status" : active, idle, halted. + * We can ignore bit 0. + */ + *state = status & IOAT_CHANSTS_STATUS; + + /* If we are just after recovering from an error the address returned by + * status will be 0, in this case we return the offset - 1 as the last + * completed. If not return the status value minus the chainaddr which + * gives us an offset into the ring. Right shifting by 6 (divide by 64) + * gives the index of the completion from the HW point of view and adding + * the offset translates the ring index from HW to SW point of view. + */ + if ((status & ~IOAT_CHANSTS_STATUS) == 0) + return ioat->offset - 1; + + return (status - ioat->ring_addr) >> 6; +} + +/* Translates IOAT ChanERRs to DMA error codes. */ +static inline enum rte_dma_status_code +__translate_status_ioat_to_dma(uint32_t chanerr) +{ + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_SRC_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_DST_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + return RTE_DMA_STATUS_INVALID_LENGTH; + else if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + return RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR; + else + return RTE_DMA_STATUS_ERROR_UNKNOWN; +} + +/* Returns details of operations that have been completed. */ +static uint16_t +ioat_completed(struct rte_dmadev *dev, uint16_t qid __rte_unused, const uint16_t max_ops, + uint16_t *last_idx, bool *has_error) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short last_completed, count; + int state, fails = 0; + + /* Do not do any work if there is an uncleared error. */ + if (ioat->failure != 0) { + *has_error = true; + *last_idx = ioat->next_read - 2; + return 0; + } + + last_completed = __get_last_completed(ioat, &state); + count = (last_completed + 1 - read) & mask; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || state != IOAT_CHANSTS_HALTED) { + ioat->next_read = read + count; + *last_idx = ioat->next_read - 1; + } else { + *has_error = true; + rte_errno = EIO; + ioat->failure = ioat->regs->chanerr; + ioat->next_read = read + count + 1; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + ioat_dev_dump(dev, stdout); + return 0; + } + __submit(ioat); + fails++; + *last_idx = ioat->next_read - 2; + } + + return count; +} + +/* Returns detailed status information about operations that have been completed. */ +static uint16_t +ioat_completed_status(struct rte_dmadev *dev, uint16_t qid __rte_unused, + uint16_t max_ops, uint16_t *last_idx, enum rte_dma_status_code *status) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short count, last_completed; + uint64_t fails = 0; + int state, i; + + last_completed = __get_last_completed(ioat, &state); + count = (last_completed + 1 - read) & mask; + + for (i = 0; i < RTE_MIN(count + 1, max_ops); i++) + status[i] = RTE_DMA_STATUS_SUCCESSFUL; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || state != IOAT_CHANSTS_HALTED) + ioat->next_read = read + count; + else { + rte_errno = EIO; + status[count] = __translate_status_ioat_to_dma(ioat->regs->chanerr); + count++; + ioat->next_read = read + count; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + ioat_dev_dump(dev, stdout); + return 0; + } + __submit(ioat); + fails++; + } + + if (ioat->failure > 0) { + status[0] = __translate_status_ioat_to_dma(ioat->failure); + count = RTE_MIN(count + 1, max_ops); + ioat->failure = 0; + } + + *last_idx = ioat->next_read - 1; + + return count; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -380,6 +519,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->completed = ioat_completed; + dmadev->completed_status = ioat_completed_status; dmadev->copy = ioat_enqueue_copy; dmadev->fill = ioat_enqueue_fill; dmadev->submit = ioat_submit; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v3 08/11] dma/ioat: add statistics 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (6 preceding siblings ...) 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 07/11] dma/ioat: add data path completion functions Conor Walsh @ 2021-09-08 10:40 ` Conor Walsh 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 09/11] dma/ioat: add support for vchan status function Conor Walsh ` (2 subsequent siblings) 10 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-08 10:40 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add statistic tracking for operations in IOAT. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 23 +++++++++++++++++++ drivers/dma/ioat/ioat_dmadev.c | 40 ++++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 9ea5708090..620b676718 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -189,3 +189,26 @@ of memory is overwritten, or filled, with a short pattern of data. Fill operations can be performed in much the same was as copy operations described above, just using the ``rte_dmadev_fill()`` function rather than the ``rte_dmadev_copy()`` function. + +Querying Device Statistics +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The statistics from the IOAT dmadev device can be got via the +``rte_dmadev_stats_get()`` API in the ``rte_dmadev`` library. + +The statistics returned for each IOAT device are: + +* ``submitted``: The number of operations submitted to the device. +* ``completed``: The number of operations successfully completed by the device. +* ``errors``: The number of operations that failed. + +The stats function can be used as follows: + +.. code-block:: C + + struct rte_dmadev_stats stats; + if (rte_dmadev_stats_get(dmadev_id, vchan, &stats) >= 0) { + printf("Total submitted ops: %lu", stats.submitted); + printf("Total completed ops: %lu", stats.completed); + printf("Total failed ops: %lu", stats.errors); + } diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 74ee21222a..0f51a33279 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -211,6 +211,8 @@ __submit(struct ioat_dmadev *ioat) { *ioat->doorbell = ioat->next_write - ioat->offset; + ioat->stats.submitted += (uint16_t)(ioat->next_write - ioat->last_write); + ioat->last_write = ioat->next_write; } @@ -339,6 +341,10 @@ ioat_dev_dump(const struct rte_dmadev *dev, FILE *f) fprintf(f, " Dest: %#lx\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); fprintf(f, " Next: %#lx\n", ioat->desc_ring[ioat->next_read & mask].next); fprintf(f, " }\n"); + fprintf(f, " Key Stats { submitted: %"PRIu64", comp: %"PRIu64", failed: %"PRIu64" }\n", + ioat->stats.submitted, + ioat->stats.completed, + ioat->stats.errors); return 0; } @@ -428,6 +434,9 @@ ioat_completed(struct rte_dmadev *dev, uint16_t qid __rte_unused, const uint16_t *last_idx = ioat->next_read - 2; } + ioat->stats.completed += count; + ioat->stats.errors += fails; + return count; } @@ -478,9 +487,38 @@ ioat_completed_status(struct rte_dmadev *dev, uint16_t qid __rte_unused, *last_idx = ioat->next_read - 1; + ioat->stats.completed += count; + ioat->stats.errors += fails; + return count; } +/* Retrieve the generic stats of a DMA device. */ +static int +ioat_stats_get(const struct rte_dmadev *dev, uint16_t vchan __rte_unused, + struct rte_dmadev_stats *rte_stats, uint32_t size) +{ + struct rte_dmadev_stats *stats = (&((struct ioat_dmadev *)dev->dev_private)->stats); + + if (size < sizeof(rte_stats)) + return -EINVAL; + if (rte_stats == NULL) + return -EINVAL; + + *rte_stats = *stats; + return 0; +} + +/* Reset the generic stat counters for the DMA device. */ +static int +ioat_stats_reset(struct rte_dmadev *dev, uint16_t vchan __rte_unused) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + memset(&ioat->stats, 0, sizeof(ioat->stats)); + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -492,6 +530,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_info_get = ioat_dev_info_get, .dev_start = ioat_dev_start, .dev_stop = ioat_dev_stop, + .stats_get = ioat_stats_get, + .stats_reset = ioat_stats_reset, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v3 09/11] dma/ioat: add support for vchan status function 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (7 preceding siblings ...) 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 08/11] dma/ioat: add statistics Conor Walsh @ 2021-09-08 10:40 ` Conor Walsh 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 10/11] dma/ioat: add burst capacity function Conor Walsh 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 11/11] devbind: move ioat device ID for ICX to dmadev category Conor Walsh 10 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-08 10:40 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add support for the rte_dmadev_vchan_status API call. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 0f51a33279..1b1381675f 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -519,6 +519,26 @@ ioat_stats_reset(struct rte_dmadev *dev, uint16_t vchan __rte_unused) return 0; } +/* Check if the IOAT device is idle. */ +static int +ioat_vchan_status(const struct rte_dmadev *dev, uint16_t vchan __rte_unused, + enum rte_dmadev_vchan_status *status) +{ + int state = 0; + const struct ioat_dmadev *ioat = dev->dev_private; + const uint16_t mask = ioat->qcfg.nb_desc - 1; + const uint16_t last = __get_last_completed(ioat, &state); + + if (state == IOAT_CHANSTS_HALTED || state == IOAT_CHANSTS_SUSPENDED) + *status = RTE_DMA_VCHAN_HALTED_ERROR; + else if (last == ((ioat->next_write - 1) & mask)) + *status = RTE_DMA_VCHAN_IDLE; + else + *status = RTE_DMA_VCHAN_ACTIVE; + + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -532,6 +552,7 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_stop = ioat_dev_stop, .stats_get = ioat_stats_get, .stats_reset = ioat_stats_reset, + .vchan_status = ioat_vchan_status, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v3 10/11] dma/ioat: add burst capacity function 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (8 preceding siblings ...) 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 09/11] dma/ioat: add support for vchan status function Conor Walsh @ 2021-09-08 10:40 ` Conor Walsh 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 11/11] devbind: move ioat device ID for ICX to dmadev category Conor Walsh 10 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-08 10:40 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Adds the ability to find the remaining space in the IOAT ring. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 1b1381675f..578b12c8a3 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -493,6 +493,19 @@ ioat_completed_status(struct rte_dmadev *dev, uint16_t qid __rte_unused, return count; } +/* Get the remaining capacity of the ring. */ +static uint16_t +ioat_burst_capacity(const struct rte_dmadev *dev, uint16_t vchan __rte_unused) +{ + struct ioat_dmadev *ioat = dev->data->dev_private; + unsigned short size = ioat->qcfg.nb_desc - 1; + unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + unsigned short space = size - (write - read); + + return space; +} + /* Retrieve the generic stats of a DMA device. */ static int ioat_stats_get(const struct rte_dmadev *dev, uint16_t vchan __rte_unused, @@ -544,6 +557,7 @@ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { static const struct rte_dmadev_ops ioat_dmadev_ops = { + .burst_capacity = ioat_burst_capacity, .dev_close = ioat_dev_close, .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v3 11/11] devbind: move ioat device ID for ICX to dmadev category 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (9 preceding siblings ...) 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 10/11] dma/ioat: add burst capacity function Conor Walsh @ 2021-09-08 10:40 ` Conor Walsh 10 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-08 10:40 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Move Intel IOAT devices on Ice Lake systems from Misc to DMA devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- usertools/dpdk-devbind.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index 98b698ccc0..afebc8cb62 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -69,14 +69,13 @@ network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] -dma_devices = [intel_idxd_spr] +dma_devices = [intel_idxd_spr, intel_ioat_icx] eventdev_devices = [cavium_sso, cavium_tim, intel_dlb, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] regex_devices = [octeontx2_ree] misc_devices = [cnxk_bphy, cnxk_bphy_cgx, intel_ioat_bdw, intel_ioat_skx, - intel_ioat_icx, intel_ntb_skx, intel_ntb_icx, - octeontx2_dma] + intel_ntb_skx, intel_ntb_icx, octeontx2_dma] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh ` (8 preceding siblings ...) 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices Conor Walsh @ 2021-09-17 15:42 ` Conor Walsh 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 01/11] dma/ioat: add device probe and removal functions Conor Walsh ` (10 more replies) 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (3 subsequent siblings) 13 siblings, 11 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-17 15:42 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh This patchset adds a dmadev driver and associated documentation to support Intel QuickData Technology devices, part of the Intel I/O Acceleration Technology (Intel I/OAT). This driver is intended to ultimately replace the current IOAT part of the IOAT rawdev driver. This patchset passes all the driver tests added in the dmadev test suite. NOTE: This patchset has several dependencies: - v22 of the dmadev set [1] - v5 of the dmadev test suite [2] - v5 of the IDXD driver [3] [1] http://patches.dpdk.org/project/dpdk/list/?series=18960 [2] http://patches.dpdk.org/project/dpdk/list/?series=19017 [3] http://patches.dpdk.org/project/dpdk/list/?series=19023 --- v4: - Changes needed to update from dmadev v21 to v22. - Fixed 32-bit build. - Made stats reset logic easier to understand. v3: - Added burst capacity function. - Stop function now waits for suspend rather than just using a sleep. - Changed from vchan idle to vchan status function. - Other minor changes to update from dmadev v19 to v21. v2: - Rebased on the above patchsets. - Added support for the vchan idle function. - Stop function now suspends IOAT channel to allow for reconfig. - dmadev_autotest can now be run multiple times using the IOAT driver without errors. - Added devbind updates for DMA devices - Removed some logically dead code found by coverity in the create function. Conor Walsh (11): dma/ioat: add device probe and removal functions dma/ioat: create dmadev instances on PCI probe dma/ioat: add datapath structures dma/ioat: add configuration functions dma/ioat: add start and stop functions dma/ioat: add data path job submission functions dma/ioat: add data path completion functions dma/ioat: add statistics dma/ioat: add support for vchan status function dma/ioat: add burst capacity function devbind: move ioat device ID for ICX to dmadev category MAINTAINERS | 6 + doc/guides/dmadevs/index.rst | 2 + doc/guides/dmadevs/ioat.rst | 214 +++++++ doc/guides/rel_notes/release_21_11.rst | 7 +- drivers/dma/ioat/ioat_dmadev.c | 739 +++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 294 ++++++++++ drivers/dma/ioat/ioat_internal.h | 44 ++ drivers/dma/ioat/meson.build | 7 + drivers/dma/ioat/version.map | 3 + drivers/dma/meson.build | 1 + usertools/dpdk-devbind.py | 5 +- 11 files changed, 1316 insertions(+), 6 deletions(-) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v4 01/11] dma/ioat: add device probe and removal functions 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices Conor Walsh @ 2021-09-17 15:42 ` Conor Walsh 2021-09-20 11:15 ` Bruce Richardson 2021-09-22 3:59 ` fengchengwen 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 02/11] dma/ioat: create dmadev instances on PCI probe Conor Walsh ` (9 subsequent siblings) 10 siblings, 2 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-17 15:42 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add the basic device probe/remove skeleton code and initial documentation for new IOAT DMA driver. Maintainers update is also included in this patch. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- MAINTAINERS | 6 +++ doc/guides/dmadevs/index.rst | 2 + doc/guides/dmadevs/ioat.rst | 64 ++++++++++++++++++++++++ doc/guides/rel_notes/release_21_11.rst | 7 +-- drivers/dma/ioat/ioat_dmadev.c | 69 ++++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 35 +++++++++++++ drivers/dma/ioat/ioat_internal.h | 20 ++++++++ drivers/dma/ioat/meson.build | 7 +++ drivers/dma/ioat/version.map | 3 ++ drivers/dma/meson.build | 1 + 10 files changed, 211 insertions(+), 3 deletions(-) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map diff --git a/MAINTAINERS b/MAINTAINERS index 9cb59b831d..70993d23e8 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1209,6 +1209,12 @@ M: Kevin Laatz <kevin.laatz@intel.com> F: drivers/dma/idxd/ F: doc/guides/dmadevs/idxd.rst +Intel IOAT - EXPERIMENTAL +M: Bruce Richardson <bruce.richardson@intel.com> +M: Conor Walsh <conor.walsh@intel.com> +F: drivers/dma/ioat/ +F: doc/guides/dmadevs/ioat.rst + RegEx Drivers ------------- diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst index 5d4abf880e..c59f4b5c92 100644 --- a/doc/guides/dmadevs/index.rst +++ b/doc/guides/dmadevs/index.rst @@ -12,3 +12,5 @@ an application through DMA API. :numbered: idxd + ioat + diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst new file mode 100644 index 0000000000..45a2e65d70 --- /dev/null +++ b/doc/guides/dmadevs/ioat.rst @@ -0,0 +1,64 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2021 Intel Corporation. + +.. include:: <isonum.txt> + +IOAT DMA Device Driver +======================= + +The ``ioat`` dmadev driver provides a poll-mode driver (PMD) for Intel\ +|reg| QuickData Technology which is part of part of Intel\ |reg| I/O +Acceleration Technology (`Intel I/OAT +<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_). +This PMD, when used on supported hardware, allows data copies, for example, +cloning packet data, to be accelerated by IOAT hardware rather than having to +be done by software, freeing up CPU cycles for other tasks. + +Hardware Requirements +---------------------- + +The ``dpdk-devbind.py`` script, included with DPDK, can be used to show the +presence of supported hardware. Running ``dpdk-devbind.py --status-dev dma`` +will show all the DMA devices on the system, IOAT devices are included in this +list. For Intel\ |reg| IOAT devices, the hardware will often be listed as +"Crystal Beach DMA", or "CBDMA" or on some newer systems '0b00' due to the +absence of pci-id database entries for them at this point. + +Compilation +------------ + +For builds using ``meson`` and ``ninja``, the driver will be built when the +target platform is x86-based. No additional compilation steps are necessary. + +Device Setup +------------- + +Intel\ |reg| IOAT devices will need to be bound to a suitable DPDK-supported +user-space IO driver such as ``vfio-pci`` in order to be used by DPDK. + +The ``dpdk-devbind.py`` script can be used to view the state of the devices using:: + + $ dpdk-devbind.py --status-dev dma + +The ``dpdk-devbind.py`` script can also be used to bind devices to a suitable driver. +For example:: + + $ dpdk-devbind.py -b vfio-pci 00:01.0 00:01.1 + +Device Probing and Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For devices bound to a suitable DPDK-supported driver (``vfio-pci``), the HW +devices will be found as part of the device scan done at application +initialization time without the need to pass parameters to the application. + +If the application does not require all the devices available an allowlist can +be used in the same way that other DPDK devices use them. + +For example:: + + $ dpdk-test -a <b:d:f> + +Once probed successfully, the device will appear as a ``dmadev``, that is a +"DMA device type" inside DPDK, and can be accessed using APIs from the +``rte_dmadev`` library. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index c0bfd9c1ba..4d2b7bde1b 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -92,10 +92,11 @@ New Features * Device allocation and it's multi-process support. * Control and data plane functions. -* **Added IDXD dmadev driver implementation.** +* **Added Intel dmadev driver implementations.** - The IDXD dmadev driver provide device drivers for the Intel DSA devices. - This device driver can be used through the generic dmadev API. + The IDXD and IOAT dmadev drivers provide device drivers for Intel DSA + and IOAT devices. These device drivers can be used through the generic + dmadev API. Removed Items diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c new file mode 100644 index 0000000000..f3491d45b1 --- /dev/null +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#include <rte_bus_pci.h> +#include <rte_dmadev_pmd.h> + +#include "ioat_internal.h" + +static struct rte_pci_driver ioat_pmd_drv; + +RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); + +#define IOAT_PMD_NAME dmadev_ioat +#define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) + +/* Probe DMA device. */ +static int +ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); + + dev->device.driver = &drv->driver; + return 0; +} + +/* Remove DMA device. */ +static int +ioat_dmadev_remove(struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + + IOAT_PMD_INFO("Closing %s on NUMA node %d", + name, dev->device.numa_node); + + return 0; +} + +static const struct rte_pci_id pci_id_ioat_map[] = { + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_SKX) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX0) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX1) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX2) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX3) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX4) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX5) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX6) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX7) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXE) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXF) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_ICX) }, + { .vendor_id = 0, /* sentinel */ }, +}; + +static struct rte_pci_driver ioat_pmd_drv = { + .id_table = pci_id_ioat_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC, + .probe = ioat_dmadev_probe, + .remove = ioat_dmadev_remove, +}; + +RTE_PMD_REGISTER_PCI(IOAT_PMD_NAME, ioat_pmd_drv); +RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_NAME, pci_id_ioat_map); +RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_NAME, "* igb_uio | uio_pci_generic | vfio-pci"); diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h new file mode 100644 index 0000000000..eeabba41ef --- /dev/null +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#ifndef IOAT_HW_DEFS_H +#define IOAT_HW_DEFS_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> + +#define IOAT_VER_3_0 0x30 +#define IOAT_VER_3_3 0x33 + +#define IOAT_VENDOR_ID 0x8086 +#define IOAT_DEVICE_ID_SKX 0x2021 +#define IOAT_DEVICE_ID_BDX0 0x6f20 +#define IOAT_DEVICE_ID_BDX1 0x6f21 +#define IOAT_DEVICE_ID_BDX2 0x6f22 +#define IOAT_DEVICE_ID_BDX3 0x6f23 +#define IOAT_DEVICE_ID_BDX4 0x6f24 +#define IOAT_DEVICE_ID_BDX5 0x6f25 +#define IOAT_DEVICE_ID_BDX6 0x6f26 +#define IOAT_DEVICE_ID_BDX7 0x6f27 +#define IOAT_DEVICE_ID_BDXE 0x6f2E +#define IOAT_DEVICE_ID_BDXF 0x6f2F +#define IOAT_DEVICE_ID_ICX 0x0b00 + +#ifdef __cplusplus +} +#endif + +#endif /* IOAT_HW_DEFS_H */ diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h new file mode 100644 index 0000000000..f1ec12a919 --- /dev/null +++ b/drivers/dma/ioat/ioat_internal.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021 Intel Corporation + */ + +#ifndef _IOAT_INTERNAL_H_ +#define _IOAT_INTERNAL_H_ + +#include "ioat_hw_defs.h" + +extern int ioat_pmd_logtype; + +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + ioat_pmd_logtype, "IOAT: %s(): " fmt "\n", __func__, ##args) + +#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) +#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) +#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) +#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) + +#endif /* _IOAT_INTERNAL_H_ */ diff --git a/drivers/dma/ioat/meson.build b/drivers/dma/ioat/meson.build new file mode 100644 index 0000000000..d67fac96fb --- /dev/null +++ b/drivers/dma/ioat/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021 Intel Corporation + +build = dpdk_conf.has('RTE_ARCH_X86') +reason = 'only supported on x86' +sources = files('ioat_dmadev.c') +deps += ['bus_pci', 'dmadev'] diff --git a/drivers/dma/ioat/version.map b/drivers/dma/ioat/version.map new file mode 100644 index 0000000000..c2e0723b4c --- /dev/null +++ b/drivers/dma/ioat/version.map @@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +}; diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build index 411be7a240..a69418ce9b 100644 --- a/drivers/dma/meson.build +++ b/drivers/dma/meson.build @@ -3,6 +3,7 @@ drivers = [ 'idxd', + 'ioat', 'skeleton', ] std_deps = ['dmadev'] -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 01/11] dma/ioat: add device probe and removal functions 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 01/11] dma/ioat: add device probe and removal functions Conor Walsh @ 2021-09-20 11:15 ` Bruce Richardson 2021-09-21 16:24 ` Conor Walsh 2021-09-22 3:59 ` fengchengwen 1 sibling, 1 reply; 132+ messages in thread From: Bruce Richardson @ 2021-09-20 11:15 UTC (permalink / raw) To: Conor Walsh; +Cc: fengchengwen, jerinj, kevin.laatz, dev On Fri, Sep 17, 2021 at 03:42:17PM +0000, Conor Walsh wrote: > Add the basic device probe/remove skeleton code and initial documentation > for new IOAT DMA driver. Maintainers update is also included in this > patch. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> > --- > MAINTAINERS | 6 +++ > doc/guides/dmadevs/index.rst | 2 + > doc/guides/dmadevs/ioat.rst | 64 ++++++++++++++++++++++++ > doc/guides/rel_notes/release_21_11.rst | 7 +-- > drivers/dma/ioat/ioat_dmadev.c | 69 ++++++++++++++++++++++++++ > drivers/dma/ioat/ioat_hw_defs.h | 35 +++++++++++++ > drivers/dma/ioat/ioat_internal.h | 20 ++++++++ > drivers/dma/ioat/meson.build | 7 +++ > drivers/dma/ioat/version.map | 3 ++ > drivers/dma/meson.build | 1 + > 10 files changed, 211 insertions(+), 3 deletions(-) > create mode 100644 doc/guides/dmadevs/ioat.rst > create mode 100644 drivers/dma/ioat/ioat_dmadev.c > create mode 100644 drivers/dma/ioat/ioat_hw_defs.h > create mode 100644 drivers/dma/ioat/ioat_internal.h > create mode 100644 drivers/dma/ioat/meson.build > create mode 100644 drivers/dma/ioat/version.map > > diff --git a/MAINTAINERS b/MAINTAINERS > index 9cb59b831d..70993d23e8 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -1209,6 +1209,12 @@ M: Kevin Laatz <kevin.laatz@intel.com> > F: drivers/dma/idxd/ > F: doc/guides/dmadevs/idxd.rst > > +Intel IOAT - EXPERIMENTAL > +M: Bruce Richardson <bruce.richardson@intel.com> > +M: Conor Walsh <conor.walsh@intel.com> > +F: drivers/dma/ioat/ > +F: doc/guides/dmadevs/ioat.rst > + > Unlike the raw/ioat driver, this dmadev driver does not have a private APIs so I'm not sure it needs the EXPERIMENTAL tag on it. > RegEx Drivers > ------------- > diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst > index 5d4abf880e..c59f4b5c92 100644 > --- a/doc/guides/dmadevs/index.rst > +++ b/doc/guides/dmadevs/index.rst > @@ -12,3 +12,5 @@ an application through DMA API. > :numbered: > > idxd > + ioat > + > diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst > new file mode 100644 > index 0000000000..45a2e65d70 > --- /dev/null > +++ b/doc/guides/dmadevs/ioat.rst > @@ -0,0 +1,64 @@ > +.. SPDX-License-Identifier: BSD-3-Clause > + Copyright(c) 2021 Intel Corporation. > + > +.. include:: <isonum.txt> > + > +IOAT DMA Device Driver > +======================= > + > +The ``ioat`` dmadev driver provides a poll-mode driver (PMD) for Intel\ > +|reg| QuickData Technology which is part of part of Intel\ |reg| I/O > +Acceleration Technology (`Intel I/OAT > +<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_). > +This PMD, when used on supported hardware, allows data copies, for example, > +cloning packet data, to be accelerated by IOAT hardware rather than having to > +be done by software, freeing up CPU cycles for other tasks. > + > +Hardware Requirements > +---------------------- > + > +The ``dpdk-devbind.py`` script, included with DPDK, can be used to show the > +presence of supported hardware. Running ``dpdk-devbind.py --status-dev dma`` > +will show all the DMA devices on the system, IOAT devices are included in this > +list. For Intel\ |reg| IOAT devices, the hardware will often be listed as > +"Crystal Beach DMA", or "CBDMA" or on some newer systems '0b00' due to the > +absence of pci-id database entries for them at this point. > + > +Compilation > +------------ > + > +For builds using ``meson`` and ``ninja``, the driver will be built when the > +target platform is x86-based. No additional compilation steps are necessary. > + > +Device Setup > +------------- > + > +Intel\ |reg| IOAT devices will need to be bound to a suitable DPDK-supported > +user-space IO driver such as ``vfio-pci`` in order to be used by DPDK. > + > +The ``dpdk-devbind.py`` script can be used to view the state of the devices using:: > + > + $ dpdk-devbind.py --status-dev dma > + > +The ``dpdk-devbind.py`` script can also be used to bind devices to a suitable driver. > +For example:: > + > + $ dpdk-devbind.py -b vfio-pci 00:01.0 00:01.1 > + > +Device Probing and Initialization > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +For devices bound to a suitable DPDK-supported driver (``vfio-pci``), the HW > +devices will be found as part of the device scan done at application > +initialization time without the need to pass parameters to the application. > + > +If the application does not require all the devices available an allowlist can > +be used in the same way that other DPDK devices use them. > + > +For example:: > + > + $ dpdk-test -a <b:d:f> > + > +Once probed successfully, the device will appear as a ``dmadev``, that is a > +"DMA device type" inside DPDK, and can be accessed using APIs from the > +``rte_dmadev`` library. > diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst > index c0bfd9c1ba..4d2b7bde1b 100644 > --- a/doc/guides/rel_notes/release_21_11.rst > +++ b/doc/guides/rel_notes/release_21_11.rst > @@ -92,10 +92,11 @@ New Features > * Device allocation and it's multi-process support. > * Control and data plane functions. > > -* **Added IDXD dmadev driver implementation.** > +* **Added Intel dmadev driver implementations.** > > - The IDXD dmadev driver provide device drivers for the Intel DSA devices. > - This device driver can be used through the generic dmadev API. > + The IDXD and IOAT dmadev drivers provide device drivers for Intel DSA > + and IOAT devices. These device drivers can be used through the generic > + dmadev API. > I'm not sure about merging two driver additions into a single release note entry - one for the doc maintainers and tree committers to comment on. Rather than "IOAT devices" I think the official name of the hardware should be used, and you probably should add "respectively" at the end of the first sentence to make it clear (once you change the name) which hardware is used by which driver. ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 01/11] dma/ioat: add device probe and removal functions 2021-09-20 11:15 ` Bruce Richardson @ 2021-09-21 16:24 ` Conor Walsh 0 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-21 16:24 UTC (permalink / raw) To: Bruce Richardson; +Cc: fengchengwen, jerinj, kevin.laatz, dev > On Fri, Sep 17, 2021 at 03:42:17PM +0000, Conor Walsh wrote: >> Add the basic device probe/remove skeleton code and initial documentation >> for new IOAT DMA driver. Maintainers update is also included in this >> patch. >> >> Signed-off-by: Conor Walsh <conor.walsh@intel.com> >> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> >> --- >> MAINTAINERS | 6 +++ >> doc/guides/dmadevs/index.rst | 2 + >> doc/guides/dmadevs/ioat.rst | 64 ++++++++++++++++++++++++ >> doc/guides/rel_notes/release_21_11.rst | 7 +-- >> drivers/dma/ioat/ioat_dmadev.c | 69 ++++++++++++++++++++++++++ >> drivers/dma/ioat/ioat_hw_defs.h | 35 +++++++++++++ >> drivers/dma/ioat/ioat_internal.h | 20 ++++++++ >> drivers/dma/ioat/meson.build | 7 +++ >> drivers/dma/ioat/version.map | 3 ++ >> drivers/dma/meson.build | 1 + >> 10 files changed, 211 insertions(+), 3 deletions(-) >> create mode 100644 doc/guides/dmadevs/ioat.rst >> create mode 100644 drivers/dma/ioat/ioat_dmadev.c >> create mode 100644 drivers/dma/ioat/ioat_hw_defs.h >> create mode 100644 drivers/dma/ioat/ioat_internal.h >> create mode 100644 drivers/dma/ioat/meson.build >> create mode 100644 drivers/dma/ioat/version.map >> >> diff --git a/MAINTAINERS b/MAINTAINERS >> index 9cb59b831d..70993d23e8 100644 >> --- a/MAINTAINERS >> +++ b/MAINTAINERS >> @@ -1209,6 +1209,12 @@ M: Kevin Laatz <kevin.laatz@intel.com> >> F: drivers/dma/idxd/ >> F: doc/guides/dmadevs/idxd.rst >> >> +Intel IOAT - EXPERIMENTAL >> +M: Bruce Richardson <bruce.richardson@intel.com> >> +M: Conor Walsh <conor.walsh@intel.com> >> +F: drivers/dma/ioat/ >> +F: doc/guides/dmadevs/ioat.rst >> + >> > Unlike the raw/ioat driver, this dmadev driver does not have a private APIs > so I'm not sure it needs the EXPERIMENTAL tag on it. I will update this in v5. <snip> >> diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst >> index c0bfd9c1ba..4d2b7bde1b 100644 >> --- a/doc/guides/rel_notes/release_21_11.rst >> +++ b/doc/guides/rel_notes/release_21_11.rst >> @@ -92,10 +92,11 @@ New Features >> * Device allocation and it's multi-process support. >> * Control and data plane functions. >> >> -* **Added IDXD dmadev driver implementation.** >> +* **Added Intel dmadev driver implementations.** >> >> - The IDXD dmadev driver provide device drivers for the Intel DSA devices. >> - This device driver can be used through the generic dmadev API. >> + The IDXD and IOAT dmadev drivers provide device drivers for Intel DSA >> + and IOAT devices. These device drivers can be used through the generic >> + dmadev API. >> > I'm not sure about merging two driver additions into a single release note > entry - one for the doc maintainers and tree committers to comment on. > > Rather than "IOAT devices" I think the official name of the hardware should > be used, and you probably should add "respectively" at the end of the first > sentence to make it clear (once you change the name) which hardware is used > by which driver. I will separate them into 2 entries and reword in v5. Thanks, Conor. ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 01/11] dma/ioat: add device probe and removal functions 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 01/11] dma/ioat: add device probe and removal functions Conor Walsh 2021-09-20 11:15 ` Bruce Richardson @ 2021-09-22 3:59 ` fengchengwen 1 sibling, 0 replies; 132+ messages in thread From: fengchengwen @ 2021-09-22 3:59 UTC (permalink / raw) To: Conor Walsh, bruce.richardson, jerinj, kevin.laatz; +Cc: dev Reviewed-by: Chengwen Feng <fengchengwen@huawei.com> On 2021/9/17 23:42, Conor Walsh wrote: > Add the basic device probe/remove skeleton code and initial documentation > for new IOAT DMA driver. Maintainers update is also included in this > patch. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> > --- > MAINTAINERS | 6 +++ > doc/guides/dmadevs/index.rst | 2 + > doc/guides/dmadevs/ioat.rst | 64 ++++++++++++++++++++++++ > doc/guides/rel_notes/release_21_11.rst | 7 +-- > drivers/dma/ioat/ioat_dmadev.c | 69 ++++++++++++++++++++++++++ > drivers/dma/ioat/ioat_hw_defs.h | 35 +++++++++++++ > drivers/dma/ioat/ioat_internal.h | 20 ++++++++ > drivers/dma/ioat/meson.build | 7 +++ > drivers/dma/ioat/version.map | 3 ++ > drivers/dma/meson.build | 1 + > 10 files changed, 211 insertions(+), 3 deletions(-) > create mode 100644 doc/guides/dmadevs/ioat.rst > create mode 100644 drivers/dma/ioat/ioat_dmadev.c > create mode 100644 drivers/dma/ioat/ioat_hw_defs.h > create mode 100644 drivers/dma/ioat/ioat_internal.h > create mode 100644 drivers/dma/ioat/meson.build > create mode 100644 drivers/dma/ioat/version.map > > diff --git a/MAINTAINERS b/MAINTAINERS > index 9cb59b831d..70993d23e8 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -1209,6 +1209,12 @@ M: Kevin Laatz <kevin.laatz@intel.com> > F: drivers/dma/idxd/ > F: doc/guides/dmadevs/idxd.rst > > +Intel IOAT - EXPERIMENTAL > +M: Bruce Richardson <bruce.richardson@intel.com> > +M: Conor Walsh <conor.walsh@intel.com> > +F: drivers/dma/ioat/ > +F: doc/guides/dmadevs/ioat.rst > + > > RegEx Drivers > ------------- > diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst > index 5d4abf880e..c59f4b5c92 100644 > --- a/doc/guides/dmadevs/index.rst > +++ b/doc/guides/dmadevs/index.rst > @@ -12,3 +12,5 @@ an application through DMA API. > :numbered: > > idxd > + ioat > + > diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst > new file mode 100644 > index 0000000000..45a2e65d70 > --- /dev/null > +++ b/doc/guides/dmadevs/ioat.rst > @@ -0,0 +1,64 @@ > +.. SPDX-License-Identifier: BSD-3-Clause > + Copyright(c) 2021 Intel Corporation. > + > +.. include:: <isonum.txt> > + > +IOAT DMA Device Driver > +======================= > + > +The ``ioat`` dmadev driver provides a poll-mode driver (PMD) for Intel\ > +|reg| QuickData Technology which is part of part of Intel\ |reg| I/O > +Acceleration Technology (`Intel I/OAT > +<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_). > +This PMD, when used on supported hardware, allows data copies, for example, > +cloning packet data, to be accelerated by IOAT hardware rather than having to > +be done by software, freeing up CPU cycles for other tasks. > + > +Hardware Requirements > +---------------------- > + > +The ``dpdk-devbind.py`` script, included with DPDK, can be used to show the > +presence of supported hardware. Running ``dpdk-devbind.py --status-dev dma`` > +will show all the DMA devices on the system, IOAT devices are included in this > +list. For Intel\ |reg| IOAT devices, the hardware will often be listed as > +"Crystal Beach DMA", or "CBDMA" or on some newer systems '0b00' due to the > +absence of pci-id database entries for them at this point. > + > +Compilation > +------------ > + > +For builds using ``meson`` and ``ninja``, the driver will be built when the > +target platform is x86-based. No additional compilation steps are necessary. > + > +Device Setup > +------------- > + > +Intel\ |reg| IOAT devices will need to be bound to a suitable DPDK-supported > +user-space IO driver such as ``vfio-pci`` in order to be used by DPDK. > + > +The ``dpdk-devbind.py`` script can be used to view the state of the devices using:: > + > + $ dpdk-devbind.py --status-dev dma > + > +The ``dpdk-devbind.py`` script can also be used to bind devices to a suitable driver. > +For example:: > + > + $ dpdk-devbind.py -b vfio-pci 00:01.0 00:01.1 > + > +Device Probing and Initialization > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +For devices bound to a suitable DPDK-supported driver (``vfio-pci``), the HW > +devices will be found as part of the device scan done at application > +initialization time without the need to pass parameters to the application. > + > +If the application does not require all the devices available an allowlist can > +be used in the same way that other DPDK devices use them. > + > +For example:: > + > + $ dpdk-test -a <b:d:f> > + > +Once probed successfully, the device will appear as a ``dmadev``, that is a > +"DMA device type" inside DPDK, and can be accessed using APIs from the > +``rte_dmadev`` library. > diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst > index c0bfd9c1ba..4d2b7bde1b 100644 > --- a/doc/guides/rel_notes/release_21_11.rst > +++ b/doc/guides/rel_notes/release_21_11.rst > @@ -92,10 +92,11 @@ New Features > * Device allocation and it's multi-process support. > * Control and data plane functions. > > -* **Added IDXD dmadev driver implementation.** > +* **Added Intel dmadev driver implementations.** > > - The IDXD dmadev driver provide device drivers for the Intel DSA devices. > - This device driver can be used through the generic dmadev API. > + The IDXD and IOAT dmadev drivers provide device drivers for Intel DSA > + and IOAT devices. These device drivers can be used through the generic > + dmadev API. > > > Removed Items > diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c > new file mode 100644 > index 0000000000..f3491d45b1 > --- /dev/null > +++ b/drivers/dma/ioat/ioat_dmadev.c > @@ -0,0 +1,69 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright(c) 2021 Intel Corporation > + */ > + > +#include <rte_bus_pci.h> > +#include <rte_dmadev_pmd.h> > + > +#include "ioat_internal.h" > + > +static struct rte_pci_driver ioat_pmd_drv; > + > +RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); > + > +#define IOAT_PMD_NAME dmadev_ioat > +#define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) > + > +/* Probe DMA device. */ > +static int > +ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) > +{ > + char name[32]; > + > + rte_pci_device_name(&dev->addr, name, sizeof(name)); > + IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); > + > + dev->device.driver = &drv->driver; > + return 0; > +} > + > +/* Remove DMA device. */ > +static int > +ioat_dmadev_remove(struct rte_pci_device *dev) > +{ > + char name[32]; > + > + rte_pci_device_name(&dev->addr, name, sizeof(name)); > + > + IOAT_PMD_INFO("Closing %s on NUMA node %d", > + name, dev->device.numa_node); > + > + return 0; > +} > + > +static const struct rte_pci_id pci_id_ioat_map[] = { > + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_SKX) }, > + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX0) }, > + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX1) }, > + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX2) }, > + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX3) }, > + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX4) }, > + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX5) }, > + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX6) }, > + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX7) }, > + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXE) }, > + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXF) }, > + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_ICX) }, > + { .vendor_id = 0, /* sentinel */ }, > +}; > + > +static struct rte_pci_driver ioat_pmd_drv = { > + .id_table = pci_id_ioat_map, > + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC, > + .probe = ioat_dmadev_probe, > + .remove = ioat_dmadev_remove, > +}; > + > +RTE_PMD_REGISTER_PCI(IOAT_PMD_NAME, ioat_pmd_drv); > +RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_NAME, pci_id_ioat_map); > +RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_NAME, "* igb_uio | uio_pci_generic | vfio-pci"); > diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h > new file mode 100644 > index 0000000000..eeabba41ef > --- /dev/null > +++ b/drivers/dma/ioat/ioat_hw_defs.h > @@ -0,0 +1,35 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright(c) 2021 Intel Corporation > + */ > + > +#ifndef IOAT_HW_DEFS_H > +#define IOAT_HW_DEFS_H > + > +#ifdef __cplusplus > +extern "C" { > +#endif > + > +#include <stdint.h> > + > +#define IOAT_VER_3_0 0x30 > +#define IOAT_VER_3_3 0x33 > + > +#define IOAT_VENDOR_ID 0x8086 > +#define IOAT_DEVICE_ID_SKX 0x2021 > +#define IOAT_DEVICE_ID_BDX0 0x6f20 > +#define IOAT_DEVICE_ID_BDX1 0x6f21 > +#define IOAT_DEVICE_ID_BDX2 0x6f22 > +#define IOAT_DEVICE_ID_BDX3 0x6f23 > +#define IOAT_DEVICE_ID_BDX4 0x6f24 > +#define IOAT_DEVICE_ID_BDX5 0x6f25 > +#define IOAT_DEVICE_ID_BDX6 0x6f26 > +#define IOAT_DEVICE_ID_BDX7 0x6f27 > +#define IOAT_DEVICE_ID_BDXE 0x6f2E > +#define IOAT_DEVICE_ID_BDXF 0x6f2F > +#define IOAT_DEVICE_ID_ICX 0x0b00 > + > +#ifdef __cplusplus > +} > +#endif > + > +#endif /* IOAT_HW_DEFS_H */ > diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h > new file mode 100644 > index 0000000000..f1ec12a919 > --- /dev/null > +++ b/drivers/dma/ioat/ioat_internal.h > @@ -0,0 +1,20 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright 2021 Intel Corporation > + */ > + > +#ifndef _IOAT_INTERNAL_H_ > +#define _IOAT_INTERNAL_H_ > + > +#include "ioat_hw_defs.h" > + > +extern int ioat_pmd_logtype; > + > +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ > + ioat_pmd_logtype, "IOAT: %s(): " fmt "\n", __func__, ##args) > + > +#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) > +#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) > +#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) > +#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) > + > +#endif /* _IOAT_INTERNAL_H_ */ > diff --git a/drivers/dma/ioat/meson.build b/drivers/dma/ioat/meson.build > new file mode 100644 > index 0000000000..d67fac96fb > --- /dev/null > +++ b/drivers/dma/ioat/meson.build > @@ -0,0 +1,7 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright 2021 Intel Corporation > + > +build = dpdk_conf.has('RTE_ARCH_X86') > +reason = 'only supported on x86' > +sources = files('ioat_dmadev.c') > +deps += ['bus_pci', 'dmadev'] > diff --git a/drivers/dma/ioat/version.map b/drivers/dma/ioat/version.map > new file mode 100644 > index 0000000000..c2e0723b4c > --- /dev/null > +++ b/drivers/dma/ioat/version.map > @@ -0,0 +1,3 @@ > +DPDK_22 { > + local: *; > +}; > diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build > index 411be7a240..a69418ce9b 100644 > --- a/drivers/dma/meson.build > +++ b/drivers/dma/meson.build > @@ -3,6 +3,7 @@ > > drivers = [ > 'idxd', > + 'ioat', > 'skeleton', > ] > std_deps = ['dmadev'] > ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v4 02/11] dma/ioat: create dmadev instances on PCI probe 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 01/11] dma/ioat: add device probe and removal functions Conor Walsh @ 2021-09-17 15:42 ` Conor Walsh 2021-09-20 13:31 ` Bruce Richardson 2021-09-22 8:04 ` fengchengwen 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 03/11] dma/ioat: add datapath structures Conor Walsh ` (8 subsequent siblings) 10 siblings, 2 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-17 15:42 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh When a suitable device is found during the PCI probe, create a dmadev instance for each channel. Internal structures and HW definitions required for device creation are also included. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 119 ++++++++++++++++++++++++++++++- drivers/dma/ioat/ioat_hw_defs.h | 45 ++++++++++++ drivers/dma/ioat/ioat_internal.h | 24 +++++++ 3 files changed, 186 insertions(+), 2 deletions(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index f3491d45b1..b815d30bcf 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -4,6 +4,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> +#include <rte_malloc.h> #include "ioat_internal.h" @@ -14,6 +15,120 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Create a DMA device. */ +static int +ioat_dmadev_create(const char *name, struct rte_pci_device *dev) +{ + static const struct rte_dma_dev_ops ioat_dmadev_ops = { }; + + struct rte_dma_dev *dmadev = NULL; + struct ioat_dmadev *ioat = NULL; + int retry = 0; + + if (!name) { + IOAT_PMD_ERR("Invalid name of the device!"); + return -EINVAL; + } + + /* Allocate device structure. */ + dmadev = rte_dma_pmd_allocate(name, dev->device.numa_node, + sizeof(dmadev->dev_private)); + if (dmadev == NULL) { + IOAT_PMD_ERR("Unable to allocate dma device"); + return -ENOMEM; + } + + dmadev->device = &dev->device; + + dmadev->data->dev_private = rte_malloc_socket(NULL, sizeof(*ioat), + 0, dmadev->device->numa_node); + dmadev->dev_private = dmadev->data->dev_private; + + dmadev->dev_ops = &ioat_dmadev_ops; + + ioat = dmadev->data->dev_private; + ioat->dmadev = dmadev; + ioat->regs = dev->mem_resource[0].addr; + ioat->doorbell = &ioat->regs->dmacount; + ioat->qcfg.nb_desc = 0; + ioat->desc_ring = NULL; + + /* Do device initialization - reset and set error behaviour. */ + if (ioat->regs->chancnt != 1) + IOAT_PMD_WARN("%s: Channel count == %d\n", __func__, + ioat->regs->chancnt); + + /* Locked by someone else. */ + if (ioat->regs->chanctrl & IOAT_CHANCTRL_CHANNEL_IN_USE) { + IOAT_PMD_WARN("%s: Channel appears locked\n", __func__); + ioat->regs->chanctrl = 0; + } + + /* clear any previous errors */ + if (ioat->regs->chanerr != 0) { + uint32_t val = ioat->regs->chanerr; + ioat->regs->chanerr = val; + } + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + rte_delay_ms(1); + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + rte_delay_ms(1); + while (ioat->regs->chancmd & IOAT_CHANCMD_RESET) { + ioat->regs->chainaddr = 0; + rte_delay_ms(1); + if (++retry >= 200) { + IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=%#"PRIx8 + ", CHANSTS=%#"PRIx64", CHANERR=%#"PRIx32"\n", + __func__, + ioat->regs->chancmd, + ioat->regs->chansts, + ioat->regs->chanerr); + return -EIO; + } + } + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + + return 0; + +} + +/* Destroy a DMA device. */ +static int +ioat_dmadev_destroy(const char *name) +{ + struct rte_dma_dev *dev; + struct ioat_dmadev *ioat; + int ret; + + if (!name) { + IOAT_PMD_ERR("Invalid device name"); + return -EINVAL; + } + + dev = &rte_dma_devices[rte_dma_get_dev_id(name)]; + if (!dev) { + IOAT_PMD_ERR("Invalid device name (%s)", name); + return -EINVAL; + } + + ioat = dev->dev_private; + if (!ioat) { + IOAT_PMD_ERR("Error getting dev_private"); + return -EINVAL; + } + + dev->dev_private = NULL; + rte_free(ioat->desc_ring); + + ret = rte_dma_pmd_release(name); + if (ret) + IOAT_PMD_DEBUG("Device cleanup failed"); + + return 0; +} + /* Probe DMA device. */ static int ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) @@ -24,7 +139,7 @@ ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); dev->device.driver = &drv->driver; - return 0; + return ioat_dmadev_create(name, dev); } /* Remove DMA device. */ @@ -38,7 +153,7 @@ ioat_dmadev_remove(struct rte_pci_device *dev) IOAT_PMD_INFO("Closing %s on NUMA node %d", name, dev->device.numa_node); - return 0; + return ioat_dmadev_destroy(name); } static const struct rte_pci_id pci_id_ioat_map[] = { diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index eeabba41ef..73bdf548b3 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -11,6 +11,8 @@ extern "C" { #include <stdint.h> +#define IOAT_PCI_CHANERR_INT_OFFSET 0x180 + #define IOAT_VER_3_0 0x30 #define IOAT_VER_3_3 0x33 @@ -28,6 +30,49 @@ extern "C" { #define IOAT_DEVICE_ID_BDXF 0x6f2F #define IOAT_DEVICE_ID_ICX 0x0b00 +#define IOAT_COMP_UPDATE_SHIFT 3 +#define IOAT_CMD_OP_SHIFT 24 + +/* DMA Channel Registers */ +#define IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK 0xF000 +#define IOAT_CHANCTRL_COMPL_DCA_EN 0x0200 +#define IOAT_CHANCTRL_CHANNEL_IN_USE 0x0100 +#define IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL 0x0020 +#define IOAT_CHANCTRL_ERR_INT_EN 0x0010 +#define IOAT_CHANCTRL_ANY_ERR_ABORT_EN 0x0008 +#define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 +#define IOAT_CHANCTRL_INT_REARM 0x0001 + +struct ioat_registers { + uint8_t chancnt; + uint8_t xfercap; + uint8_t genctrl; + uint8_t intrctrl; + uint32_t attnstatus; + uint8_t cbver; /* 0x08 */ + uint8_t reserved4[0x3]; /* 0x09 */ + uint16_t intrdelay; /* 0x0C */ + uint16_t cs_status; /* 0x0E */ + uint32_t dmacapability; /* 0x10 */ + uint8_t reserved5[0x6C]; /* 0x14 */ + uint16_t chanctrl; /* 0x80 */ + uint8_t reserved6[0x2]; /* 0x82 */ + uint8_t chancmd; /* 0x84 */ + uint8_t reserved3[1]; /* 0x85 */ + uint16_t dmacount; /* 0x86 */ + uint64_t chansts; /* 0x88 */ + uint64_t chainaddr; /* 0x90 */ + uint64_t chancmp; /* 0x98 */ + uint8_t reserved2[0x8]; /* 0xA0 */ + uint32_t chanerr; /* 0xA8 */ + uint32_t chanerrmask; /* 0xAC */ +} __rte_packed; + +#define IOAT_CHANCMD_RESET 0x20 +#define IOAT_CHANCMD_SUSPEND 0x04 + +#define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ + #ifdef __cplusplus } #endif diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h index f1ec12a919..a4e323f360 100644 --- a/drivers/dma/ioat/ioat_internal.h +++ b/drivers/dma/ioat/ioat_internal.h @@ -7,6 +7,30 @@ #include "ioat_hw_defs.h" +struct ioat_dmadev { + struct rte_dma_dev *dmadev; + struct rte_dma_vchan_conf qcfg; + struct rte_dma_stats stats; + + volatile uint16_t *doorbell __rte_cache_aligned; + phys_addr_t status_addr; + phys_addr_t ring_addr; + + struct ioat_dma_hw_desc *desc_ring; + + unsigned short next_read; + unsigned short next_write; + unsigned short last_write; /* Used to compute submitted count. */ + unsigned short offset; /* Used after a device recovery when counts -> 0. */ + unsigned int failure; /* Used to store chanerr for error handling. */ + + /* To report completions, the device will write status back here. */ + volatile uint64_t status __rte_cache_aligned; + + /* Pointer to the register bar. */ + volatile struct ioat_registers *regs; +}; + extern int ioat_pmd_logtype; #define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 02/11] dma/ioat: create dmadev instances on PCI probe 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 02/11] dma/ioat: create dmadev instances on PCI probe Conor Walsh @ 2021-09-20 13:31 ` Bruce Richardson 2021-09-21 16:25 ` Conor Walsh 2021-09-22 8:04 ` fengchengwen 1 sibling, 1 reply; 132+ messages in thread From: Bruce Richardson @ 2021-09-20 13:31 UTC (permalink / raw) To: Conor Walsh; +Cc: fengchengwen, jerinj, kevin.laatz, dev On Fri, Sep 17, 2021 at 03:42:18PM +0000, Conor Walsh wrote: > When a suitable device is found during the PCI probe, create a dmadev > instance for each channel. Internal structures and HW definitions required > for device creation are also included. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> > --- > drivers/dma/ioat/ioat_dmadev.c | 119 ++++++++++++++++++++++++++++++- > drivers/dma/ioat/ioat_hw_defs.h | 45 ++++++++++++ > drivers/dma/ioat/ioat_internal.h | 24 +++++++ > 3 files changed, 186 insertions(+), 2 deletions(-) > > diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c > index f3491d45b1..b815d30bcf 100644 > --- a/drivers/dma/ioat/ioat_dmadev.c > +++ b/drivers/dma/ioat/ioat_dmadev.c > @@ -4,6 +4,7 @@ > <snip> > +/* Destroy a DMA device. */ > +static int > +ioat_dmadev_destroy(const char *name) > +{ > + struct rte_dma_dev *dev; > + struct ioat_dmadev *ioat; > + int ret; > + > + if (!name) { > + IOAT_PMD_ERR("Invalid device name"); > + return -EINVAL; > + } > + > + dev = &rte_dma_devices[rte_dma_get_dev_id(name)]; > + if (!dev) { > + IOAT_PMD_ERR("Invalid device name (%s)", name); > + return -EINVAL; > + } > + I think you need to independently check the return value from rte_dma_get_dev_id, rather than assuming when it returns an error value the resultant index location will hold a null pointer. > + ioat = dev->dev_private; > + if (!ioat) { > + IOAT_PMD_ERR("Error getting dev_private"); > + return -EINVAL; > + } > + > + dev->dev_private = NULL; > + rte_free(ioat->desc_ring); > + > + ret = rte_dma_pmd_release(name); The rte_dma_pmd_allocate function reserves memory for the private data, so the release function should free that memory too. However, you have assigned private_data to NULL just above, so that probably won't work. > + if (ret) > + IOAT_PMD_DEBUG("Device cleanup failed"); > + > + return 0; > +} > + <snip> ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 02/11] dma/ioat: create dmadev instances on PCI probe 2021-09-20 13:31 ` Bruce Richardson @ 2021-09-21 16:25 ` Conor Walsh 0 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-21 16:25 UTC (permalink / raw) To: Bruce Richardson; +Cc: fengchengwen, jerinj, kevin.laatz, dev > On Fri, Sep 17, 2021 at 03:42:18PM +0000, Conor Walsh wrote: >> When a suitable device is found during the PCI probe, create a dmadev >> instance for each channel. Internal structures and HW definitions required >> for device creation are also included. >> >> Signed-off-by: Conor Walsh <conor.walsh@intel.com> >> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> >> --- >> drivers/dma/ioat/ioat_dmadev.c | 119 ++++++++++++++++++++++++++++++- >> drivers/dma/ioat/ioat_hw_defs.h | 45 ++++++++++++ >> drivers/dma/ioat/ioat_internal.h | 24 +++++++ >> 3 files changed, 186 insertions(+), 2 deletions(-) >> >> diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c >> index f3491d45b1..b815d30bcf 100644 >> --- a/drivers/dma/ioat/ioat_dmadev.c >> +++ b/drivers/dma/ioat/ioat_dmadev.c >> @@ -4,6 +4,7 @@ >> > <snip> > >> +/* Destroy a DMA device. */ >> +static int >> +ioat_dmadev_destroy(const char *name) >> +{ >> + struct rte_dma_dev *dev; >> + struct ioat_dmadev *ioat; >> + int ret; >> + >> + if (!name) { >> + IOAT_PMD_ERR("Invalid device name"); >> + return -EINVAL; >> + } >> + >> + dev = &rte_dma_devices[rte_dma_get_dev_id(name)]; >> + if (!dev) { >> + IOAT_PMD_ERR("Invalid device name (%s)", name); >> + return -EINVAL; >> + } >> + > I think you need to independently check the return value from > rte_dma_get_dev_id, rather than assuming when it returns an error value the > resultant index location will hold a null pointer. I will assign rte_dma_get_dev_id(name) to a variable and check it before use in v5. > >> + ioat = dev->dev_private; >> + if (!ioat) { >> + IOAT_PMD_ERR("Error getting dev_private"); >> + return -EINVAL; >> + } >> + >> + dev->dev_private = NULL; >> + rte_free(ioat->desc_ring); >> + >> + ret = rte_dma_pmd_release(name); > The rte_dma_pmd_allocate function reserves memory for the private data, so > the release function should free that memory too. However, you have > assigned private_data to NULL just above, so that probably won't work. I think I will actually remove the "dev->dev_private = NULL" in v5 as this is handled by the lib. Thanks, Conor. >> + if (ret) >> + IOAT_PMD_DEBUG("Device cleanup failed"); >> + >> + return 0; >> +} >> + > <snip> ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 02/11] dma/ioat: create dmadev instances on PCI probe 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 02/11] dma/ioat: create dmadev instances on PCI probe Conor Walsh 2021-09-20 13:31 ` Bruce Richardson @ 2021-09-22 8:04 ` fengchengwen 2021-09-22 16:40 ` Conor Walsh 1 sibling, 1 reply; 132+ messages in thread From: fengchengwen @ 2021-09-22 8:04 UTC (permalink / raw) To: Conor Walsh, bruce.richardson, jerinj, kevin.laatz; +Cc: dev On 2021/9/17 23:42, Conor Walsh wrote: > When a suitable device is found during the PCI probe, create a dmadev > instance for each channel. Internal structures and HW definitions required > for device creation are also included. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> > --- > drivers/dma/ioat/ioat_dmadev.c | 119 ++++++++++++++++++++++++++++++- > drivers/dma/ioat/ioat_hw_defs.h | 45 ++++++++++++ > drivers/dma/ioat/ioat_internal.h | 24 +++++++ > 3 files changed, 186 insertions(+), 2 deletions(-) > > diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c > index f3491d45b1..b815d30bcf 100644 > --- a/drivers/dma/ioat/ioat_dmadev.c > +++ b/drivers/dma/ioat/ioat_dmadev.c > @@ -4,6 +4,7 @@ > > #include <rte_bus_pci.h> > #include <rte_dmadev_pmd.h> > +#include <rte_malloc.h> > > #include "ioat_internal.h" > > @@ -14,6 +15,120 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); > #define IOAT_PMD_NAME dmadev_ioat > #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) > > +/* Create a DMA device. */ > +static int > +ioat_dmadev_create(const char *name, struct rte_pci_device *dev) > +{ > + static const struct rte_dma_dev_ops ioat_dmadev_ops = { }; > + > + struct rte_dma_dev *dmadev = NULL; > + struct ioat_dmadev *ioat = NULL; > + int retry = 0; > + > + if (!name) { > + IOAT_PMD_ERR("Invalid name of the device!"); > + return -EINVAL; > + } > + > + /* Allocate device structure. */ > + dmadev = rte_dma_pmd_allocate(name, dev->device.numa_node, > + sizeof(dmadev->dev_private)); > + if (dmadev == NULL) { > + IOAT_PMD_ERR("Unable to allocate dma device"); > + return -ENOMEM; > + } > + > + dmadev->device = &dev->device; > + > + dmadev->data->dev_private = rte_malloc_socket(NULL, sizeof(*ioat), > + 0, dmadev->device->numa_node); > + dmadev->dev_private = dmadev->data->dev_private; The dmalib will malloc dev_private, so please invoke like: dmadev = rte_dma_pmd_allocate(name, dev->device.numa_node, sizeof(*ioat)); > + > + dmadev->dev_ops = &ioat_dmadev_ops; > + > + ioat = dmadev->data->dev_private; > + ioat->dmadev = dmadev; > + ioat->regs = dev->mem_resource[0].addr; > + ioat->doorbell = &ioat->regs->dmacount; > + ioat->qcfg.nb_desc = 0; > + ioat->desc_ring = NULL; > + > + /* Do device initialization - reset and set error behaviour. */ > + if (ioat->regs->chancnt != 1) > + IOAT_PMD_WARN("%s: Channel count == %d\n", __func__, > + ioat->regs->chancnt); > + > + /* Locked by someone else. */ > + if (ioat->regs->chanctrl & IOAT_CHANCTRL_CHANNEL_IN_USE) { > + IOAT_PMD_WARN("%s: Channel appears locked\n", __func__); > + ioat->regs->chanctrl = 0; > + } > + > + /* clear any previous errors */ > + if (ioat->regs->chanerr != 0) { > + uint32_t val = ioat->regs->chanerr; > + ioat->regs->chanerr = val; > + } > + > + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; > + rte_delay_ms(1); > + ioat->regs->chancmd = IOAT_CHANCMD_RESET; > + rte_delay_ms(1); > + while (ioat->regs->chancmd & IOAT_CHANCMD_RESET) { > + ioat->regs->chainaddr = 0; > + rte_delay_ms(1); > + if (++retry >= 200) { > + IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=%#"PRIx8 > + ", CHANSTS=%#"PRIx64", CHANERR=%#"PRIx32"\n", > + __func__, > + ioat->regs->chancmd, > + ioat->regs->chansts, > + ioat->regs->chanerr); please release the dmadev. > + return -EIO; > + } > + } > + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | > + IOAT_CHANCTRL_ERR_COMPLETION_EN; > + please change dmadev state to RTE_DMA_DEV_READY. > + return 0; > + > +} > + > +/* Destroy a DMA device. */ > +static int > +ioat_dmadev_destroy(const char *name) > +{ > + struct rte_dma_dev *dev; > + struct ioat_dmadev *ioat; > + int ret; > + > + if (!name) { > + IOAT_PMD_ERR("Invalid device name"); > + return -EINVAL; > + } > + > + dev = &rte_dma_devices[rte_dma_get_dev_id(name)]; > + if (!dev) { > + IOAT_PMD_ERR("Invalid device name (%s)", name); > + return -EINVAL; > + } > + > + ioat = dev->dev_private; > + if (!ioat) { > + IOAT_PMD_ERR("Error getting dev_private"); > + return -EINVAL; > + } > + > + dev->dev_private = NULL; > + rte_free(ioat->desc_ring); > + > + ret = rte_dma_pmd_release(name); > + if (ret) > + IOAT_PMD_DEBUG("Device cleanup failed"); driver only need call rte_dma_pmd_relese, and the dev_private will freed in dmalib. > + > + return 0; > +} > + > /* Probe DMA device. */ > static int > ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) > @@ -24,7 +139,7 @@ ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) > IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); > > dev->device.driver = &drv->driver; > - return 0; > + return ioat_dmadev_create(name, dev); > } > > /* Remove DMA device. */ > @@ -38,7 +153,7 @@ ioat_dmadev_remove(struct rte_pci_device *dev) > IOAT_PMD_INFO("Closing %s on NUMA node %d", > name, dev->device.numa_node); > > - return 0; > + return ioat_dmadev_destroy(name); > } > > static const struct rte_pci_id pci_id_ioat_map[] = { > diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h > index eeabba41ef..73bdf548b3 100644 > --- a/drivers/dma/ioat/ioat_hw_defs.h > +++ b/drivers/dma/ioat/ioat_hw_defs.h > @@ -11,6 +11,8 @@ extern "C" { > > #include <stdint.h> > > +#define IOAT_PCI_CHANERR_INT_OFFSET 0x180 > + > #define IOAT_VER_3_0 0x30 > #define IOAT_VER_3_3 0x33 > > @@ -28,6 +30,49 @@ extern "C" { > #define IOAT_DEVICE_ID_BDXF 0x6f2F > #define IOAT_DEVICE_ID_ICX 0x0b00 > > +#define IOAT_COMP_UPDATE_SHIFT 3 > +#define IOAT_CMD_OP_SHIFT 24 > + > +/* DMA Channel Registers */ > +#define IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK 0xF000 > +#define IOAT_CHANCTRL_COMPL_DCA_EN 0x0200 > +#define IOAT_CHANCTRL_CHANNEL_IN_USE 0x0100 > +#define IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL 0x0020 > +#define IOAT_CHANCTRL_ERR_INT_EN 0x0010 > +#define IOAT_CHANCTRL_ANY_ERR_ABORT_EN 0x0008 > +#define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 > +#define IOAT_CHANCTRL_INT_REARM 0x0001 > + > +struct ioat_registers { > + uint8_t chancnt; > + uint8_t xfercap; > + uint8_t genctrl; > + uint8_t intrctrl; > + uint32_t attnstatus; > + uint8_t cbver; /* 0x08 */ > + uint8_t reserved4[0x3]; /* 0x09 */ > + uint16_t intrdelay; /* 0x0C */ > + uint16_t cs_status; /* 0x0E */ > + uint32_t dmacapability; /* 0x10 */ > + uint8_t reserved5[0x6C]; /* 0x14 */ > + uint16_t chanctrl; /* 0x80 */ > + uint8_t reserved6[0x2]; /* 0x82 */ > + uint8_t chancmd; /* 0x84 */ > + uint8_t reserved3[1]; /* 0x85 */ > + uint16_t dmacount; /* 0x86 */ > + uint64_t chansts; /* 0x88 */ > + uint64_t chainaddr; /* 0x90 */ > + uint64_t chancmp; /* 0x98 */ > + uint8_t reserved2[0x8]; /* 0xA0 */ > + uint32_t chanerr; /* 0xA8 */ > + uint32_t chanerrmask; /* 0xAC */ > +} __rte_packed; > + > +#define IOAT_CHANCMD_RESET 0x20 > +#define IOAT_CHANCMD_SUSPEND 0x04 > + > +#define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ > + > #ifdef __cplusplus > } > #endif > diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h > index f1ec12a919..a4e323f360 100644 > --- a/drivers/dma/ioat/ioat_internal.h > +++ b/drivers/dma/ioat/ioat_internal.h > @@ -7,6 +7,30 @@ > > #include "ioat_hw_defs.h" > > +struct ioat_dmadev { > + struct rte_dma_dev *dmadev; maybe the rte_dma_dev_data more appropriate if support multi-process. > + struct rte_dma_vchan_conf qcfg; > + struct rte_dma_stats stats; > + > + volatile uint16_t *doorbell __rte_cache_aligned; > + phys_addr_t status_addr; > + phys_addr_t ring_addr; > + > + struct ioat_dma_hw_desc *desc_ring; > + > + unsigned short next_read; > + unsigned short next_write; > + unsigned short last_write; /* Used to compute submitted count. */ > + unsigned short offset; /* Used after a device recovery when counts -> 0. */ > + unsigned int failure; /* Used to store chanerr for error handling. */ > + > + /* To report completions, the device will write status back here. */ > + volatile uint64_t status __rte_cache_aligned; > + > + /* Pointer to the register bar. */ > + volatile struct ioat_registers *regs; > +}; > + > extern int ioat_pmd_logtype; > > #define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ > ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 02/11] dma/ioat: create dmadev instances on PCI probe 2021-09-22 8:04 ` fengchengwen @ 2021-09-22 16:40 ` Conor Walsh 0 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-22 16:40 UTC (permalink / raw) To: fengchengwen, bruce.richardson, jerinj, kevin.laatz; +Cc: dev On 22/09/2021 09:04, fengchengwen wrote: > On 2021/9/17 23:42, Conor Walsh wrote: >> When a suitable device is found during the PCI probe, create a dmadev >> instance for each channel. Internal structures and HW definitions required >> for device creation are also included. >> >> Signed-off-by: Conor Walsh <conor.walsh@intel.com> >> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> >> --- >> drivers/dma/ioat/ioat_dmadev.c | 119 ++++++++++++++++++++++++++++++- >> drivers/dma/ioat/ioat_hw_defs.h | 45 ++++++++++++ >> drivers/dma/ioat/ioat_internal.h | 24 +++++++ >> 3 files changed, 186 insertions(+), 2 deletions(-) >> >> diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c >> index f3491d45b1..b815d30bcf 100644 >> --- a/drivers/dma/ioat/ioat_dmadev.c >> +++ b/drivers/dma/ioat/ioat_dmadev.c >> @@ -4,6 +4,7 @@ >> >> #include <rte_bus_pci.h> >> #include <rte_dmadev_pmd.h> >> +#include <rte_malloc.h> >> >> #include "ioat_internal.h" >> >> @@ -14,6 +15,120 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); >> #define IOAT_PMD_NAME dmadev_ioat >> #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) >> >> +/* Create a DMA device. */ >> +static int >> +ioat_dmadev_create(const char *name, struct rte_pci_device *dev) >> +{ >> + static const struct rte_dma_dev_ops ioat_dmadev_ops = { }; >> + >> + struct rte_dma_dev *dmadev = NULL; >> + struct ioat_dmadev *ioat = NULL; >> + int retry = 0; >> + >> + if (!name) { >> + IOAT_PMD_ERR("Invalid name of the device!"); >> + return -EINVAL; >> + } >> + >> + /* Allocate device structure. */ >> + dmadev = rte_dma_pmd_allocate(name, dev->device.numa_node, >> + sizeof(dmadev->dev_private)); >> + if (dmadev == NULL) { >> + IOAT_PMD_ERR("Unable to allocate dma device"); >> + return -ENOMEM; >> + } >> + >> + dmadev->device = &dev->device; >> + >> + dmadev->data->dev_private = rte_malloc_socket(NULL, sizeof(*ioat), >> + 0, dmadev->device->numa_node); >> + dmadev->dev_private = dmadev->data->dev_private; > The dmalib will malloc dev_private, so please invoke like: > dmadev = rte_dma_pmd_allocate(name, dev->device.numa_node, sizeof(*ioat)); I will update this in v5. >> + dmadev->dev_ops = &ioat_dmadev_ops; >> + >> + ioat = dmadev->data->dev_private; >> + ioat->dmadev = dmadev; >> + ioat->regs = dev->mem_resource[0].addr; >> + ioat->doorbell = &ioat->regs->dmacount; >> + ioat->qcfg.nb_desc = 0; >> + ioat->desc_ring = NULL; >> + >> + /* Do device initialization - reset and set error behaviour. */ >> + if (ioat->regs->chancnt != 1) >> + IOAT_PMD_WARN("%s: Channel count == %d\n", __func__, >> + ioat->regs->chancnt); >> + >> + /* Locked by someone else. */ >> + if (ioat->regs->chanctrl & IOAT_CHANCTRL_CHANNEL_IN_USE) { >> + IOAT_PMD_WARN("%s: Channel appears locked\n", __func__); >> + ioat->regs->chanctrl = 0; >> + } >> + >> + /* clear any previous errors */ >> + if (ioat->regs->chanerr != 0) { >> + uint32_t val = ioat->regs->chanerr; >> + ioat->regs->chanerr = val; >> + } >> + >> + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; >> + rte_delay_ms(1); >> + ioat->regs->chancmd = IOAT_CHANCMD_RESET; >> + rte_delay_ms(1); >> + while (ioat->regs->chancmd & IOAT_CHANCMD_RESET) { >> + ioat->regs->chainaddr = 0; >> + rte_delay_ms(1); >> + if (++retry >= 200) { >> + IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=%#"PRIx8 >> + ", CHANSTS=%#"PRIx64", CHANERR=%#"PRIx32"\n", >> + __func__, >> + ioat->regs->chancmd, >> + ioat->regs->chansts, >> + ioat->regs->chanerr); > please release the dmadev. Will add in v5. >> + return -EIO; >> + } >> + } >> + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | >> + IOAT_CHANCTRL_ERR_COMPLETION_EN; >> + > please change dmadev state to RTE_DMA_DEV_READY. I will add this in v5. >> + return 0; >> + >> +} >> + >> +/* Destroy a DMA device. */ >> +static int >> +ioat_dmadev_destroy(const char *name) >> +{ >> + struct rte_dma_dev *dev; >> + struct ioat_dmadev *ioat; >> + int ret; >> + >> + if (!name) { >> + IOAT_PMD_ERR("Invalid device name"); >> + return -EINVAL; >> + } >> + >> + dev = &rte_dma_devices[rte_dma_get_dev_id(name)]; >> + if (!dev) { >> + IOAT_PMD_ERR("Invalid device name (%s)", name); >> + return -EINVAL; >> + } >> + >> + ioat = dev->dev_private; >> + if (!ioat) { >> + IOAT_PMD_ERR("Error getting dev_private"); >> + return -EINVAL; >> + } >> + >> + dev->dev_private = NULL; >> + rte_free(ioat->desc_ring); >> + >> + ret = rte_dma_pmd_release(name); >> + if (ret) >> + IOAT_PMD_DEBUG("Device cleanup failed"); > driver only need call rte_dma_pmd_relese, and the dev_private will freed > in dmalib. I have removed the "dev->dev_private = NULL;" line for v5, that was probably stopping the free in the lib. > >> + return 0; >> +} >> + >> /* Probe DMA device. */ >> static int >> ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) >> @@ -24,7 +139,7 @@ ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) >> IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); >> >> dev->device.driver = &drv->driver; >> - return 0; >> + return ioat_dmadev_create(name, dev); >> } >> >> /* Remove DMA device. */ >> @@ -38,7 +153,7 @@ ioat_dmadev_remove(struct rte_pci_device *dev) >> IOAT_PMD_INFO("Closing %s on NUMA node %d", >> name, dev->device.numa_node); >> >> - return 0; >> + return ioat_dmadev_destroy(name); >> } >> >> static const struct rte_pci_id pci_id_ioat_map[] = { >> diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h >> index eeabba41ef..73bdf548b3 100644 >> --- a/drivers/dma/ioat/ioat_hw_defs.h >> +++ b/drivers/dma/ioat/ioat_hw_defs.h >> @@ -11,6 +11,8 @@ extern "C" { >> >> #include <stdint.h> >> >> +#define IOAT_PCI_CHANERR_INT_OFFSET 0x180 >> + >> #define IOAT_VER_3_0 0x30 >> #define IOAT_VER_3_3 0x33 >> >> @@ -28,6 +30,49 @@ extern "C" { >> #define IOAT_DEVICE_ID_BDXF 0x6f2F >> #define IOAT_DEVICE_ID_ICX 0x0b00 >> >> +#define IOAT_COMP_UPDATE_SHIFT 3 >> +#define IOAT_CMD_OP_SHIFT 24 >> + >> +/* DMA Channel Registers */ >> +#define IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK 0xF000 >> +#define IOAT_CHANCTRL_COMPL_DCA_EN 0x0200 >> +#define IOAT_CHANCTRL_CHANNEL_IN_USE 0x0100 >> +#define IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL 0x0020 >> +#define IOAT_CHANCTRL_ERR_INT_EN 0x0010 >> +#define IOAT_CHANCTRL_ANY_ERR_ABORT_EN 0x0008 >> +#define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 >> +#define IOAT_CHANCTRL_INT_REARM 0x0001 >> + >> +struct ioat_registers { >> + uint8_t chancnt; >> + uint8_t xfercap; >> + uint8_t genctrl; >> + uint8_t intrctrl; >> + uint32_t attnstatus; >> + uint8_t cbver; /* 0x08 */ >> + uint8_t reserved4[0x3]; /* 0x09 */ >> + uint16_t intrdelay; /* 0x0C */ >> + uint16_t cs_status; /* 0x0E */ >> + uint32_t dmacapability; /* 0x10 */ >> + uint8_t reserved5[0x6C]; /* 0x14 */ >> + uint16_t chanctrl; /* 0x80 */ >> + uint8_t reserved6[0x2]; /* 0x82 */ >> + uint8_t chancmd; /* 0x84 */ >> + uint8_t reserved3[1]; /* 0x85 */ >> + uint16_t dmacount; /* 0x86 */ >> + uint64_t chansts; /* 0x88 */ >> + uint64_t chainaddr; /* 0x90 */ >> + uint64_t chancmp; /* 0x98 */ >> + uint8_t reserved2[0x8]; /* 0xA0 */ >> + uint32_t chanerr; /* 0xA8 */ >> + uint32_t chanerrmask; /* 0xAC */ >> +} __rte_packed; >> + >> +#define IOAT_CHANCMD_RESET 0x20 >> +#define IOAT_CHANCMD_SUSPEND 0x04 >> + >> +#define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ >> + >> #ifdef __cplusplus >> } >> #endif >> diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h >> index f1ec12a919..a4e323f360 100644 >> --- a/drivers/dma/ioat/ioat_internal.h >> +++ b/drivers/dma/ioat/ioat_internal.h >> @@ -7,6 +7,30 @@ >> >> #include "ioat_hw_defs.h" >> >> +struct ioat_dmadev { >> + struct rte_dma_dev *dmadev; > maybe the rte_dma_dev_data more appropriate if support multi-process. Changed for v5. >> + struct rte_dma_vchan_conf qcfg; >> + struct rte_dma_stats stats; >> + >> + volatile uint16_t *doorbell __rte_cache_aligned; >> + phys_addr_t status_addr; >> + phys_addr_t ring_addr; >> + >> + struct ioat_dma_hw_desc *desc_ring; >> + >> + unsigned short next_read; >> + unsigned short next_write; >> + unsigned short last_write; /* Used to compute submitted count. */ >> + unsigned short offset; /* Used after a device recovery when counts -> 0. */ >> + unsigned int failure; /* Used to store chanerr for error handling. */ >> + >> + /* To report completions, the device will write status back here. */ >> + volatile uint64_t status __rte_cache_aligned; >> + >> + /* Pointer to the register bar. */ >> + volatile struct ioat_registers *regs; >> +}; >> + >> extern int ioat_pmd_logtype; >> >> #define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ >> ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v4 03/11] dma/ioat: add datapath structures 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 01/11] dma/ioat: add device probe and removal functions Conor Walsh 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 02/11] dma/ioat: create dmadev instances on PCI probe Conor Walsh @ 2021-09-17 15:42 ` Conor Walsh 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 04/11] dma/ioat: add configuration functions Conor Walsh ` (7 subsequent siblings) 10 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-17 15:42 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add data structures required for the data path of IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 61 ++++++++- drivers/dma/ioat/ioat_hw_defs.h | 214 ++++++++++++++++++++++++++++++++ 2 files changed, 274 insertions(+), 1 deletion(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index b815d30bcf..370be3ec72 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -15,11 +15,70 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Dump DMA device info. */ +static int +ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint64_t chansts_masked = ioat->regs->chansts & IOAT_CHANSTS_STATUS; + uint32_t chanerr = ioat->regs->chanerr; + uint64_t mask = (ioat->qcfg.nb_desc - 1); + fprintf(f, "========= IOAT =========\n"); + fprintf(f, " Channel status: %s [0x%"PRIx64"]\n", + chansts_readable[chansts_masked], chansts_masked); + fprintf(f, " ChainADDR: 0x%"PRIu64"\n", ioat->regs->chainaddr); + if (chanerr == 0) { + fprintf(f, " No Channel Errors\n"); + } else { + fprintf(f, " ChanERR: 0x%"PRIu32"\n", chanerr); + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + fprintf(f, " Invalid Source Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + fprintf(f, " Invalid Destination Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + fprintf(f, " Invalid Descriptor Length\n"); + if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + fprintf(f, " Descriptor Read Error\n"); + if ((chanerr & ~(IOAT_CHANERR_INVALID_SRC_ADDR_MASK | + IOAT_CHANERR_INVALID_DST_ADDR_MASK | + IOAT_CHANERR_INVALID_LENGTH_MASK | + IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK)) != 0) + fprintf(f, " Unknown Error(s)\n"); + } + fprintf(f, "== Private Data ==\n"); + fprintf(f, " Config: { ring_size: %u }\n", ioat->qcfg.nb_desc); + fprintf(f, " Status: 0x%"PRIx64"\n", ioat->status); + fprintf(f, " Status IOVA: 0x%"PRIx64"\n", ioat->status_addr); + fprintf(f, " Status ADDR: %p\n", &ioat->status); + fprintf(f, " Ring IOVA: 0x%"PRIx64"\n", ioat->ring_addr); + fprintf(f, " Ring ADDR: 0x%"PRIx64"\n", ioat->desc_ring[0].next-64); + fprintf(f, " Next write: %"PRIu16"\n", ioat->next_write); + fprintf(f, " Next read: %"PRIu16"\n", ioat->next_read); + struct ioat_dma_hw_desc *desc_ring = &ioat->desc_ring[(ioat->next_write - 1) & mask]; + fprintf(f, " Last Descriptor Written {\n"); + fprintf(f, " Size: %"PRIu32"\n", desc_ring->size); + fprintf(f, " Control: 0x%"PRIx32"\n", desc_ring->u.control_raw); + fprintf(f, " Src: 0x%"PRIx64"\n", desc_ring->src_addr); + fprintf(f, " Dest: 0x%"PRIx64"\n", desc_ring->dest_addr); + fprintf(f, " Next: 0x%"PRIx64"\n", desc_ring->next); + fprintf(f, " }\n"); + fprintf(f, " Next Descriptor {\n"); + fprintf(f, " Size: %"PRIu32"\n", ioat->desc_ring[ioat->next_read & mask].size); + fprintf(f, " Src: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].src_addr); + fprintf(f, " Dest: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); + fprintf(f, " Next: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].next); + fprintf(f, " }\n"); + + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { - static const struct rte_dma_dev_ops ioat_dmadev_ops = { }; + static const struct rte_dma_dev_ops ioat_dmadev_ops = { + .dev_dump = ioat_dev_dump, + }; struct rte_dma_dev *dmadev = NULL; struct ioat_dmadev *ioat = NULL; diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index 73bdf548b3..1f47209e60 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -43,6 +43,14 @@ extern "C" { #define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 #define IOAT_CHANCTRL_INT_REARM 0x0001 +/* DMA Channel Capabilities */ +#define IOAT_DMACAP_PB (1 << 0) +#define IOAT_DMACAP_DCA (1 << 4) +#define IOAT_DMACAP_BFILL (1 << 6) +#define IOAT_DMACAP_XOR (1 << 8) +#define IOAT_DMACAP_PQ (1 << 9) +#define IOAT_DMACAP_DMA_DIF (1 << 10) + struct ioat_registers { uint8_t chancnt; uint8_t xfercap; @@ -71,8 +79,214 @@ struct ioat_registers { #define IOAT_CHANCMD_RESET 0x20 #define IOAT_CHANCMD_SUSPEND 0x04 +#define IOAT_CHANSTS_STATUS 0x7ULL +#define IOAT_CHANSTS_ACTIVE 0x0 +#define IOAT_CHANSTS_IDLE 0x1 +#define IOAT_CHANSTS_SUSPENDED 0x2 +#define IOAT_CHANSTS_HALTED 0x3 +#define IOAT_CHANSTS_ARMED 0x4 + +#define IOAT_CHANERR_INVALID_SRC_ADDR_MASK (1 << 0) +#define IOAT_CHANERR_INVALID_DST_ADDR_MASK (1 << 1) +#define IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK (1 << 8) +#define IOAT_CHANERR_INVALID_LENGTH_MASK (1 << 10) + +const char *chansts_readable[] = { + "ACTIVE", /* 0x0 */ + "IDLE", /* 0x1 */ + "SUSPENDED", /* 0x2 */ + "HALTED", /* 0x3 */ + "ARMED" /* 0x4 */ +}; + +#define IOAT_CHANSTS_UNAFFILIATED_ERROR 0x8ULL +#define IOAT_CHANSTS_SOFT_ERROR 0x10ULL + +#define IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK (~0x3FULL) + #define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ +struct ioat_dma_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t null: 1; + uint32_t src_page_break: 1; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_COPY 0x00 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t reserved2; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_fill_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t reserved: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t reserved2: 2; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t reserved3: 15; +#define IOAT_OP_FILL 0x01 + uint32_t op: 8; + } control; + } u; + uint64_t src_data; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t next_dest_addr; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_xor_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_XOR 0x87 +#define IOAT_OP_XOR_VAL 0x88 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint64_t src_addr4; + uint64_t src_addr5; +}; + +struct ioat_xor_ext_hw_desc { + uint64_t src_addr6; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t next; + uint64_t reserved[4]; +}; + +struct ioat_pq_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 11; +#define IOAT_OP_PQ 0x89 +#define IOAT_OP_PQ_VAL 0x8a + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint8_t coef[8]; + uint64_t q_addr; +}; + +struct ioat_pq_ext_hw_desc { + uint64_t src_addr4; + uint64_t src_addr5; + uint64_t src_addr6; + uint64_t next; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t reserved[2]; +}; + +struct ioat_pq_update_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_cnt: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 3; + uint32_t coef: 8; +#define IOAT_OP_PQ_UP 0x8b + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t p_src; + uint64_t q_src; + uint64_t q_addr; +}; + +union ioat_hw_desc { + struct ioat_dma_hw_desc dma; + struct ioat_fill_hw_desc fill; + struct ioat_xor_hw_desc xor_desc; + struct ioat_xor_ext_hw_desc xor_ext; + struct ioat_pq_hw_desc pq; + struct ioat_pq_ext_hw_desc pq_ext; + struct ioat_pq_update_hw_desc pq_update; +}; + +#define GENSTS_DEV_STATE_MASK 0x03 +#define CMDSTATUS_ACTIVE_SHIFT 31 +#define CMDSTATUS_ACTIVE_MASK (1 << 31) +#define CMDSTATUS_ERR_MASK 0xFF + #ifdef __cplusplus } #endif -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v4 04/11] dma/ioat: add configuration functions 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (2 preceding siblings ...) 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 03/11] dma/ioat: add datapath structures Conor Walsh @ 2021-09-17 15:42 ` Conor Walsh 2021-09-22 8:08 ` fengchengwen 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 05/11] dma/ioat: add start and stop functions Conor Walsh ` (6 subsequent siblings) 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-17 15:42 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add functions for device configuration. The info_get and close functions are included here also. info_get can be useful for checking successful configuration and close is used by the dmadev api when releasing a configured device. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 24 +++++++++ drivers/dma/ioat/ioat_dmadev.c | 93 ++++++++++++++++++++++++++++++++++ 2 files changed, 117 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 45a2e65d70..6092599fe4 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -62,3 +62,27 @@ For example:: Once probed successfully, the device will appear as a ``dmadev``, that is a "DMA device type" inside DPDK, and can be accessed using APIs from the ``rte_dmadev`` library. + +Device Configuration +~~~~~~~~~~~~~~~~~~~~~ + +Configuring an IOAT dmadev device is done using the ``rte_dma_configure()`` +and ``rte_dma_vchan_setup()`` APIs. ``rte_dma_configure()`` uses the structure +``rte_dma_conf`` to configure an IOAT device, within this struct the number +of virtual DMA channels for the device is set and silent mode can be enabled. +Each IOAT device can only use 1 virtual DMA channel and silent mode is not +supported so these will always be set to ``1`` and ``false``. +``rte_dma_vchan_setup()`` uses the structure ``rte_dma_vchan_conf`` to setup +a virtual DMA channel within this struct the transfer direction and ring size +are set. Generally for an onboard Intel\ |reg| IOAT device the transfer direction +will be set to ``RTE_DMA_DIR_MEM_TO_MEM`` to copy from memory to memory (more info +available in the dmadev API guide). The ring size must be a power of two, between +64 and 4096. + +The following code shows how the device is configured in ``test_dmadev.c``: + +.. literalinclude:: ../../../app/test/test_dmadev.c + :language: c + :start-after: Setup of the dmadev device. 8< + :end-before: >8 End of setup of the dmadev device. + :dedent: 1 diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 370be3ec72..d58022e258 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -12,9 +12,98 @@ static struct rte_pci_driver ioat_pmd_drv; RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); +#define DESC_SZ sizeof(struct ioat_dma_hw_desc) + #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Configure a device. */ +static int +ioat_dev_configure(struct rte_dma_dev *dev __rte_unused, const struct rte_dma_conf *dev_conf, + uint32_t conf_sz) +{ + if (sizeof(struct rte_dma_conf) != conf_sz) + return -EINVAL; + + if (dev_conf->nb_vchans != 1) + return -EINVAL; + + return 0; +} + +/* Setup a virtual channel for IOAT, only 1 vchan is supported. */ +static int +ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + const struct rte_dma_vchan_conf *qconf, uint32_t qconf_sz) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint16_t max_desc = qconf->nb_desc; + int i; + + if (sizeof(struct rte_dma_vchan_conf) != qconf_sz) + return -EINVAL; + + ioat->qcfg = *qconf; + + if (!rte_is_power_of_2(max_desc)) { + max_desc = rte_align32pow2(max_desc); + IOAT_PMD_DEBUG("DMA dev %u using %u descriptors", dev->data->dev_id, max_desc); + ioat->qcfg.nb_desc = max_desc; + } + + /* In case we are reconfiguring a device, free any existing memory. */ + rte_free(ioat->desc_ring); + + ioat->desc_ring = rte_zmalloc(NULL, sizeof(*ioat->desc_ring) * max_desc, 0); + if (ioat->desc_ring == NULL) + return -ENOMEM; + + ioat->ring_addr = rte_mem_virt2iova(ioat->desc_ring); + + ioat->status_addr = rte_mem_virt2iova(ioat) + offsetof(struct ioat_dmadev, status); + + /* Ensure all counters are reset, if reconfiguring/restarting device. */ + ioat->next_read = 0; + ioat->next_write = 0; + ioat->last_write = 0; + ioat->offset = 0; + ioat->failure = 0; + + /* Configure descriptor ring - each one points to next. */ + for (i = 0; i < ioat->qcfg.nb_desc; i++) { + ioat->desc_ring[i].next = ioat->ring_addr + + (((i + 1) % ioat->qcfg.nb_desc) * DESC_SZ); + } + + return 0; +} + +/* Get device information of a device. */ +static int +ioat_dev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *info, uint32_t size) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + if (size < sizeof(*info)) + return -EINVAL; + info->dev_capa = RTE_DMA_CAPA_MEM_TO_MEM | + RTE_DMA_CAPA_OPS_COPY | + RTE_DMA_CAPA_OPS_FILL; + info->max_vchans = 1; + info->min_desc = 32; + info->max_desc = 4096; + info->nb_vchans = (ioat->desc_ring == NULL); + return 0; +} + +/* Close a configured device. */ +static int +ioat_dev_close(struct rte_dma_dev *dev) +{ + RTE_SET_USED(dev); + return 0; +} + /* Dump DMA device info. */ static int ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) @@ -77,7 +166,11 @@ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { static const struct rte_dma_dev_ops ioat_dmadev_ops = { + .dev_close = ioat_dev_close, + .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, + .dev_info_get = ioat_dev_info_get, + .vchan_setup = ioat_vchan_setup, }; struct rte_dma_dev *dmadev = NULL; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 04/11] dma/ioat: add configuration functions 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 04/11] dma/ioat: add configuration functions Conor Walsh @ 2021-09-22 8:08 ` fengchengwen 2021-09-22 16:41 ` Conor Walsh 0 siblings, 1 reply; 132+ messages in thread From: fengchengwen @ 2021-09-22 8:08 UTC (permalink / raw) To: Conor Walsh, bruce.richardson, jerinj, kevin.laatz; +Cc: dev On 2021/9/17 23:42, Conor Walsh wrote: > Add functions for device configuration. The info_get and close functions > are included here also. info_get can be useful for checking successful > configuration and close is used by the dmadev api when releasing a > configured device. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> > --- > doc/guides/dmadevs/ioat.rst | 24 +++++++++ > drivers/dma/ioat/ioat_dmadev.c | 93 ++++++++++++++++++++++++++++++++++ > 2 files changed, 117 insertions(+) > > diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst > index 45a2e65d70..6092599fe4 100644 > --- a/doc/guides/dmadevs/ioat.rst > +++ b/doc/guides/dmadevs/ioat.rst > @@ -62,3 +62,27 @@ For example:: > Once probed successfully, the device will appear as a ``dmadev``, that is a > "DMA device type" inside DPDK, and can be accessed using APIs from the > ``rte_dmadev`` library. > + > +Device Configuration > +~~~~~~~~~~~~~~~~~~~~~ > + > +Configuring an IOAT dmadev device is done using the ``rte_dma_configure()`` > +and ``rte_dma_vchan_setup()`` APIs. ``rte_dma_configure()`` uses the structure > +``rte_dma_conf`` to configure an IOAT device, within this struct the number > +of virtual DMA channels for the device is set and silent mode can be enabled. > +Each IOAT device can only use 1 virtual DMA channel and silent mode is not > +supported so these will always be set to ``1`` and ``false``. > +``rte_dma_vchan_setup()`` uses the structure ``rte_dma_vchan_conf`` to setup > +a virtual DMA channel within this struct the transfer direction and ring size > +are set. Generally for an onboard Intel\ |reg| IOAT device the transfer direction > +will be set to ``RTE_DMA_DIR_MEM_TO_MEM`` to copy from memory to memory (more info > +available in the dmadev API guide). The ring size must be a power of two, between > +64 and 4096. > + > +The following code shows how the device is configured in ``test_dmadev.c``: > + > +.. literalinclude:: ../../../app/test/test_dmadev.c > + :language: c > + :start-after: Setup of the dmadev device. 8< > + :end-before: >8 End of setup of the dmadev device. > + :dedent: 1 > diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c > index 370be3ec72..d58022e258 100644 > --- a/drivers/dma/ioat/ioat_dmadev.c > +++ b/drivers/dma/ioat/ioat_dmadev.c > @@ -12,9 +12,98 @@ static struct rte_pci_driver ioat_pmd_drv; > > RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); > > +#define DESC_SZ sizeof(struct ioat_dma_hw_desc) > + > #define IOAT_PMD_NAME dmadev_ioat > #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) > > +/* Configure a device. */ > +static int > +ioat_dev_configure(struct rte_dma_dev *dev __rte_unused, const struct rte_dma_conf *dev_conf, > + uint32_t conf_sz) > +{ > + if (sizeof(struct rte_dma_conf) != conf_sz) > + return -EINVAL; > + > + if (dev_conf->nb_vchans != 1) > + return -EINVAL; > + > + return 0; > +} > + > +/* Setup a virtual channel for IOAT, only 1 vchan is supported. */ > +static int > +ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, > + const struct rte_dma_vchan_conf *qconf, uint32_t qconf_sz) > +{ > + struct ioat_dmadev *ioat = dev->dev_private; > + uint16_t max_desc = qconf->nb_desc; > + int i; > + > + if (sizeof(struct rte_dma_vchan_conf) != qconf_sz) > + return -EINVAL; > + > + ioat->qcfg = *qconf; > + > + if (!rte_is_power_of_2(max_desc)) { > + max_desc = rte_align32pow2(max_desc); > + IOAT_PMD_DEBUG("DMA dev %u using %u descriptors", dev->data->dev_id, max_desc); > + ioat->qcfg.nb_desc = max_desc; > + } > + > + /* In case we are reconfiguring a device, free any existing memory. */ > + rte_free(ioat->desc_ring); > + > + ioat->desc_ring = rte_zmalloc(NULL, sizeof(*ioat->desc_ring) * max_desc, 0); > + if (ioat->desc_ring == NULL) > + return -ENOMEM; > + > + ioat->ring_addr = rte_mem_virt2iova(ioat->desc_ring); > + > + ioat->status_addr = rte_mem_virt2iova(ioat) + offsetof(struct ioat_dmadev, status); > + > + /* Ensure all counters are reset, if reconfiguring/restarting device. */ > + ioat->next_read = 0; > + ioat->next_write = 0; > + ioat->last_write = 0; > + ioat->offset = 0; > + ioat->failure = 0; > + > + /* Configure descriptor ring - each one points to next. */ > + for (i = 0; i < ioat->qcfg.nb_desc; i++) { > + ioat->desc_ring[i].next = ioat->ring_addr + > + (((i + 1) % ioat->qcfg.nb_desc) * DESC_SZ); > + } > + > + return 0; > +} > + > +/* Get device information of a device. */ > +static int > +ioat_dev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *info, uint32_t size) > +{ > + struct ioat_dmadev *ioat = dev->dev_private; > + > + if (size < sizeof(*info)) > + return -EINVAL; > + info->dev_capa = RTE_DMA_CAPA_MEM_TO_MEM | > + RTE_DMA_CAPA_OPS_COPY | > + RTE_DMA_CAPA_OPS_FILL; > + info->max_vchans = 1; > + info->min_desc = 32; > + info->max_desc = 4096; > + info->nb_vchans = (ioat->desc_ring == NULL); The nb_vchans will overwrite by library. > + return 0; > +} > + > +/* Close a configured device. */ > +static int > +ioat_dev_close(struct rte_dma_dev *dev) > +{ > + RTE_SET_USED(dev); > + return 0; > +} > + > /* Dump DMA device info. */ > static int > ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) > @@ -77,7 +166,11 @@ static int > ioat_dmadev_create(const char *name, struct rte_pci_device *dev) > { > static const struct rte_dma_dev_ops ioat_dmadev_ops = { > + .dev_close = ioat_dev_close, > + .dev_configure = ioat_dev_configure, > .dev_dump = ioat_dev_dump, > + .dev_info_get = ioat_dev_info_get, > + .vchan_setup = ioat_vchan_setup, > }; > > struct rte_dma_dev *dmadev = NULL; > ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 04/11] dma/ioat: add configuration functions 2021-09-22 8:08 ` fengchengwen @ 2021-09-22 16:41 ` Conor Walsh 0 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-22 16:41 UTC (permalink / raw) To: fengchengwen, bruce.richardson, jerinj, kevin.laatz; +Cc: dev On 22/09/2021 09:08, fengchengwen wrote: > On 2021/9/17 23:42, Conor Walsh wrote: >> Add functions for device configuration. The info_get and close functions >> are included here also. info_get can be useful for checking successful >> configuration and close is used by the dmadev api when releasing a >> configured device. >> >> Signed-off-by: Conor Walsh <conor.walsh@intel.com> >> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> >> --- >> doc/guides/dmadevs/ioat.rst | 24 +++++++++ >> drivers/dma/ioat/ioat_dmadev.c | 93 ++++++++++++++++++++++++++++++++++ >> 2 files changed, 117 insertions(+) >> >> diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst >> index 45a2e65d70..6092599fe4 100644 >> --- a/doc/guides/dmadevs/ioat.rst >> +++ b/doc/guides/dmadevs/ioat.rst >> @@ -62,3 +62,27 @@ For example:: >> Once probed successfully, the device will appear as a ``dmadev``, that is a >> "DMA device type" inside DPDK, and can be accessed using APIs from the >> ``rte_dmadev`` library. >> + >> +Device Configuration >> +~~~~~~~~~~~~~~~~~~~~~ >> + >> +Configuring an IOAT dmadev device is done using the ``rte_dma_configure()`` >> +and ``rte_dma_vchan_setup()`` APIs. ``rte_dma_configure()`` uses the structure >> +``rte_dma_conf`` to configure an IOAT device, within this struct the number >> +of virtual DMA channels for the device is set and silent mode can be enabled. >> +Each IOAT device can only use 1 virtual DMA channel and silent mode is not >> +supported so these will always be set to ``1`` and ``false``. >> +``rte_dma_vchan_setup()`` uses the structure ``rte_dma_vchan_conf`` to setup >> +a virtual DMA channel within this struct the transfer direction and ring size >> +are set. Generally for an onboard Intel\ |reg| IOAT device the transfer direction >> +will be set to ``RTE_DMA_DIR_MEM_TO_MEM`` to copy from memory to memory (more info >> +available in the dmadev API guide). The ring size must be a power of two, between >> +64 and 4096. >> + >> +The following code shows how the device is configured in ``test_dmadev.c``: >> + >> +.. literalinclude:: ../../../app/test/test_dmadev.c >> + :language: c >> + :start-after: Setup of the dmadev device. 8< >> + :end-before: >8 End of setup of the dmadev device. >> + :dedent: 1 >> diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c >> index 370be3ec72..d58022e258 100644 >> --- a/drivers/dma/ioat/ioat_dmadev.c >> +++ b/drivers/dma/ioat/ioat_dmadev.c >> @@ -12,9 +12,98 @@ static struct rte_pci_driver ioat_pmd_drv; >> >> RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); >> >> +#define DESC_SZ sizeof(struct ioat_dma_hw_desc) >> + >> #define IOAT_PMD_NAME dmadev_ioat >> #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) >> >> +/* Configure a device. */ >> +static int >> +ioat_dev_configure(struct rte_dma_dev *dev __rte_unused, const struct rte_dma_conf *dev_conf, >> + uint32_t conf_sz) >> +{ >> + if (sizeof(struct rte_dma_conf) != conf_sz) >> + return -EINVAL; >> + >> + if (dev_conf->nb_vchans != 1) >> + return -EINVAL; >> + >> + return 0; >> +} >> + >> +/* Setup a virtual channel for IOAT, only 1 vchan is supported. */ >> +static int >> +ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, >> + const struct rte_dma_vchan_conf *qconf, uint32_t qconf_sz) >> +{ >> + struct ioat_dmadev *ioat = dev->dev_private; >> + uint16_t max_desc = qconf->nb_desc; >> + int i; >> + >> + if (sizeof(struct rte_dma_vchan_conf) != qconf_sz) >> + return -EINVAL; >> + >> + ioat->qcfg = *qconf; >> + >> + if (!rte_is_power_of_2(max_desc)) { >> + max_desc = rte_align32pow2(max_desc); >> + IOAT_PMD_DEBUG("DMA dev %u using %u descriptors", dev->data->dev_id, max_desc); >> + ioat->qcfg.nb_desc = max_desc; >> + } >> + >> + /* In case we are reconfiguring a device, free any existing memory. */ >> + rte_free(ioat->desc_ring); >> + >> + ioat->desc_ring = rte_zmalloc(NULL, sizeof(*ioat->desc_ring) * max_desc, 0); >> + if (ioat->desc_ring == NULL) >> + return -ENOMEM; >> + >> + ioat->ring_addr = rte_mem_virt2iova(ioat->desc_ring); >> + >> + ioat->status_addr = rte_mem_virt2iova(ioat) + offsetof(struct ioat_dmadev, status); >> + >> + /* Ensure all counters are reset, if reconfiguring/restarting device. */ >> + ioat->next_read = 0; >> + ioat->next_write = 0; >> + ioat->last_write = 0; >> + ioat->offset = 0; >> + ioat->failure = 0; >> + >> + /* Configure descriptor ring - each one points to next. */ >> + for (i = 0; i < ioat->qcfg.nb_desc; i++) { >> + ioat->desc_ring[i].next = ioat->ring_addr + >> + (((i + 1) % ioat->qcfg.nb_desc) * DESC_SZ); >> + } >> + >> + return 0; >> +} >> + >> +/* Get device information of a device. */ >> +static int >> +ioat_dev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *info, uint32_t size) >> +{ >> + struct ioat_dmadev *ioat = dev->dev_private; >> + >> + if (size < sizeof(*info)) >> + return -EINVAL; >> + info->dev_capa = RTE_DMA_CAPA_MEM_TO_MEM | >> + RTE_DMA_CAPA_OPS_COPY | >> + RTE_DMA_CAPA_OPS_FILL; >> + info->max_vchans = 1; >> + info->min_desc = 32; >> + info->max_desc = 4096; >> + info->nb_vchans = (ioat->desc_ring == NULL); > The nb_vchans will overwrite by library. Updated for v5, should be upstreamed in the next few days. Thanks for the review Chengwen! /Conor. > >> + return 0; >> +} >> + >> +/* Close a configured device. */ >> +static int >> +ioat_dev_close(struct rte_dma_dev *dev) >> +{ >> + RTE_SET_USED(dev); >> + return 0; >> +} >> + >> /* Dump DMA device info. */ >> static int >> ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) >> @@ -77,7 +166,11 @@ static int >> ioat_dmadev_create(const char *name, struct rte_pci_device *dev) >> { >> static const struct rte_dma_dev_ops ioat_dmadev_ops = { >> + .dev_close = ioat_dev_close, >> + .dev_configure = ioat_dev_configure, >> .dev_dump = ioat_dev_dump, >> + .dev_info_get = ioat_dev_info_get, >> + .vchan_setup = ioat_vchan_setup, >> }; >> >> struct rte_dma_dev *dmadev = NULL; >> ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v4 05/11] dma/ioat: add start and stop functions 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (3 preceding siblings ...) 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 04/11] dma/ioat: add configuration functions Conor Walsh @ 2021-09-17 15:42 ` Conor Walsh 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 06/11] dma/ioat: add data path job submission functions Conor Walsh ` (5 subsequent siblings) 10 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-17 15:42 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add start, stop and recover functions for IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 3 ++ drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 6092599fe4..a64d67bf89 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -86,3 +86,6 @@ The following code shows how the device is configured in ``test_dmadev.c``: :start-after: Setup of the dmadev device. 8< :end-before: >8 End of setup of the dmadev device. :dedent: 1 + +Once configured, the device can then be made ready for use by calling the +``rte_dma_start()`` API. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index d58022e258..a47567ca66 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -78,6 +78,96 @@ ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, return 0; } +/* Recover IOAT device. */ +static inline int +__ioat_recover(struct ioat_dmadev *ioat) +{ + uint32_t chanerr, retry = 0; + uint16_t mask = ioat->qcfg.nb_desc - 1; + + /* Clear any channel errors. Reading and writing to chanerr does this. */ + chanerr = ioat->regs->chanerr; + ioat->regs->chanerr = chanerr; + + /* Reset Channel. */ + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + + /* Write new chain address to trigger state change. */ + ioat->regs->chainaddr = ioat->desc_ring[(ioat->next_read - 1) & mask].next; + /* Ensure channel control and status addr are correct. */ + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + ioat->regs->chancmp = ioat->status_addr; + + /* Allow HW time to move to the ARMED state. */ + do { + rte_pause(); + retry++; + } while (ioat->regs->chansts != IOAT_CHANSTS_ARMED && retry < 200); + + /* Exit as failure if device is still HALTED. */ + if (ioat->regs->chansts != IOAT_CHANSTS_ARMED) + return -1; + + /* Store next write as offset as recover will move HW and SW ring out of sync. */ + ioat->offset = ioat->next_read; + + /* Prime status register with previous address. */ + ioat->status = ioat->desc_ring[(ioat->next_read - 2) & mask].next; + + return 0; +} + +/* Start a configured device. */ +static int +ioat_dev_start(struct rte_dma_dev *dev) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + if (ioat->qcfg.nb_desc == 0 || ioat->desc_ring == NULL) + return -EBUSY; + + /* Inform hardware of where the descriptor ring is. */ + ioat->regs->chainaddr = ioat->ring_addr; + /* Inform hardware of where to write the status/completions. */ + ioat->regs->chancmp = ioat->status_addr; + + /* Prime the status register to be set to the last element. */ + ioat->status = ioat->ring_addr + ((ioat->qcfg.nb_desc - 1) * DESC_SZ); + + printf("IOAT.status: %s [0x%"PRIx64"]\n", + chansts_readable[ioat->status & IOAT_CHANSTS_STATUS], + ioat->status); + + if ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_HALTED) { + IOAT_PMD_WARN("Device HALTED on start, attempting to recover\n"); + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device couldn't be recovered"); + return -1; + } + } + + return 0; +} + +/* Stop a configured device. */ +static int +ioat_dev_stop(struct rte_dma_dev *dev) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint32_t retry = 0; + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + + do { + rte_pause(); + retry++; + } while ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) != IOAT_CHANSTS_SUSPENDED + && retry < 200); + + return ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_SUSPENDED) ? 0 : -1; +} + /* Get device information of a device. */ static int ioat_dev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *info, uint32_t size) @@ -170,6 +260,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, .dev_info_get = ioat_dev_info_get, + .dev_start = ioat_dev_start, + .dev_stop = ioat_dev_stop, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v4 06/11] dma/ioat: add data path job submission functions 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (4 preceding siblings ...) 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 05/11] dma/ioat: add start and stop functions Conor Walsh @ 2021-09-17 15:42 ` Conor Walsh 2021-09-20 13:36 ` Bruce Richardson 2021-09-22 8:18 ` fengchengwen 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 07/11] dma/ioat: add data path completion functions Conor Walsh ` (4 subsequent siblings) 10 siblings, 2 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-17 15:42 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add data path functions for enqueuing and submitting operations to IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 54 ++++++++++++++++++++ drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index a64d67bf89..2464207e20 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -89,3 +89,57 @@ The following code shows how the device is configured in ``test_dmadev.c``: Once configured, the device can then be made ready for use by calling the ``rte_dma_start()`` API. + +Performing Data Copies +~~~~~~~~~~~~~~~~~~~~~~~ + +To perform data copies using IOAT dmadev devices, the functions +``rte_dma_copy()`` and ``rte_dma_submit()`` should be used. Alternatively +``rte_dma_copy()`` can be called with the ``RTE_DMA_OP_FLAG_SUBMIT`` flag +set. + +The ``rte_dma_copy()`` function enqueues a single copy to the +device ring for copying at a later point. The parameters to the function +include the device ID of the desired device, the virtual DMA channel required +(always 0 for IOAT), the IOVA addresses of both the source and destination +buffers, the length of the data to be copied and any operation flags. The +function will return the index of the enqueued job which can be use to +track that operation. + +While the ``rte_dma_copy()`` function enqueues a copy operation on the device +ring, the copy will not actually be performed until after the application calls +the ``rte_dma_submit()`` function. This function informs the device hardware +of the elements enqueued on the ring, and the device will begin to process them. +It is expected that, for efficiency reasons, a burst of operations will be +enqueued to the device via multiple enqueue calls between calls to the +``rte_dma_submit()`` function. If desired you can pass the +``RTE_DMA_OP_FLAG_SUBMIT`` flag when calling ``rte_dma_copy()`` and this will +tell the device to perform the enqueued operation and any unperformed operations +before it. The ``RTE_DMA_OP_FLAG_SUBMIT`` flag can be passed instead of calling +the ``rte_dma_submit()`` function for example on the last enqueue of the burst. + +The following code from demonstrates how to enqueue a burst of copies to the +device and start the hardware processing of them: + +.. code-block:: C + + for (i = 0; i < BURST_SIZE; i++) { + if (rte_dma_copy(dev_id, vchan, rte_mbuf_data_iova(srcs[i]), + rte_mbuf_data_iova(dsts[i]), COPY_LEN, 0) < 0) { + PRINT_ERR("Error with rte_dma_copy for buffer %u\n", i); + return -1; + } + } + if (rte_dma_submit(dev_id, vchan) < 0) { + PRINT_ERR("Error with performing operations\n", i); + return -1; + } + +Filling an Area of Memory +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The driver also has support for the ``fill`` operation, where an area +of memory is overwritten, or filled, with a short pattern of data. +Fill operations can be performed in much the same was as copy operations +described above, just using the ``rte_dma_fill()`` function rather +than the ``rte_dma_copy()`` function. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index a47567ca66..edcc882d63 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -5,6 +5,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> #include <rte_malloc.h> +#include <rte_prefetch.h> #include "ioat_internal.h" @@ -17,6 +18,12 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* IOAT operations. */ +enum rte_ioat_ops { + ioat_op_copy = 0, /* Standard DMA Operation */ + ioat_op_fill /* Block Fill */ +}; + /* Configure a device. */ static int ioat_dev_configure(struct rte_dma_dev *dev __rte_unused, const struct rte_dma_conf *dev_conf, @@ -194,6 +201,87 @@ ioat_dev_close(struct rte_dma_dev *dev) return 0; } +/* Trigger hardware to begin performing enqueued operations. */ +static inline void +__submit(struct ioat_dmadev *ioat) +{ + *ioat->doorbell = ioat->next_write - ioat->offset; + + ioat->last_write = ioat->next_write; +} + +/* External submit function wrapper. */ +static int +ioat_submit(struct rte_dma_dev *dev, uint16_t qid __rte_unused) +{ + struct ioat_dmadev *ioat = (struct ioat_dmadev *)dev->dev_private; + + __submit(ioat); + + return 0; +} + +/* Write descriptor for enqueue. */ +static inline int +__write_desc(struct rte_dma_dev *dev, uint32_t op, uint64_t src, phys_addr_t dst, + unsigned int length, uint64_t flags) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint16_t ret; + const unsigned short mask = ioat->qcfg.nb_desc - 1; + const unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + const unsigned short space = mask + read - write; + struct ioat_dma_hw_desc *desc; + + if (space == 0) + return -ENOSPC; + + ioat->next_write = write + 1; + write &= mask; + + desc = &ioat->desc_ring[write]; + desc->size = length; + desc->u.control_raw = (uint32_t)((op << IOAT_CMD_OP_SHIFT) | + (1 << IOAT_COMP_UPDATE_SHIFT)); + + /* In IOAT the fence ensures that all operations including the current one + * are completed before moving on, DMAdev assumes that the fence ensures + * all operations before the current one are completed before starting + * the current one, so in IOAT we set the fence for the previous descriptor. + */ + if (flags & RTE_DMA_OP_FLAG_FENCE) + ioat->desc_ring[(write - 1) & mask].u.control.fence = 1; + + desc->src_addr = src; + desc->dest_addr = dst; + + rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]); + + ret = (uint16_t)(ioat->next_write - 1); + + if (flags & RTE_DMA_OP_FLAG_SUBMIT) + __submit(ioat); + + return ret; +} + +/* Enqueue a fill operation onto the ioat device. */ +static int +ioat_enqueue_fill(struct rte_dma_dev *dev, uint16_t qid __rte_unused, uint64_t pattern, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev, ioat_op_fill, pattern, dst, length, flags); +} + +/* Enqueue a copy operation onto the ioat device. */ +static int +ioat_enqueue_copy(struct rte_dma_dev *dev, uint16_t qid __rte_unused, rte_iova_t src, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev, ioat_op_copy, src, dst, length, flags); +} + /* Dump DMA device info. */ static int ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) @@ -290,6 +378,10 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->copy = ioat_enqueue_copy; + dmadev->fill = ioat_enqueue_fill; + dmadev->submit = ioat_submit; + ioat = dmadev->data->dev_private; ioat->dmadev = dmadev; ioat->regs = dev->mem_resource[0].addr; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 06/11] dma/ioat: add data path job submission functions 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 06/11] dma/ioat: add data path job submission functions Conor Walsh @ 2021-09-20 13:36 ` Bruce Richardson 2021-09-21 16:25 ` Conor Walsh 2021-09-22 8:18 ` fengchengwen 1 sibling, 1 reply; 132+ messages in thread From: Bruce Richardson @ 2021-09-20 13:36 UTC (permalink / raw) To: Conor Walsh; +Cc: fengchengwen, jerinj, kevin.laatz, dev On Fri, Sep 17, 2021 at 03:42:22PM +0000, Conor Walsh wrote: > Add data path functions for enqueuing and submitting operations to > IOAT devices. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> > --- > doc/guides/dmadevs/ioat.rst | 54 ++++++++++++++++++++ > drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ > 2 files changed, 146 insertions(+) > > diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst > index a64d67bf89..2464207e20 100644 > --- a/doc/guides/dmadevs/ioat.rst > +++ b/doc/guides/dmadevs/ioat.rst > @@ -89,3 +89,57 @@ The following code shows how the device is configured in ``test_dmadev.c``: > > Once configured, the device can then be made ready for use by calling the > ``rte_dma_start()`` API. > + > +Performing Data Copies > +~~~~~~~~~~~~~~~~~~~~~~~ > + > +To perform data copies using IOAT dmadev devices, the functions > +``rte_dma_copy()`` and ``rte_dma_submit()`` should be used. Alternatively > +``rte_dma_copy()`` can be called with the ``RTE_DMA_OP_FLAG_SUBMIT`` flag > +set. > + > +The ``rte_dma_copy()`` function enqueues a single copy to the > +device ring for copying at a later point. The parameters to the function > +include the device ID of the desired device, the virtual DMA channel required > +(always 0 for IOAT), the IOVA addresses of both the source and destination > +buffers, the length of the data to be copied and any operation flags. The > +function will return the index of the enqueued job which can be use to > +track that operation. > + > +While the ``rte_dma_copy()`` function enqueues a copy operation on the device > +ring, the copy will not actually be performed until after the application calls > +the ``rte_dma_submit()`` function. This function informs the device hardware > +of the elements enqueued on the ring, and the device will begin to process them. > +It is expected that, for efficiency reasons, a burst of operations will be > +enqueued to the device via multiple enqueue calls between calls to the > +``rte_dma_submit()`` function. If desired you can pass the > +``RTE_DMA_OP_FLAG_SUBMIT`` flag when calling ``rte_dma_copy()`` and this will > +tell the device to perform the enqueued operation and any unperformed operations > +before it. The ``RTE_DMA_OP_FLAG_SUBMIT`` flag can be passed instead of calling > +the ``rte_dma_submit()`` function for example on the last enqueue of the burst. > + > +The following code from demonstrates how to enqueue a burst of copies to the > +device and start the hardware processing of them: > + > +.. code-block:: C > + > + for (i = 0; i < BURST_SIZE; i++) { > + if (rte_dma_copy(dev_id, vchan, rte_mbuf_data_iova(srcs[i]), > + rte_mbuf_data_iova(dsts[i]), COPY_LEN, 0) < 0) { > + PRINT_ERR("Error with rte_dma_copy for buffer %u\n", i); > + return -1; > + } > + } > + if (rte_dma_submit(dev_id, vchan) < 0) { > + PRINT_ERR("Error with performing operations\n", i); > + return -1; > + } > + > +Filling an Area of Memory > +~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +The driver also has support for the ``fill`` operation, where an area > +of memory is overwritten, or filled, with a short pattern of data. > +Fill operations can be performed in much the same was as copy operations > +described above, just using the ``rte_dma_fill()`` function rather > +than the ``rte_dma_copy()`` function. Similar to the feedback on the idxd driver, I think we need to see how much of this text is already present in the generic dmadev documentation and re-use or reference that. If it's not present, then these patches should add it to the common doc, not a separate driver-specific doc. /Bruce ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 06/11] dma/ioat: add data path job submission functions 2021-09-20 13:36 ` Bruce Richardson @ 2021-09-21 16:25 ` Conor Walsh 0 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-21 16:25 UTC (permalink / raw) To: Bruce Richardson; +Cc: fengchengwen, jerinj, kevin.laatz, dev On 20/09/2021 14:36, Bruce Richardson wrote: > On Fri, Sep 17, 2021 at 03:42:22PM +0000, Conor Walsh wrote: >> Add data path functions for enqueuing and submitting operations to >> IOAT devices. >> >> Signed-off-by: Conor Walsh <conor.walsh@intel.com> >> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> >> --- >> doc/guides/dmadevs/ioat.rst | 54 ++++++++++++++++++++ >> drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ >> 2 files changed, 146 insertions(+) >> >> diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst >> index a64d67bf89..2464207e20 100644 >> --- a/doc/guides/dmadevs/ioat.rst >> +++ b/doc/guides/dmadevs/ioat.rst >> @@ -89,3 +89,57 @@ The following code shows how the device is configured in ``test_dmadev.c``: >> >> Once configured, the device can then be made ready for use by calling the >> ``rte_dma_start()`` API. >> + >> +Performing Data Copies >> +~~~~~~~~~~~~~~~~~~~~~~~ >> + >> +To perform data copies using IOAT dmadev devices, the functions >> +``rte_dma_copy()`` and ``rte_dma_submit()`` should be used. Alternatively >> +``rte_dma_copy()`` can be called with the ``RTE_DMA_OP_FLAG_SUBMIT`` flag >> +set. >> + >> +The ``rte_dma_copy()`` function enqueues a single copy to the >> +device ring for copying at a later point. The parameters to the function >> +include the device ID of the desired device, the virtual DMA channel required >> +(always 0 for IOAT), the IOVA addresses of both the source and destination >> +buffers, the length of the data to be copied and any operation flags. The >> +function will return the index of the enqueued job which can be use to >> +track that operation. >> + >> +While the ``rte_dma_copy()`` function enqueues a copy operation on the device >> +ring, the copy will not actually be performed until after the application calls >> +the ``rte_dma_submit()`` function. This function informs the device hardware >> +of the elements enqueued on the ring, and the device will begin to process them. >> +It is expected that, for efficiency reasons, a burst of operations will be >> +enqueued to the device via multiple enqueue calls between calls to the >> +``rte_dma_submit()`` function. If desired you can pass the >> +``RTE_DMA_OP_FLAG_SUBMIT`` flag when calling ``rte_dma_copy()`` and this will >> +tell the device to perform the enqueued operation and any unperformed operations >> +before it. The ``RTE_DMA_OP_FLAG_SUBMIT`` flag can be passed instead of calling >> +the ``rte_dma_submit()`` function for example on the last enqueue of the burst. >> + >> +The following code from demonstrates how to enqueue a burst of copies to the >> +device and start the hardware processing of them: >> + >> +.. code-block:: C >> + >> + for (i = 0; i < BURST_SIZE; i++) { >> + if (rte_dma_copy(dev_id, vchan, rte_mbuf_data_iova(srcs[i]), >> + rte_mbuf_data_iova(dsts[i]), COPY_LEN, 0) < 0) { >> + PRINT_ERR("Error with rte_dma_copy for buffer %u\n", i); >> + return -1; >> + } >> + } >> + if (rte_dma_submit(dev_id, vchan) < 0) { >> + PRINT_ERR("Error with performing operations\n", i); >> + return -1; >> + } >> + >> +Filling an Area of Memory >> +~~~~~~~~~~~~~~~~~~~~~~~~~~ >> + >> +The driver also has support for the ``fill`` operation, where an area >> +of memory is overwritten, or filled, with a short pattern of data. >> +Fill operations can be performed in much the same was as copy operations >> +described above, just using the ``rte_dma_fill()`` function rather >> +than the ``rte_dma_copy()`` function. > Similar to the feedback on the idxd driver, I think we need to see how much > of this text is already present in the generic dmadev documentation and > re-use or reference that. If it's not present, then these patches should > add it to the common doc, not a separate driver-specific doc. > > /Bruce I will work with Kevin to rewrite these to reduce the amount of duplication between our drivers and for future drivers in the next version. Thanks, Conor. ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 06/11] dma/ioat: add data path job submission functions 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 06/11] dma/ioat: add data path job submission functions Conor Walsh 2021-09-20 13:36 ` Bruce Richardson @ 2021-09-22 8:18 ` fengchengwen 1 sibling, 0 replies; 132+ messages in thread From: fengchengwen @ 2021-09-22 8:18 UTC (permalink / raw) To: Conor Walsh, bruce.richardson, jerinj, kevin.laatz; +Cc: dev Reviewed-by: Chengwen Feng <fengchengwen@huawei.com> On 2021/9/17 23:42, Conor Walsh wrote: > Add data path functions for enqueuing and submitting operations to > IOAT devices. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> > --- > doc/guides/dmadevs/ioat.rst | 54 ++++++++++++++++++++ > drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ > 2 files changed, 146 insertions(+) > > diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst > index a64d67bf89..2464207e20 100644 > --- a/doc/guides/dmadevs/ioat.rst > +++ b/doc/guides/dmadevs/ioat.rst > @@ -89,3 +89,57 @@ The following code shows how the device is configured in ``test_dmadev.c``: > > Once configured, the device can then be made ready for use by calling the > ``rte_dma_start()`` API. > + > +Performing Data Copies > +~~~~~~~~~~~~~~~~~~~~~~~ > + > +To perform data copies using IOAT dmadev devices, the functions > +``rte_dma_copy()`` and ``rte_dma_submit()`` should be used. Alternatively > +``rte_dma_copy()`` can be called with the ``RTE_DMA_OP_FLAG_SUBMIT`` flag > +set. > + > +The ``rte_dma_copy()`` function enqueues a single copy to the > +device ring for copying at a later point. The parameters to the function > +include the device ID of the desired device, the virtual DMA channel required > +(always 0 for IOAT), the IOVA addresses of both the source and destination > +buffers, the length of the data to be copied and any operation flags. The > +function will return the index of the enqueued job which can be use to > +track that operation. > + > +While the ``rte_dma_copy()`` function enqueues a copy operation on the device > +ring, the copy will not actually be performed until after the application calls > +the ``rte_dma_submit()`` function. This function informs the device hardware > +of the elements enqueued on the ring, and the device will begin to process them. > +It is expected that, for efficiency reasons, a burst of operations will be > +enqueued to the device via multiple enqueue calls between calls to the > +``rte_dma_submit()`` function. If desired you can pass the > +``RTE_DMA_OP_FLAG_SUBMIT`` flag when calling ``rte_dma_copy()`` and this will > +tell the device to perform the enqueued operation and any unperformed operations > +before it. The ``RTE_DMA_OP_FLAG_SUBMIT`` flag can be passed instead of calling > +the ``rte_dma_submit()`` function for example on the last enqueue of the burst. > + > +The following code from demonstrates how to enqueue a burst of copies to the > +device and start the hardware processing of them: > + > +.. code-block:: C > + > + for (i = 0; i < BURST_SIZE; i++) { > + if (rte_dma_copy(dev_id, vchan, rte_mbuf_data_iova(srcs[i]), > + rte_mbuf_data_iova(dsts[i]), COPY_LEN, 0) < 0) { > + PRINT_ERR("Error with rte_dma_copy for buffer %u\n", i); > + return -1; > + } > + } > + if (rte_dma_submit(dev_id, vchan) < 0) { > + PRINT_ERR("Error with performing operations\n", i); > + return -1; > + } > + > +Filling an Area of Memory > +~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +The driver also has support for the ``fill`` operation, where an area > +of memory is overwritten, or filled, with a short pattern of data. > +Fill operations can be performed in much the same was as copy operations > +described above, just using the ``rte_dma_fill()`` function rather > +than the ``rte_dma_copy()`` function. > diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c > index a47567ca66..edcc882d63 100644 > --- a/drivers/dma/ioat/ioat_dmadev.c > +++ b/drivers/dma/ioat/ioat_dmadev.c > @@ -5,6 +5,7 @@ > #include <rte_bus_pci.h> > #include <rte_dmadev_pmd.h> > #include <rte_malloc.h> > +#include <rte_prefetch.h> > > #include "ioat_internal.h" > > @@ -17,6 +18,12 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); > #define IOAT_PMD_NAME dmadev_ioat > #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) > > +/* IOAT operations. */ > +enum rte_ioat_ops { > + ioat_op_copy = 0, /* Standard DMA Operation */ > + ioat_op_fill /* Block Fill */ > +}; > + > /* Configure a device. */ > static int > ioat_dev_configure(struct rte_dma_dev *dev __rte_unused, const struct rte_dma_conf *dev_conf, > @@ -194,6 +201,87 @@ ioat_dev_close(struct rte_dma_dev *dev) > return 0; > } > > +/* Trigger hardware to begin performing enqueued operations. */ > +static inline void > +__submit(struct ioat_dmadev *ioat) > +{ > + *ioat->doorbell = ioat->next_write - ioat->offset; > + > + ioat->last_write = ioat->next_write; > +} > + > +/* External submit function wrapper. */ > +static int > +ioat_submit(struct rte_dma_dev *dev, uint16_t qid __rte_unused) > +{ > + struct ioat_dmadev *ioat = (struct ioat_dmadev *)dev->dev_private; > + > + __submit(ioat); > + > + return 0; > +} > + > +/* Write descriptor for enqueue. */ > +static inline int > +__write_desc(struct rte_dma_dev *dev, uint32_t op, uint64_t src, phys_addr_t dst, > + unsigned int length, uint64_t flags) > +{ > + struct ioat_dmadev *ioat = dev->dev_private; > + uint16_t ret; > + const unsigned short mask = ioat->qcfg.nb_desc - 1; > + const unsigned short read = ioat->next_read; > + unsigned short write = ioat->next_write; > + const unsigned short space = mask + read - write; > + struct ioat_dma_hw_desc *desc; > + > + if (space == 0) > + return -ENOSPC; > + > + ioat->next_write = write + 1; > + write &= mask; > + > + desc = &ioat->desc_ring[write]; > + desc->size = length; > + desc->u.control_raw = (uint32_t)((op << IOAT_CMD_OP_SHIFT) | > + (1 << IOAT_COMP_UPDATE_SHIFT)); > + > + /* In IOAT the fence ensures that all operations including the current one > + * are completed before moving on, DMAdev assumes that the fence ensures > + * all operations before the current one are completed before starting > + * the current one, so in IOAT we set the fence for the previous descriptor. > + */ > + if (flags & RTE_DMA_OP_FLAG_FENCE) > + ioat->desc_ring[(write - 1) & mask].u.control.fence = 1; > + > + desc->src_addr = src; > + desc->dest_addr = dst; > + > + rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]); > + > + ret = (uint16_t)(ioat->next_write - 1); > + > + if (flags & RTE_DMA_OP_FLAG_SUBMIT) > + __submit(ioat); > + > + return ret; > +} > + > +/* Enqueue a fill operation onto the ioat device. */ > +static int > +ioat_enqueue_fill(struct rte_dma_dev *dev, uint16_t qid __rte_unused, uint64_t pattern, > + rte_iova_t dst, unsigned int length, uint64_t flags) > +{ > + return __write_desc(dev, ioat_op_fill, pattern, dst, length, flags); > +} > + > +/* Enqueue a copy operation onto the ioat device. */ > +static int > +ioat_enqueue_copy(struct rte_dma_dev *dev, uint16_t qid __rte_unused, rte_iova_t src, > + rte_iova_t dst, unsigned int length, uint64_t flags) > +{ > + return __write_desc(dev, ioat_op_copy, src, dst, length, flags); > +} > + > /* Dump DMA device info. */ > static int > ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) > @@ -290,6 +378,10 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) > > dmadev->dev_ops = &ioat_dmadev_ops; > > + dmadev->copy = ioat_enqueue_copy; > + dmadev->fill = ioat_enqueue_fill; > + dmadev->submit = ioat_submit; > + > ioat = dmadev->data->dev_private; > ioat->dmadev = dmadev; > ioat->regs = dev->mem_resource[0].addr; > ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v4 07/11] dma/ioat: add data path completion functions 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (5 preceding siblings ...) 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 06/11] dma/ioat: add data path job submission functions Conor Walsh @ 2021-09-17 15:42 ` Conor Walsh 2021-09-20 13:39 ` Bruce Richardson 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 08/11] dma/ioat: add statistics Conor Walsh ` (3 subsequent siblings) 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-17 15:42 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add the data path functions for gathering completed operations from IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 48 ++++++++++- drivers/dma/ioat/ioat_dmadev.c | 141 +++++++++++++++++++++++++++++++++ 2 files changed, 188 insertions(+), 1 deletion(-) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 2464207e20..42ecf1fb57 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -96,7 +96,9 @@ Performing Data Copies To perform data copies using IOAT dmadev devices, the functions ``rte_dma_copy()`` and ``rte_dma_submit()`` should be used. Alternatively ``rte_dma_copy()`` can be called with the ``RTE_DMA_OP_FLAG_SUBMIT`` flag -set. +set. Once copies have been completed, the completion will be reported back when +the application calls ``rte_dma_completed()`` or +``rte_dma_completed_status()``. The ``rte_dma_copy()`` function enqueues a single copy to the device ring for copying at a later point. The parameters to the function @@ -135,6 +137,50 @@ device and start the hardware processing of them: return -1; } +To retrieve information about completed copies, the API +``rte_dma_completed()`` or ``rte_dma_completed_status ()``should be used. +``rte_dma_completed()`` will return to the application how many operations +have successfully completed up to a supplied maximum, it can also pass back the +index of the last completed operation and if a completion within the requested +batch has failed. If ``rte_dma_completed()`` passes back ``has_error`` as +``true`` then ``rte_dma_completed_status()`` should be called to retrieve +the detailed status of each completion in that batch. Once an error has occurred +``rte_dma_completed()`` will not process any more completions until +``rte_dma_completed_status()`` has been called to get the error information. +The supplied ``status`` array will contain either ``RTE_DMA_STATUS_SUCCESSFUL`` +if the operation was successful or a DMA error code if the operation failed. +The status codes supported by IOAT are: + +* ``RTE_DMA_STATUS_SUCCESSFUL``: The operation was successful. +* ``RTE_DMA_STATUS_INVALID_SRC_ADDR``: The operation failed due to an invalid source address. +* ``RTE_DMA_STATUS_INVALID_DST_ADDR``: The operation failed due to an invalid destination address. +* ``RTE_DMA_STATUS_INVALID_LENGTH``: The operation failed due to an invalid descriptor length. +* ``RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR``: The device could not read the descriptor. +* ``RTE_DMA_STATUS_ERROR_UNKNOWN``: The operation failed due to an unspecified error. + +``rte_dma_completed_status()`` can be used on it's own instead of with +``rte_dma_completed()`` but calling ``rte_dma_completed_status()`` may be +slower than ``rte_dma_completed()``. + +The following code shows how to retrieve the number of successfully completed +copies within a burst and then using ``rte_dma_completed_status()`` to check +which operation failed and reset the device: + +.. code-block:: C + + enum rte_dma_status_code status[COMP_BURST_SZ]; + uint16_t count, idx, status_count; + bool error = 0; + + count = rte_dma_completed(dev_id, vchan, COMP_BURST_SZ, &idx, &error); + + if (error){ + status_count = rte_dma_completed_status(dev_id, vchan, COMP_BURST_SZ, &idx, status); + } + +After this snippet the application would use the status array to find which copy +failed and handle it appropriately. + Filling an Area of Memory ~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index edcc882d63..fc89babc4a 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -6,6 +6,7 @@ #include <rte_dmadev_pmd.h> #include <rte_malloc.h> #include <rte_prefetch.h> +#include <rte_errno.h> #include "ioat_internal.h" @@ -339,6 +340,144 @@ ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) return 0; } +/* Returns the index of the last completed operation. */ +static inline uint16_t +__get_last_completed(const struct ioat_dmadev *ioat, int *state) +{ + /* Status register contains the address of the completed operation */ + uint64_t status = ioat->status; + + /* lower 3 bits indicate "transfer status" : active, idle, halted. + * We can ignore bit 0. + */ + *state = status & IOAT_CHANSTS_STATUS; + + /* If we are just after recovering from an error the address returned by + * status will be 0, in this case we return the offset - 1 as the last + * completed. If not return the status value minus the chainaddr which + * gives us an offset into the ring. Right shifting by 6 (divide by 64) + * gives the index of the completion from the HW point of view and adding + * the offset translates the ring index from HW to SW point of view. + */ + if ((status & ~IOAT_CHANSTS_STATUS) == 0) + return ioat->offset - 1; + + return (status - ioat->ring_addr) >> 6; +} + +/* Translates IOAT ChanERRs to DMA error codes. */ +static inline enum rte_dma_status_code +__translate_status_ioat_to_dma(uint32_t chanerr) +{ + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_SRC_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_DST_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + return RTE_DMA_STATUS_INVALID_LENGTH; + else if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + return RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR; + else + return RTE_DMA_STATUS_ERROR_UNKNOWN; +} + +/* Returns details of operations that have been completed. */ +static uint16_t +ioat_completed(struct rte_dma_dev *dev, uint16_t qid __rte_unused, const uint16_t max_ops, + uint16_t *last_idx, bool *has_error) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short last_completed, count; + int state, fails = 0; + + /* Do not do any work if there is an uncleared error. */ + if (ioat->failure != 0) { + *has_error = true; + *last_idx = ioat->next_read - 2; + return 0; + } + + last_completed = __get_last_completed(ioat, &state); + count = (last_completed + 1 - read) & mask; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || state != IOAT_CHANSTS_HALTED) { + ioat->next_read = read + count; + *last_idx = ioat->next_read - 1; + } else { + *has_error = true; + rte_errno = EIO; + ioat->failure = ioat->regs->chanerr; + ioat->next_read = read + count + 1; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + ioat_dev_dump(dev, stdout); + return 0; + } + __submit(ioat); + fails++; + *last_idx = ioat->next_read - 2; + } + + return count; +} + +/* Returns detailed status information about operations that have been completed. */ +static uint16_t +ioat_completed_status(struct rte_dma_dev *dev, uint16_t qid __rte_unused, + uint16_t max_ops, uint16_t *last_idx, enum rte_dma_status_code *status) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short count, last_completed; + uint64_t fails = 0; + int state, i; + + last_completed = __get_last_completed(ioat, &state); + count = (last_completed + 1 - read) & mask; + + for (i = 0; i < RTE_MIN(count + 1, max_ops); i++) + status[i] = RTE_DMA_STATUS_SUCCESSFUL; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || state != IOAT_CHANSTS_HALTED) + ioat->next_read = read + count; + else { + rte_errno = EIO; + status[count] = __translate_status_ioat_to_dma(ioat->regs->chanerr); + count++; + ioat->next_read = read + count; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + ioat_dev_dump(dev, stdout); + return 0; + } + __submit(ioat); + fails++; + } + + if (ioat->failure > 0) { + status[0] = __translate_status_ioat_to_dma(ioat->failure); + count = RTE_MIN(count + 1, max_ops); + ioat->failure = 0; + } + + *last_idx = ioat->next_read - 1; + + return count; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -378,6 +517,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->completed = ioat_completed; + dmadev->completed_status = ioat_completed_status; dmadev->copy = ioat_enqueue_copy; dmadev->fill = ioat_enqueue_fill; dmadev->submit = ioat_submit; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 07/11] dma/ioat: add data path completion functions 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 07/11] dma/ioat: add data path completion functions Conor Walsh @ 2021-09-20 13:39 ` Bruce Richardson 0 siblings, 0 replies; 132+ messages in thread From: Bruce Richardson @ 2021-09-20 13:39 UTC (permalink / raw) To: Conor Walsh; +Cc: fengchengwen, jerinj, kevin.laatz, dev On Fri, Sep 17, 2021 at 03:42:23PM +0000, Conor Walsh wrote: > Add the data path functions for gathering completed operations > from IOAT devices. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> > --- For the code part: Acked-by: Bruce Richardson <bruce.richardson@intel.com> However, the docs need to be made common with other drivers. ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v4 08/11] dma/ioat: add statistics 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (6 preceding siblings ...) 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 07/11] dma/ioat: add data path completion functions Conor Walsh @ 2021-09-17 15:42 ` Conor Walsh 2021-09-20 13:48 ` Bruce Richardson 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 09/11] dma/ioat: add support for vchan status function Conor Walsh ` (2 subsequent siblings) 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-17 15:42 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add statistic tracking for operations in IOAT. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 23 ++++++++++++++++++ drivers/dma/ioat/ioat_dmadev.c | 43 ++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 42ecf1fb57..69f3c56a7a 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -189,3 +189,26 @@ of memory is overwritten, or filled, with a short pattern of data. Fill operations can be performed in much the same was as copy operations described above, just using the ``rte_dma_fill()`` function rather than the ``rte_dma_copy()`` function. + +Querying Device Statistics +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The statistics from the IOAT dmadev device can be got via the +``rte_dma_stats_get()`` API in the ``rte_dmadev`` library. + +The statistics returned for each IOAT device are: + +* ``submitted``: The number of operations submitted to the device. +* ``completed``: The number of operations completed by the device (both successful and failed). +* ``errors``: The number of operations that failed. + +The stats function can be used as follows: + +.. code-block:: C + + struct rte_dma_stats stats; + if (rte_dma_stats_get(dmadev_id, vchan, &stats) >= 0) { + printf("Total submitted ops: %lu", stats.submitted); + printf("Total completed ops: %lu", stats.completed); + printf("Total failed ops: %lu", stats.errors); + } diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index fc89babc4a..f596637da4 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -77,6 +77,9 @@ ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, ioat->offset = 0; ioat->failure = 0; + /* Reset Stats. */ + ioat->stats = (struct rte_dma_stats){0}; + /* Configure descriptor ring - each one points to next. */ for (i = 0; i < ioat->qcfg.nb_desc; i++) { ioat->desc_ring[i].next = ioat->ring_addr + @@ -208,6 +211,8 @@ __submit(struct ioat_dmadev *ioat) { *ioat->doorbell = ioat->next_write - ioat->offset; + ioat->stats.submitted += (uint16_t)(ioat->next_write - ioat->last_write); + ioat->last_write = ioat->next_write; } @@ -336,6 +341,10 @@ ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) fprintf(f, " Dest: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); fprintf(f, " Next: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].next); fprintf(f, " }\n"); + fprintf(f, " Key Stats { submitted: %"PRIu64", comp: %"PRIu64", failed: %"PRIu64" }\n", + ioat->stats.submitted, + ioat->stats.completed, + ioat->stats.errors); return 0; } @@ -425,6 +434,9 @@ ioat_completed(struct rte_dma_dev *dev, uint16_t qid __rte_unused, const uint16_ *last_idx = ioat->next_read - 2; } + ioat->stats.completed += count; + ioat->stats.errors += fails; + return count; } @@ -475,9 +487,38 @@ ioat_completed_status(struct rte_dma_dev *dev, uint16_t qid __rte_unused, *last_idx = ioat->next_read - 1; + ioat->stats.completed += count; + ioat->stats.errors += fails; + return count; } +/* Retrieve the generic stats of a DMA device. */ +static int +ioat_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + struct rte_dma_stats *rte_stats, uint32_t size) +{ + struct rte_dma_stats *stats = (&((struct ioat_dmadev *)dev->dev_private)->stats); + + if (size < sizeof(rte_stats)) + return -EINVAL; + if (rte_stats == NULL) + return -EINVAL; + + *rte_stats = *stats; + return 0; +} + +/* Reset the generic stat counters for the DMA device. */ +static int +ioat_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + ioat->stats = (struct rte_dma_stats){0}; + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -489,6 +530,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_info_get = ioat_dev_info_get, .dev_start = ioat_dev_start, .dev_stop = ioat_dev_stop, + .stats_get = ioat_stats_get, + .stats_reset = ioat_stats_reset, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 08/11] dma/ioat: add statistics 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 08/11] dma/ioat: add statistics Conor Walsh @ 2021-09-20 13:48 ` Bruce Richardson 0 siblings, 0 replies; 132+ messages in thread From: Bruce Richardson @ 2021-09-20 13:48 UTC (permalink / raw) To: Conor Walsh; +Cc: fengchengwen, jerinj, kevin.laatz, dev On Fri, Sep 17, 2021 at 03:42:24PM +0000, Conor Walsh wrote: > Add statistic tracking for operations in IOAT. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> > --- > doc/guides/dmadevs/ioat.rst | 23 ++++++++++++++++++ > drivers/dma/ioat/ioat_dmadev.c | 43 ++++++++++++++++++++++++++++++++++ > 2 files changed, 66 insertions(+) > Acked-by: Bruce Richardson <bruce.richardson@intel.com> ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v4 09/11] dma/ioat: add support for vchan status function 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (7 preceding siblings ...) 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 08/11] dma/ioat: add statistics Conor Walsh @ 2021-09-17 15:42 ` Conor Walsh 2021-09-20 13:49 ` Bruce Richardson 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 10/11] dma/ioat: add burst capacity function Conor Walsh 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 11/11] devbind: move ioat device ID for ICX to dmadev category Conor Walsh 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-17 15:42 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add support for the rte_dmadev_vchan_status API call. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index f596637da4..b93289516e 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -519,6 +519,26 @@ ioat_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused) return 0; } +/* Check if the IOAT device is idle. */ +static int +ioat_vchan_status(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + enum rte_dma_vchan_status *status) +{ + int state = 0; + const struct ioat_dmadev *ioat = dev->dev_private; + const uint16_t mask = ioat->qcfg.nb_desc - 1; + const uint16_t last = __get_last_completed(ioat, &state); + + if (state == IOAT_CHANSTS_HALTED || state == IOAT_CHANSTS_SUSPENDED) + *status = RTE_DMA_VCHAN_HALTED_ERROR; + else if (last == ((ioat->next_write - 1) & mask)) + *status = RTE_DMA_VCHAN_IDLE; + else + *status = RTE_DMA_VCHAN_ACTIVE; + + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -532,6 +552,7 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_stop = ioat_dev_stop, .stats_get = ioat_stats_get, .stats_reset = ioat_stats_reset, + .vchan_status = ioat_vchan_status, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 09/11] dma/ioat: add support for vchan status function 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 09/11] dma/ioat: add support for vchan status function Conor Walsh @ 2021-09-20 13:49 ` Bruce Richardson 0 siblings, 0 replies; 132+ messages in thread From: Bruce Richardson @ 2021-09-20 13:49 UTC (permalink / raw) To: Conor Walsh; +Cc: fengchengwen, jerinj, kevin.laatz, dev On Fri, Sep 17, 2021 at 03:42:25PM +0000, Conor Walsh wrote: > Add support for the rte_dmadev_vchan_status API call. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> > --- Acked-by: Bruce Richardson <bruce.richardson@intel.com> ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v4 10/11] dma/ioat: add burst capacity function 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (8 preceding siblings ...) 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 09/11] dma/ioat: add support for vchan status function Conor Walsh @ 2021-09-17 15:42 ` Conor Walsh 2021-09-20 13:49 ` Bruce Richardson 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 11/11] devbind: move ioat device ID for ICX to dmadev category Conor Walsh 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-17 15:42 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Adds the ability to find the remaining space in the IOAT ring. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index b93289516e..e82a7424e5 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -493,6 +493,19 @@ ioat_completed_status(struct rte_dma_dev *dev, uint16_t qid __rte_unused, return count; } +/* Get the remaining capacity of the ring. */ +static uint16_t +ioat_burst_capacity(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused) +{ + struct ioat_dmadev *ioat = dev->data->dev_private; + unsigned short size = ioat->qcfg.nb_desc - 1; + unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + unsigned short space = size - (write - read); + + return space; +} + /* Retrieve the generic stats of a DMA device. */ static int ioat_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, @@ -544,6 +557,7 @@ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { static const struct rte_dma_dev_ops ioat_dmadev_ops = { + .burst_capacity = ioat_burst_capacity, .dev_close = ioat_dev_close, .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 10/11] dma/ioat: add burst capacity function 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 10/11] dma/ioat: add burst capacity function Conor Walsh @ 2021-09-20 13:49 ` Bruce Richardson 0 siblings, 0 replies; 132+ messages in thread From: Bruce Richardson @ 2021-09-20 13:49 UTC (permalink / raw) To: Conor Walsh; +Cc: fengchengwen, jerinj, kevin.laatz, dev On Fri, Sep 17, 2021 at 03:42:26PM +0000, Conor Walsh wrote: > Adds the ability to find the remaining space in the IOAT ring. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> > --- Acked-by: Bruce Richardson <bruce.richardson@intel.com> ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v4 11/11] devbind: move ioat device ID for ICX to dmadev category 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices Conor Walsh ` (9 preceding siblings ...) 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 10/11] dma/ioat: add burst capacity function Conor Walsh @ 2021-09-17 15:42 ` Conor Walsh 2021-09-20 13:53 ` Bruce Richardson 10 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-09-17 15:42 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Move Intel IOAT devices on Ice Lake systems from Misc to DMA devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- usertools/dpdk-devbind.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index 98b698ccc0..afebc8cb62 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -69,14 +69,13 @@ network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] -dma_devices = [intel_idxd_spr] +dma_devices = [intel_idxd_spr, intel_ioat_icx] eventdev_devices = [cavium_sso, cavium_tim, intel_dlb, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] regex_devices = [octeontx2_ree] misc_devices = [cnxk_bphy, cnxk_bphy_cgx, intel_ioat_bdw, intel_ioat_skx, - intel_ioat_icx, intel_ntb_skx, intel_ntb_icx, - octeontx2_dma] + intel_ntb_skx, intel_ntb_icx, octeontx2_dma] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 11/11] devbind: move ioat device ID for ICX to dmadev category 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 11/11] devbind: move ioat device ID for ICX to dmadev category Conor Walsh @ 2021-09-20 13:53 ` Bruce Richardson 2021-09-21 16:26 ` Conor Walsh 0 siblings, 1 reply; 132+ messages in thread From: Bruce Richardson @ 2021-09-20 13:53 UTC (permalink / raw) To: Conor Walsh; +Cc: fengchengwen, jerinj, kevin.laatz, dev On Fri, Sep 17, 2021 at 03:42:27PM +0000, Conor Walsh wrote: > Move Intel IOAT devices on Ice Lake systems from Misc to DMA devices. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> > --- > usertools/dpdk-devbind.py | 5 ++--- > 1 file changed, 2 insertions(+), 3 deletions(-) > > diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py > index 98b698ccc0..afebc8cb62 100755 > --- a/usertools/dpdk-devbind.py > +++ b/usertools/dpdk-devbind.py > @@ -69,14 +69,13 @@ > network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] > baseband_devices = [acceleration_class] > crypto_devices = [encryption_class, intel_processor_class] > -dma_devices = [intel_idxd_spr] > +dma_devices = [intel_idxd_spr, intel_ioat_icx] > eventdev_devices = [cavium_sso, cavium_tim, intel_dlb, octeontx2_sso] > mempool_devices = [cavium_fpa, octeontx2_npa] > compress_devices = [cavium_zip] > regex_devices = [octeontx2_ree] > misc_devices = [cnxk_bphy, cnxk_bphy_cgx, intel_ioat_bdw, intel_ioat_skx, > - intel_ioat_icx, intel_ntb_skx, intel_ntb_icx, > - octeontx2_dma] > + intel_ntb_skx, intel_ntb_icx, octeontx2_dma] > I think the ioat_bdw and ioat_skx elements should also go down as DMA devices. With that change: Reviewed-by: Bruce Richardson <bruce.richardson@intel.com> ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v4 11/11] devbind: move ioat device ID for ICX to dmadev category 2021-09-20 13:53 ` Bruce Richardson @ 2021-09-21 16:26 ` Conor Walsh 0 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-21 16:26 UTC (permalink / raw) To: Bruce Richardson; +Cc: fengchengwen, jerinj, kevin.laatz, dev > On Fri, Sep 17, 2021 at 03:42:27PM +0000, Conor Walsh wrote: >> Move Intel IOAT devices on Ice Lake systems from Misc to DMA devices. >> >> Signed-off-by: Conor Walsh <conor.walsh@intel.com> >> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> >> --- >> usertools/dpdk-devbind.py | 5 ++--- >> 1 file changed, 2 insertions(+), 3 deletions(-) >> >> diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py >> index 98b698ccc0..afebc8cb62 100755 >> --- a/usertools/dpdk-devbind.py >> +++ b/usertools/dpdk-devbind.py >> @@ -69,14 +69,13 @@ >> network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] >> baseband_devices = [acceleration_class] >> crypto_devices = [encryption_class, intel_processor_class] >> -dma_devices = [intel_idxd_spr] >> +dma_devices = [intel_idxd_spr, intel_ioat_icx] >> eventdev_devices = [cavium_sso, cavium_tim, intel_dlb, octeontx2_sso] >> mempool_devices = [cavium_fpa, octeontx2_npa] >> compress_devices = [cavium_zip] >> regex_devices = [octeontx2_ree] >> misc_devices = [cnxk_bphy, cnxk_bphy_cgx, intel_ioat_bdw, intel_ioat_skx, >> - intel_ioat_icx, intel_ntb_skx, intel_ntb_icx, >> - octeontx2_dma] >> + intel_ntb_skx, intel_ntb_icx, octeontx2_dma] >> > I think the ioat_bdw and ioat_skx elements should also go down as DMA > devices. > > With that change: > > Reviewed-by: Bruce Richardson <bruce.richardson@intel.com> I will change this in v5 and deprecate the rawdev IOAT driver. Thanks for the review Bruce, Conor. ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh ` (9 preceding siblings ...) 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices Conor Walsh @ 2021-09-24 14:33 ` Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 01/12] dma/ioat: add device probe and removal functions Conor Walsh ` (11 more replies) 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (2 subsequent siblings) 13 siblings, 12 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-24 14:33 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh This patchset adds a dmadev driver and associated documentation to support Intel QuickData Technology devices, part of the Intel I/O Acceleration Technology (Intel I/OAT). This driver is intended to ultimately replace the current IOAT part of the IOAT rawdev driver. This patchset passes all the driver tests added in the dmadev test suite. NOTE: This patchset has several dependencies: - v23 of the dmadev set [1] - v6 of the dmadev test suite [2] - v6 of the IDXD driver [3] [1] http://patches.dpdk.org/project/dpdk/list/?series=19140 [2] http://patches.dpdk.org/project/dpdk/list/?series=19138 [3] http://patches.dpdk.org/project/dpdk/list/?series=19144 --- v5: - Updated to v23 of the dmadev lib. - Removed experimental tag for driver from MAINTAINERS. - Seperated IOAT and IDXD announcements in release notes. - Added missing check for rte_dma_get_dev_id in destroy. - Fixed memleak in destroy caused by NULL pointer. - Rewrote part of the docs to reduce duplication with DMA and IDXD. - Added patch to deprecate the rawdev IOAT driver. - Reworked destroy and close functions. - Added RTE_DMA_CAPA_HANDLES_ERRORS flag for IOAT versions >=3.4. - Other minor changes to IOAT driver. v4: - Changes needed to update from dmadev v21 to v22. - Fixed 32-bit build. - Made stats reset logic easier to understand. v3: - Added burst capacity function. - Stop function now waits for suspend rather than just using a sleep. - Changed from vchan idle to vchan status function. - Other minor changes to update from dmadev v19 to v21. v2: - Rebased on the above patchsets. - Added support for the vchan idle function. - Stop function now suspends IOAT channel to allow for reconfig. - dmadev_autotest can now be run multiple times using the IOAT driver without errors. - Added devbind updates for DMA devices - Removed some logically dead code found by coverity in the create function. Conor Walsh (12): dma/ioat: add device probe and removal functions dma/ioat: create dmadev instances on PCI probe dma/ioat: add datapath structures dma/ioat: add configuration functions dma/ioat: add start and stop functions dma/ioat: add data path job submission functions dma/ioat: add data path completion functions dma/ioat: add statistics dma/ioat: add support for vchan status function dma/ioat: add burst capacity function devbind: move ioat device IDs to dmadev category raw/ioat: deprecate ioat rawdev driver MAINTAINERS | 8 +- doc/guides/dmadevs/index.rst | 2 + doc/guides/dmadevs/ioat.rst | 131 +++++ doc/guides/rawdevs/ioat.rst | 4 + doc/guides/rel_notes/release_21_11.rst | 6 + drivers/dma/ioat/ioat_dmadev.c | 738 +++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 295 ++++++++++ drivers/dma/ioat/ioat_internal.h | 47 ++ drivers/dma/ioat/meson.build | 7 + drivers/dma/ioat/version.map | 3 + drivers/dma/meson.build | 1 + usertools/dpdk-devbind.py | 6 +- 12 files changed, 1243 insertions(+), 5 deletions(-) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v5 01/12] dma/ioat: add device probe and removal functions 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh @ 2021-09-24 14:33 ` Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 02/12] dma/ioat: create dmadev instances on PCI probe Conor Walsh ` (10 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-24 14:33 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add the basic device probe/remove skeleton code and initial documentation for new IOAT DMA driver. Maintainers update is also included in this patch. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Reviewed-by: Chengwen Feng <fengchengwen@huawei.com> --- MAINTAINERS | 6 +++ doc/guides/dmadevs/index.rst | 2 + doc/guides/dmadevs/ioat.rst | 69 ++++++++++++++++++++++++++ doc/guides/rel_notes/release_21_11.rst | 6 +++ drivers/dma/ioat/ioat_dmadev.c | 69 ++++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 35 +++++++++++++ drivers/dma/ioat/ioat_internal.h | 20 ++++++++ drivers/dma/ioat/meson.build | 7 +++ drivers/dma/ioat/version.map | 3 ++ drivers/dma/meson.build | 1 + 10 files changed, 218 insertions(+) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map diff --git a/MAINTAINERS b/MAINTAINERS index 497219e948..ccabba9169 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1209,6 +1209,12 @@ M: Kevin Laatz <kevin.laatz@intel.com> F: drivers/dma/idxd/ F: doc/guides/dmadevs/idxd.rst +Intel IOAT +M: Bruce Richardson <bruce.richardson@intel.com> +M: Conor Walsh <conor.walsh@intel.com> +F: drivers/dma/ioat/ +F: doc/guides/dmadevs/ioat.rst + RegEx Drivers ------------- diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst index 5d4abf880e..c59f4b5c92 100644 --- a/doc/guides/dmadevs/index.rst +++ b/doc/guides/dmadevs/index.rst @@ -12,3 +12,5 @@ an application through DMA API. :numbered: idxd + ioat + diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst new file mode 100644 index 0000000000..9ae1d8a2ad --- /dev/null +++ b/doc/guides/dmadevs/ioat.rst @@ -0,0 +1,69 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2021 Intel Corporation. + +.. include:: <isonum.txt> + +IOAT DMA Device Driver +======================= + +The ``ioat`` dmadev driver provides a poll-mode driver (PMD) for Intel\ +|reg| QuickData Technology which is part of part of Intel\ |reg| I/O +Acceleration Technology (`Intel I/OAT +<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_). +This PMD, when used on supported hardware, allows data copies, for example, +cloning packet data, to be accelerated by IOAT hardware rather than having to +be done by software, freeing up CPU cycles for other tasks. + +Hardware Requirements +---------------------- + +The ``dpdk-devbind.py`` script, included with DPDK, can be used to show the +presence of supported hardware. Running ``dpdk-devbind.py --status-dev dma`` +will show all the DMA devices on the system, IOAT devices are included in this +list. For Intel\ |reg| IOAT devices, the hardware will often be listed as +"Crystal Beach DMA", or "CBDMA" or on some newer systems '0b00' due to the +absence of pci-id database entries for them at this point. + +.. note:: + Error handling is not supported by this driver on hardware prior to + Intel Ice Lake. Unsupported systems include Broadwell, Skylake and + Cascade Lake. + +Compilation +------------ + +For builds using ``meson`` and ``ninja``, the driver will be built when the +target platform is x86-based. No additional compilation steps are necessary. + +Device Setup +------------- + +Intel\ |reg| IOAT devices will need to be bound to a suitable DPDK-supported +user-space IO driver such as ``vfio-pci`` in order to be used by DPDK. + +The ``dpdk-devbind.py`` script can be used to view the state of the devices using:: + + $ dpdk-devbind.py --status-dev dma + +The ``dpdk-devbind.py`` script can also be used to bind devices to a suitable driver. +For example:: + + $ dpdk-devbind.py -b vfio-pci 00:01.0 00:01.1 + +Device Probing and Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For devices bound to a suitable DPDK-supported driver (``vfio-pci``), the HW +devices will be found as part of the device scan done at application +initialization time without the need to pass parameters to the application. + +If the application does not require all the devices available an allowlist can +be used in the same way that other DPDK devices use them. + +For example:: + + $ dpdk-test -a <b:d:f> + +Once probed successfully, the device will appear as a ``dmadev``, that is a +"DMA device type" inside DPDK, and can be accessed using APIs from the +``rte_dmadev`` library. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index c980e729f8..e34957069f 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -103,6 +103,12 @@ New Features The IDXD dmadev driver provide device drivers for the Intel DSA devices. This device driver can be used through the generic dmadev API. +* **Added IOAT dmadev driver implementation.** + + The Intel I/O Acceleration Technology (IOAT) dmadev driver provides a device + driver for Intel IOAT devices such as Crystal Beach DMA (CBDMA) on Ice Lake, + Skylake and Broadwell. This device driver can be used through the generic dmadev API. + Removed Items ------------- diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c new file mode 100644 index 0000000000..f3491d45b1 --- /dev/null +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#include <rte_bus_pci.h> +#include <rte_dmadev_pmd.h> + +#include "ioat_internal.h" + +static struct rte_pci_driver ioat_pmd_drv; + +RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); + +#define IOAT_PMD_NAME dmadev_ioat +#define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) + +/* Probe DMA device. */ +static int +ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); + + dev->device.driver = &drv->driver; + return 0; +} + +/* Remove DMA device. */ +static int +ioat_dmadev_remove(struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + + IOAT_PMD_INFO("Closing %s on NUMA node %d", + name, dev->device.numa_node); + + return 0; +} + +static const struct rte_pci_id pci_id_ioat_map[] = { + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_SKX) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX0) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX1) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX2) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX3) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX4) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX5) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX6) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX7) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXE) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXF) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_ICX) }, + { .vendor_id = 0, /* sentinel */ }, +}; + +static struct rte_pci_driver ioat_pmd_drv = { + .id_table = pci_id_ioat_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC, + .probe = ioat_dmadev_probe, + .remove = ioat_dmadev_remove, +}; + +RTE_PMD_REGISTER_PCI(IOAT_PMD_NAME, ioat_pmd_drv); +RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_NAME, pci_id_ioat_map); +RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_NAME, "* igb_uio | uio_pci_generic | vfio-pci"); diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h new file mode 100644 index 0000000000..eeabba41ef --- /dev/null +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#ifndef IOAT_HW_DEFS_H +#define IOAT_HW_DEFS_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> + +#define IOAT_VER_3_0 0x30 +#define IOAT_VER_3_3 0x33 + +#define IOAT_VENDOR_ID 0x8086 +#define IOAT_DEVICE_ID_SKX 0x2021 +#define IOAT_DEVICE_ID_BDX0 0x6f20 +#define IOAT_DEVICE_ID_BDX1 0x6f21 +#define IOAT_DEVICE_ID_BDX2 0x6f22 +#define IOAT_DEVICE_ID_BDX3 0x6f23 +#define IOAT_DEVICE_ID_BDX4 0x6f24 +#define IOAT_DEVICE_ID_BDX5 0x6f25 +#define IOAT_DEVICE_ID_BDX6 0x6f26 +#define IOAT_DEVICE_ID_BDX7 0x6f27 +#define IOAT_DEVICE_ID_BDXE 0x6f2E +#define IOAT_DEVICE_ID_BDXF 0x6f2F +#define IOAT_DEVICE_ID_ICX 0x0b00 + +#ifdef __cplusplus +} +#endif + +#endif /* IOAT_HW_DEFS_H */ diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h new file mode 100644 index 0000000000..f1ec12a919 --- /dev/null +++ b/drivers/dma/ioat/ioat_internal.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021 Intel Corporation + */ + +#ifndef _IOAT_INTERNAL_H_ +#define _IOAT_INTERNAL_H_ + +#include "ioat_hw_defs.h" + +extern int ioat_pmd_logtype; + +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + ioat_pmd_logtype, "IOAT: %s(): " fmt "\n", __func__, ##args) + +#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) +#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) +#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) +#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) + +#endif /* _IOAT_INTERNAL_H_ */ diff --git a/drivers/dma/ioat/meson.build b/drivers/dma/ioat/meson.build new file mode 100644 index 0000000000..d67fac96fb --- /dev/null +++ b/drivers/dma/ioat/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021 Intel Corporation + +build = dpdk_conf.has('RTE_ARCH_X86') +reason = 'only supported on x86' +sources = files('ioat_dmadev.c') +deps += ['bus_pci', 'dmadev'] diff --git a/drivers/dma/ioat/version.map b/drivers/dma/ioat/version.map new file mode 100644 index 0000000000..c2e0723b4c --- /dev/null +++ b/drivers/dma/ioat/version.map @@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +}; diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build index 411be7a240..a69418ce9b 100644 --- a/drivers/dma/meson.build +++ b/drivers/dma/meson.build @@ -3,6 +3,7 @@ drivers = [ 'idxd', + 'ioat', 'skeleton', ] std_deps = ['dmadev'] -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v5 02/12] dma/ioat: create dmadev instances on PCI probe 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 01/12] dma/ioat: add device probe and removal functions Conor Walsh @ 2021-09-24 14:33 ` Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 03/12] dma/ioat: add datapath structures Conor Walsh ` (9 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-24 14:33 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh When a suitable device is found during the PCI probe, create a dmadev instance for each channel. Internal structures and HW definitions required for device creation are also included. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 102 ++++++++++++++++++++++++++++++- drivers/dma/ioat/ioat_hw_defs.h | 45 ++++++++++++++ drivers/dma/ioat/ioat_internal.h | 27 ++++++++ 3 files changed, 172 insertions(+), 2 deletions(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index f3491d45b1..df3c72363a 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -4,6 +4,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> +#include <rte_malloc.h> #include "ioat_internal.h" @@ -14,6 +15,103 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Create a DMA device. */ +static int +ioat_dmadev_create(const char *name, struct rte_pci_device *dev) +{ + static const struct rte_dma_dev_ops ioat_dmadev_ops = { }; + + struct rte_dma_dev *dmadev = NULL; + struct ioat_dmadev *ioat = NULL; + int retry = 0; + + if (!name) { + IOAT_PMD_ERR("Invalid name of the device!"); + return -EINVAL; + } + + /* Allocate device structure. */ + dmadev = rte_dma_pmd_allocate(name, dev->device.numa_node, sizeof(*ioat)); + if (dmadev == NULL) { + IOAT_PMD_ERR("Unable to allocate dma device"); + return -ENOMEM; + } + + dmadev->device = &dev->device; + + dmadev->dev_private = dmadev->data->dev_private; + + dmadev->dev_ops = &ioat_dmadev_ops; + + ioat = dmadev->data->dev_private; + ioat->regs = dev->mem_resource[0].addr; + ioat->doorbell = &ioat->regs->dmacount; + ioat->qcfg.nb_desc = 0; + ioat->desc_ring = NULL; + ioat->version = ioat->regs->cbver; + + /* Do device initialization - reset and set error behaviour. */ + if (ioat->regs->chancnt != 1) + IOAT_PMD_WARN("%s: Channel count == %d\n", __func__, + ioat->regs->chancnt); + + /* Locked by someone else. */ + if (ioat->regs->chanctrl & IOAT_CHANCTRL_CHANNEL_IN_USE) { + IOAT_PMD_WARN("%s: Channel appears locked\n", __func__); + ioat->regs->chanctrl = 0; + } + + /* clear any previous errors */ + if (ioat->regs->chanerr != 0) { + uint32_t val = ioat->regs->chanerr; + ioat->regs->chanerr = val; + } + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + rte_delay_ms(1); + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + rte_delay_ms(1); + while (ioat->regs->chancmd & IOAT_CHANCMD_RESET) { + ioat->regs->chainaddr = 0; + rte_delay_ms(1); + if (++retry >= 200) { + IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=%#"PRIx8 + ", CHANSTS=%#"PRIx64", CHANERR=%#"PRIx32"\n", + __func__, + ioat->regs->chancmd, + ioat->regs->chansts, + ioat->regs->chanerr); + rte_dma_pmd_release(name); + return -EIO; + } + } + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + + dmadev->state = RTE_DMA_DEV_READY; + + return 0; + +} + +/* Destroy a DMA device. */ +static int +ioat_dmadev_destroy(const char *name) +{ + int ret; + + if (!name) { + IOAT_PMD_ERR("Invalid device name"); + return -EINVAL; + } + + ret = rte_dma_pmd_release(name); + if (ret) + IOAT_PMD_DEBUG("Device cleanup failed"); + + return 0; +} + /* Probe DMA device. */ static int ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) @@ -24,7 +122,7 @@ ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); dev->device.driver = &drv->driver; - return 0; + return ioat_dmadev_create(name, dev); } /* Remove DMA device. */ @@ -38,7 +136,7 @@ ioat_dmadev_remove(struct rte_pci_device *dev) IOAT_PMD_INFO("Closing %s on NUMA node %d", name, dev->device.numa_node); - return 0; + return ioat_dmadev_destroy(name); } static const struct rte_pci_id pci_id_ioat_map[] = { diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index eeabba41ef..73bdf548b3 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -11,6 +11,8 @@ extern "C" { #include <stdint.h> +#define IOAT_PCI_CHANERR_INT_OFFSET 0x180 + #define IOAT_VER_3_0 0x30 #define IOAT_VER_3_3 0x33 @@ -28,6 +30,49 @@ extern "C" { #define IOAT_DEVICE_ID_BDXF 0x6f2F #define IOAT_DEVICE_ID_ICX 0x0b00 +#define IOAT_COMP_UPDATE_SHIFT 3 +#define IOAT_CMD_OP_SHIFT 24 + +/* DMA Channel Registers */ +#define IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK 0xF000 +#define IOAT_CHANCTRL_COMPL_DCA_EN 0x0200 +#define IOAT_CHANCTRL_CHANNEL_IN_USE 0x0100 +#define IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL 0x0020 +#define IOAT_CHANCTRL_ERR_INT_EN 0x0010 +#define IOAT_CHANCTRL_ANY_ERR_ABORT_EN 0x0008 +#define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 +#define IOAT_CHANCTRL_INT_REARM 0x0001 + +struct ioat_registers { + uint8_t chancnt; + uint8_t xfercap; + uint8_t genctrl; + uint8_t intrctrl; + uint32_t attnstatus; + uint8_t cbver; /* 0x08 */ + uint8_t reserved4[0x3]; /* 0x09 */ + uint16_t intrdelay; /* 0x0C */ + uint16_t cs_status; /* 0x0E */ + uint32_t dmacapability; /* 0x10 */ + uint8_t reserved5[0x6C]; /* 0x14 */ + uint16_t chanctrl; /* 0x80 */ + uint8_t reserved6[0x2]; /* 0x82 */ + uint8_t chancmd; /* 0x84 */ + uint8_t reserved3[1]; /* 0x85 */ + uint16_t dmacount; /* 0x86 */ + uint64_t chansts; /* 0x88 */ + uint64_t chainaddr; /* 0x90 */ + uint64_t chancmp; /* 0x98 */ + uint8_t reserved2[0x8]; /* 0xA0 */ + uint32_t chanerr; /* 0xA8 */ + uint32_t chanerrmask; /* 0xAC */ +} __rte_packed; + +#define IOAT_CHANCMD_RESET 0x20 +#define IOAT_CHANCMD_SUSPEND 0x04 + +#define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ + #ifdef __cplusplus } #endif diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h index f1ec12a919..83ef5973f5 100644 --- a/drivers/dma/ioat/ioat_internal.h +++ b/drivers/dma/ioat/ioat_internal.h @@ -7,6 +7,33 @@ #include "ioat_hw_defs.h" +struct ioat_dmadev { + struct rte_dma_dev_data *dmadev; + struct rte_dma_vchan_conf qcfg; + struct rte_dma_stats stats; + + volatile uint16_t *doorbell __rte_cache_aligned; + phys_addr_t status_addr; + phys_addr_t ring_addr; + + struct ioat_dma_hw_desc *desc_ring; + + unsigned short next_read; + unsigned short next_write; + unsigned short last_write; /* Used to compute submitted count. */ + unsigned short offset; /* Used after a device recovery when counts -> 0. */ + unsigned int failure; /* Used to store chanerr for error handling. */ + + /* To report completions, the device will write status back here. */ + volatile uint64_t status __rte_cache_aligned; + + /* Pointer to the register bar. */ + volatile struct ioat_registers *regs; + + /* Store the IOAT version. */ + uint8_t version; +}; + extern int ioat_pmd_logtype; #define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v5 03/12] dma/ioat: add datapath structures 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 01/12] dma/ioat: add device probe and removal functions Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 02/12] dma/ioat: create dmadev instances on PCI probe Conor Walsh @ 2021-09-24 14:33 ` Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 04/12] dma/ioat: add configuration functions Conor Walsh ` (8 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-24 14:33 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add data structures required for the data path of IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 63 +++++++++- drivers/dma/ioat/ioat_hw_defs.h | 215 ++++++++++++++++++++++++++++++++ 2 files changed, 277 insertions(+), 1 deletion(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index df3c72363a..b132283ba5 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -15,11 +15,72 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Dump DMA device info. */ +static int +ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint64_t chansts_masked = ioat->regs->chansts & IOAT_CHANSTS_STATUS; + uint32_t chanerr = ioat->regs->chanerr; + uint64_t mask = (ioat->qcfg.nb_desc - 1); + char ver = ioat->version; + fprintf(f, "========= IOAT =========\n"); + fprintf(f, " IOAT version: %d.%d\n", ver >> 4, ver & 0xF); + fprintf(f, " Channel status: %s [0x%"PRIx64"]\n", + chansts_readable[chansts_masked], chansts_masked); + fprintf(f, " ChainADDR: 0x%"PRIu64"\n", ioat->regs->chainaddr); + if (chanerr == 0) { + fprintf(f, " No Channel Errors\n"); + } else { + fprintf(f, " ChanERR: 0x%"PRIu32"\n", chanerr); + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + fprintf(f, " Invalid Source Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + fprintf(f, " Invalid Destination Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + fprintf(f, " Invalid Descriptor Length\n"); + if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + fprintf(f, " Descriptor Read Error\n"); + if ((chanerr & ~(IOAT_CHANERR_INVALID_SRC_ADDR_MASK | + IOAT_CHANERR_INVALID_DST_ADDR_MASK | + IOAT_CHANERR_INVALID_LENGTH_MASK | + IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK)) != 0) + fprintf(f, " Unknown Error(s)\n"); + } + fprintf(f, "== Private Data ==\n"); + fprintf(f, " Config: { ring_size: %u }\n", ioat->qcfg.nb_desc); + fprintf(f, " Status: 0x%"PRIx64"\n", ioat->status); + fprintf(f, " Status IOVA: 0x%"PRIx64"\n", ioat->status_addr); + fprintf(f, " Status ADDR: %p\n", &ioat->status); + fprintf(f, " Ring IOVA: 0x%"PRIx64"\n", ioat->ring_addr); + fprintf(f, " Ring ADDR: 0x%"PRIx64"\n", ioat->desc_ring[0].next-64); + fprintf(f, " Next write: %"PRIu16"\n", ioat->next_write); + fprintf(f, " Next read: %"PRIu16"\n", ioat->next_read); + struct ioat_dma_hw_desc *desc_ring = &ioat->desc_ring[(ioat->next_write - 1) & mask]; + fprintf(f, " Last Descriptor Written {\n"); + fprintf(f, " Size: %"PRIu32"\n", desc_ring->size); + fprintf(f, " Control: 0x%"PRIx32"\n", desc_ring->u.control_raw); + fprintf(f, " Src: 0x%"PRIx64"\n", desc_ring->src_addr); + fprintf(f, " Dest: 0x%"PRIx64"\n", desc_ring->dest_addr); + fprintf(f, " Next: 0x%"PRIx64"\n", desc_ring->next); + fprintf(f, " }\n"); + fprintf(f, " Next Descriptor {\n"); + fprintf(f, " Size: %"PRIu32"\n", ioat->desc_ring[ioat->next_read & mask].size); + fprintf(f, " Src: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].src_addr); + fprintf(f, " Dest: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); + fprintf(f, " Next: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].next); + fprintf(f, " }\n"); + + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { - static const struct rte_dma_dev_ops ioat_dmadev_ops = { }; + static const struct rte_dma_dev_ops ioat_dmadev_ops = { + .dev_dump = ioat_dev_dump, + }; struct rte_dma_dev *dmadev = NULL; struct ioat_dmadev *ioat = NULL; diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index 73bdf548b3..dc3493a78f 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -15,6 +15,7 @@ extern "C" { #define IOAT_VER_3_0 0x30 #define IOAT_VER_3_3 0x33 +#define IOAT_VER_3_4 0x34 #define IOAT_VENDOR_ID 0x8086 #define IOAT_DEVICE_ID_SKX 0x2021 @@ -43,6 +44,14 @@ extern "C" { #define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 #define IOAT_CHANCTRL_INT_REARM 0x0001 +/* DMA Channel Capabilities */ +#define IOAT_DMACAP_PB (1 << 0) +#define IOAT_DMACAP_DCA (1 << 4) +#define IOAT_DMACAP_BFILL (1 << 6) +#define IOAT_DMACAP_XOR (1 << 8) +#define IOAT_DMACAP_PQ (1 << 9) +#define IOAT_DMACAP_DMA_DIF (1 << 10) + struct ioat_registers { uint8_t chancnt; uint8_t xfercap; @@ -71,8 +80,214 @@ struct ioat_registers { #define IOAT_CHANCMD_RESET 0x20 #define IOAT_CHANCMD_SUSPEND 0x04 +#define IOAT_CHANSTS_STATUS 0x7ULL +#define IOAT_CHANSTS_ACTIVE 0x0 +#define IOAT_CHANSTS_IDLE 0x1 +#define IOAT_CHANSTS_SUSPENDED 0x2 +#define IOAT_CHANSTS_HALTED 0x3 +#define IOAT_CHANSTS_ARMED 0x4 + +#define IOAT_CHANERR_INVALID_SRC_ADDR_MASK (1 << 0) +#define IOAT_CHANERR_INVALID_DST_ADDR_MASK (1 << 1) +#define IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK (1 << 8) +#define IOAT_CHANERR_INVALID_LENGTH_MASK (1 << 10) + +const char *chansts_readable[] = { + "ACTIVE", /* 0x0 */ + "IDLE", /* 0x1 */ + "SUSPENDED", /* 0x2 */ + "HALTED", /* 0x3 */ + "ARMED" /* 0x4 */ +}; + +#define IOAT_CHANSTS_UNAFFILIATED_ERROR 0x8ULL +#define IOAT_CHANSTS_SOFT_ERROR 0x10ULL + +#define IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK (~0x3FULL) + #define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ +struct ioat_dma_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t null: 1; + uint32_t src_page_break: 1; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_COPY 0x00 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t reserved2; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_fill_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t reserved: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t reserved2: 2; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t reserved3: 15; +#define IOAT_OP_FILL 0x01 + uint32_t op: 8; + } control; + } u; + uint64_t src_data; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t next_dest_addr; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_xor_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_XOR 0x87 +#define IOAT_OP_XOR_VAL 0x88 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint64_t src_addr4; + uint64_t src_addr5; +}; + +struct ioat_xor_ext_hw_desc { + uint64_t src_addr6; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t next; + uint64_t reserved[4]; +}; + +struct ioat_pq_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 11; +#define IOAT_OP_PQ 0x89 +#define IOAT_OP_PQ_VAL 0x8a + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint8_t coef[8]; + uint64_t q_addr; +}; + +struct ioat_pq_ext_hw_desc { + uint64_t src_addr4; + uint64_t src_addr5; + uint64_t src_addr6; + uint64_t next; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t reserved[2]; +}; + +struct ioat_pq_update_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_cnt: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 3; + uint32_t coef: 8; +#define IOAT_OP_PQ_UP 0x8b + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t p_src; + uint64_t q_src; + uint64_t q_addr; +}; + +union ioat_hw_desc { + struct ioat_dma_hw_desc dma; + struct ioat_fill_hw_desc fill; + struct ioat_xor_hw_desc xor_desc; + struct ioat_xor_ext_hw_desc xor_ext; + struct ioat_pq_hw_desc pq; + struct ioat_pq_ext_hw_desc pq_ext; + struct ioat_pq_update_hw_desc pq_update; +}; + +#define GENSTS_DEV_STATE_MASK 0x03 +#define CMDSTATUS_ACTIVE_SHIFT 31 +#define CMDSTATUS_ACTIVE_MASK (1 << 31) +#define CMDSTATUS_ERR_MASK 0xFF + #ifdef __cplusplus } #endif -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v5 04/12] dma/ioat: add configuration functions 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (2 preceding siblings ...) 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 03/12] dma/ioat: add datapath structures Conor Walsh @ 2021-09-24 14:33 ` Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 05/12] dma/ioat: add start and stop functions Conor Walsh ` (7 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-24 14:33 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add functions for device configuration. The info_get and close functions are included here also. info_get can be useful for checking successful configuration and close is used by the dmadev api when releasing a configured device. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 19 ++++++ drivers/dma/ioat/ioat_dmadev.c | 107 +++++++++++++++++++++++++++++++++ 2 files changed, 126 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 9ae1d8a2ad..b1f847d273 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -67,3 +67,22 @@ For example:: Once probed successfully, the device will appear as a ``dmadev``, that is a "DMA device type" inside DPDK, and can be accessed using APIs from the ``rte_dmadev`` library. + +Using IOAT DMAdev Devices +-------------------------- + +To use IOAT devices from an application, the ``dmadev`` API can be used. + +Device Configuration +~~~~~~~~~~~~~~~~~~~~~ + +Refer to the :ref:`Device Configuration <dmadev_device_configuration>` and +:ref:`Configuration of Virtual DMA Channels <dmadev_vchan_configuration>` sections +of the dmadev library documentation for details on device configuration API usage. + +IOAT configuration requirements: + +* ``ring_size`` must be a power of two, between 64 and 4096. +* Only one ``vchan`` is supported per device. +* Silent mode is not supported. +* The transfer direction must be set to ``RTE_DMA_DIR_MEM_TO_MEM`` to copy from memory to memory. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index b132283ba5..92c4e2b04f 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -12,9 +12,112 @@ static struct rte_pci_driver ioat_pmd_drv; RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); +#define DESC_SZ sizeof(struct ioat_dma_hw_desc) + #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Configure a device. */ +static int +ioat_dev_configure(struct rte_dma_dev *dev __rte_unused, const struct rte_dma_conf *dev_conf, + uint32_t conf_sz) +{ + if (sizeof(struct rte_dma_conf) != conf_sz) + return -EINVAL; + + if (dev_conf->nb_vchans != 1) + return -EINVAL; + + return 0; +} + +/* Setup a virtual channel for IOAT, only 1 vchan is supported. */ +static int +ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + const struct rte_dma_vchan_conf *qconf, uint32_t qconf_sz) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint16_t max_desc = qconf->nb_desc; + int i; + + if (sizeof(struct rte_dma_vchan_conf) != qconf_sz) + return -EINVAL; + + ioat->qcfg = *qconf; + + if (!rte_is_power_of_2(max_desc)) { + max_desc = rte_align32pow2(max_desc); + IOAT_PMD_DEBUG("DMA dev %u using %u descriptors", dev->data->dev_id, max_desc); + ioat->qcfg.nb_desc = max_desc; + } + + /* In case we are reconfiguring a device, free any existing memory. */ + rte_free(ioat->desc_ring); + + ioat->desc_ring = rte_zmalloc(NULL, sizeof(*ioat->desc_ring) * max_desc, 0); + if (ioat->desc_ring == NULL) + return -ENOMEM; + + ioat->ring_addr = rte_mem_virt2iova(ioat->desc_ring); + + ioat->status_addr = rte_mem_virt2iova(ioat) + offsetof(struct ioat_dmadev, status); + + /* Ensure all counters are reset, if reconfiguring/restarting device. */ + ioat->next_read = 0; + ioat->next_write = 0; + ioat->last_write = 0; + ioat->offset = 0; + ioat->failure = 0; + + /* Configure descriptor ring - each one points to next. */ + for (i = 0; i < ioat->qcfg.nb_desc; i++) { + ioat->desc_ring[i].next = ioat->ring_addr + + (((i + 1) % ioat->qcfg.nb_desc) * DESC_SZ); + } + + return 0; +} + +/* Get device information of a device. */ +static int +ioat_dev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *info, uint32_t size) +{ + struct ioat_dmadev *ioat = dev->dev_private; + if (size < sizeof(*info)) + return -EINVAL; + info->dev_capa = RTE_DMA_CAPA_MEM_TO_MEM | + RTE_DMA_CAPA_OPS_COPY | + RTE_DMA_CAPA_OPS_FILL; + if (ioat->version >= IOAT_VER_3_4) + info->dev_capa |= RTE_DMA_CAPA_HANDLES_ERRORS; + info->max_vchans = 1; + info->min_desc = 32; + info->max_desc = 4096; + return 0; +} + +/* Close a configured device. */ +static int +ioat_dev_close(struct rte_dma_dev *dev) +{ + struct ioat_dmadev *ioat; + + if (!dev) { + IOAT_PMD_ERR("Invalid device"); + return -EINVAL; + } + + ioat = dev->dev_private; + if (!ioat) { + IOAT_PMD_ERR("Error getting dev_private"); + return -EINVAL; + } + + rte_free(ioat->desc_ring); + + return 0; +} + /* Dump DMA device info. */ static int ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) @@ -79,7 +182,11 @@ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { static const struct rte_dma_dev_ops ioat_dmadev_ops = { + .dev_close = ioat_dev_close, + .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, + .dev_info_get = ioat_dev_info_get, + .vchan_setup = ioat_vchan_setup, }; struct rte_dma_dev *dmadev = NULL; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v5 05/12] dma/ioat: add start and stop functions 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (3 preceding siblings ...) 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 04/12] dma/ioat: add configuration functions Conor Walsh @ 2021-09-24 14:33 ` Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 06/12] dma/ioat: add data path job submission functions Conor Walsh ` (6 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-24 14:33 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add start, stop and recover functions for IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 3 ++ drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index b1f847d273..d93d28023f 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -86,3 +86,6 @@ IOAT configuration requirements: * Only one ``vchan`` is supported per device. * Silent mode is not supported. * The transfer direction must be set to ``RTE_DMA_DIR_MEM_TO_MEM`` to copy from memory to memory. + +Once configured, the device can then be made ready for use by calling the +``rte_dma_start()`` API. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 92c4e2b04f..96bf55135f 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -78,6 +78,96 @@ ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, return 0; } +/* Recover IOAT device. */ +static inline int +__ioat_recover(struct ioat_dmadev *ioat) +{ + uint32_t chanerr, retry = 0; + uint16_t mask = ioat->qcfg.nb_desc - 1; + + /* Clear any channel errors. Reading and writing to chanerr does this. */ + chanerr = ioat->regs->chanerr; + ioat->regs->chanerr = chanerr; + + /* Reset Channel. */ + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + + /* Write new chain address to trigger state change. */ + ioat->regs->chainaddr = ioat->desc_ring[(ioat->next_read - 1) & mask].next; + /* Ensure channel control and status addr are correct. */ + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + ioat->regs->chancmp = ioat->status_addr; + + /* Allow HW time to move to the ARMED state. */ + do { + rte_pause(); + retry++; + } while (ioat->regs->chansts != IOAT_CHANSTS_ARMED && retry < 200); + + /* Exit as failure if device is still HALTED. */ + if (ioat->regs->chansts != IOAT_CHANSTS_ARMED) + return -1; + + /* Store next write as offset as recover will move HW and SW ring out of sync. */ + ioat->offset = ioat->next_read; + + /* Prime status register with previous address. */ + ioat->status = ioat->desc_ring[(ioat->next_read - 2) & mask].next; + + return 0; +} + +/* Start a configured device. */ +static int +ioat_dev_start(struct rte_dma_dev *dev) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + if (ioat->qcfg.nb_desc == 0 || ioat->desc_ring == NULL) + return -EBUSY; + + /* Inform hardware of where the descriptor ring is. */ + ioat->regs->chainaddr = ioat->ring_addr; + /* Inform hardware of where to write the status/completions. */ + ioat->regs->chancmp = ioat->status_addr; + + /* Prime the status register to be set to the last element. */ + ioat->status = ioat->ring_addr + ((ioat->qcfg.nb_desc - 1) * DESC_SZ); + + printf("IOAT.status: %s [0x%"PRIx64"]\n", + chansts_readable[ioat->status & IOAT_CHANSTS_STATUS], + ioat->status); + + if ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_HALTED) { + IOAT_PMD_WARN("Device HALTED on start, attempting to recover\n"); + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device couldn't be recovered"); + return -1; + } + } + + return 0; +} + +/* Stop a configured device. */ +static int +ioat_dev_stop(struct rte_dma_dev *dev) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint32_t retry = 0; + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + + do { + rte_pause(); + retry++; + } while ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) != IOAT_CHANSTS_SUSPENDED + && retry < 200); + + return ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_SUSPENDED) ? 0 : -1; +} + /* Get device information of a device. */ static int ioat_dev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *info, uint32_t size) @@ -186,6 +276,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, .dev_info_get = ioat_dev_info_get, + .dev_start = ioat_dev_start, + .dev_stop = ioat_dev_stop, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v5 06/12] dma/ioat: add data path job submission functions 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (4 preceding siblings ...) 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 05/12] dma/ioat: add start and stop functions Conor Walsh @ 2021-09-24 14:33 ` Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 07/12] dma/ioat: add data path completion functions Conor Walsh ` (5 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-24 14:33 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add data path functions for enqueuing and submitting operations to IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Reviewed-by: Chengwen Feng <fengchengwen@huawei.com> --- doc/guides/dmadevs/ioat.rst | 9 ++++ drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index d93d28023f..ec8ce5a8e5 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -89,3 +89,12 @@ IOAT configuration requirements: Once configured, the device can then be made ready for use by calling the ``rte_dma_start()`` API. + +Performing Data Copies +~~~~~~~~~~~~~~~~~~~~~~~ + +Refer to the :ref:`Enqueue / Dequeue APIs <dmadev_enqueue_dequeue>` section of the dmadev library +documentation for details on operation enqueue and submission API usage. + +It is expected that, for efficiency reasons, a burst of operations will be enqueued to the +device via multiple enqueue calls between calls to the ``rte_dma_submit()`` function. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 96bf55135f..0e92c80fb0 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -5,6 +5,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> #include <rte_malloc.h> +#include <rte_prefetch.h> #include "ioat_internal.h" @@ -17,6 +18,12 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* IOAT operations. */ +enum rte_ioat_ops { + ioat_op_copy = 0, /* Standard DMA Operation */ + ioat_op_fill /* Block Fill */ +}; + /* Configure a device. */ static int ioat_dev_configure(struct rte_dma_dev *dev __rte_unused, const struct rte_dma_conf *dev_conf, @@ -208,6 +215,87 @@ ioat_dev_close(struct rte_dma_dev *dev) return 0; } +/* Trigger hardware to begin performing enqueued operations. */ +static inline void +__submit(struct ioat_dmadev *ioat) +{ + *ioat->doorbell = ioat->next_write - ioat->offset; + + ioat->last_write = ioat->next_write; +} + +/* External submit function wrapper. */ +static int +ioat_submit(struct rte_dma_dev *dev, uint16_t qid __rte_unused) +{ + struct ioat_dmadev *ioat = (struct ioat_dmadev *)dev->dev_private; + + __submit(ioat); + + return 0; +} + +/* Write descriptor for enqueue. */ +static inline int +__write_desc(struct rte_dma_dev *dev, uint32_t op, uint64_t src, phys_addr_t dst, + unsigned int length, uint64_t flags) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint16_t ret; + const unsigned short mask = ioat->qcfg.nb_desc - 1; + const unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + const unsigned short space = mask + read - write; + struct ioat_dma_hw_desc *desc; + + if (space == 0) + return -ENOSPC; + + ioat->next_write = write + 1; + write &= mask; + + desc = &ioat->desc_ring[write]; + desc->size = length; + desc->u.control_raw = (uint32_t)((op << IOAT_CMD_OP_SHIFT) | + (1 << IOAT_COMP_UPDATE_SHIFT)); + + /* In IOAT the fence ensures that all operations including the current one + * are completed before moving on, DMAdev assumes that the fence ensures + * all operations before the current one are completed before starting + * the current one, so in IOAT we set the fence for the previous descriptor. + */ + if (flags & RTE_DMA_OP_FLAG_FENCE) + ioat->desc_ring[(write - 1) & mask].u.control.fence = 1; + + desc->src_addr = src; + desc->dest_addr = dst; + + rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]); + + ret = (uint16_t)(ioat->next_write - 1); + + if (flags & RTE_DMA_OP_FLAG_SUBMIT) + __submit(ioat); + + return ret; +} + +/* Enqueue a fill operation onto the ioat device. */ +static int +ioat_enqueue_fill(struct rte_dma_dev *dev, uint16_t qid __rte_unused, uint64_t pattern, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev, ioat_op_fill, pattern, dst, length, flags); +} + +/* Enqueue a copy operation onto the ioat device. */ +static int +ioat_enqueue_copy(struct rte_dma_dev *dev, uint16_t qid __rte_unused, rte_iova_t src, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev, ioat_op_copy, src, dst, length, flags); +} + /* Dump DMA device info. */ static int ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) @@ -303,6 +391,10 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->copy = ioat_enqueue_copy; + dmadev->fill = ioat_enqueue_fill; + dmadev->submit = ioat_submit; + ioat = dmadev->data->dev_private; ioat->regs = dev->mem_resource[0].addr; ioat->doorbell = &ioat->regs->dmacount; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v5 07/12] dma/ioat: add data path completion functions 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (5 preceding siblings ...) 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 06/12] dma/ioat: add data path job submission functions Conor Walsh @ 2021-09-24 14:33 ` Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 08/12] dma/ioat: add statistics Conor Walsh ` (4 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-24 14:33 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add the data path functions for gathering completed operations from IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- doc/guides/dmadevs/ioat.rst | 33 +++++++- drivers/dma/ioat/ioat_dmadev.c | 141 +++++++++++++++++++++++++++++++++ 2 files changed, 173 insertions(+), 1 deletion(-) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index ec8ce5a8e5..fc1b3131c7 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -94,7 +94,38 @@ Performing Data Copies ~~~~~~~~~~~~~~~~~~~~~~~ Refer to the :ref:`Enqueue / Dequeue APIs <dmadev_enqueue_dequeue>` section of the dmadev library -documentation for details on operation enqueue and submission API usage. +documentation for details on operation enqueue, submission and completion API usage. It is expected that, for efficiency reasons, a burst of operations will be enqueued to the device via multiple enqueue calls between calls to the ``rte_dma_submit()`` function. + +When gathering completions, ``rte_dma_completed()`` should be used, up until the point an error +occurs with an operation. If an error was encountered, ``rte_dma_completed_status()`` must be used +to reset the device and continue processing operations. This function will also gather the status +of each individual operation which is filled in to the ``status`` array provided as parameter +by the application. + +The status codes supported by IOAT are: + +* ``RTE_DMA_STATUS_SUCCESSFUL``: The operation was successful. +* ``RTE_DMA_STATUS_INVALID_SRC_ADDR``: The operation failed due to an invalid source address. +* ``RTE_DMA_STATUS_INVALID_DST_ADDR``: The operation failed due to an invalid destination address. +* ``RTE_DMA_STATUS_INVALID_LENGTH``: The operation failed due to an invalid descriptor length. +* ``RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR``: The device could not read the descriptor. +* ``RTE_DMA_STATUS_ERROR_UNKNOWN``: The operation failed due to an unspecified error. + +The following code shows how to retrieve the number of successfully completed +copies within a burst and then uses ``rte_dma_completed_status()`` to check +which operation failed and reset the device to continue processing operations: + +.. code-block:: C + + enum rte_dma_status_code status[COMP_BURST_SZ]; + uint16_t count, idx, status_count; + bool error = 0; + + count = rte_dma_completed(dev_id, vchan, COMP_BURST_SZ, &idx, &error); + + if (error){ + status_count = rte_dma_completed_status(dev_id, vchan, COMP_BURST_SZ, &idx, status); + } diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 0e92c80fb0..b2b7ebb3db 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -6,6 +6,7 @@ #include <rte_dmadev_pmd.h> #include <rte_malloc.h> #include <rte_prefetch.h> +#include <rte_errno.h> #include "ioat_internal.h" @@ -355,6 +356,144 @@ ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) return 0; } +/* Returns the index of the last completed operation. */ +static inline uint16_t +__get_last_completed(const struct ioat_dmadev *ioat, int *state) +{ + /* Status register contains the address of the completed operation */ + uint64_t status = ioat->status; + + /* lower 3 bits indicate "transfer status" : active, idle, halted. + * We can ignore bit 0. + */ + *state = status & IOAT_CHANSTS_STATUS; + + /* If we are just after recovering from an error the address returned by + * status will be 0, in this case we return the offset - 1 as the last + * completed. If not return the status value minus the chainaddr which + * gives us an offset into the ring. Right shifting by 6 (divide by 64) + * gives the index of the completion from the HW point of view and adding + * the offset translates the ring index from HW to SW point of view. + */ + if ((status & ~IOAT_CHANSTS_STATUS) == 0) + return ioat->offset - 1; + + return (status - ioat->ring_addr) >> 6; +} + +/* Translates IOAT ChanERRs to DMA error codes. */ +static inline enum rte_dma_status_code +__translate_status_ioat_to_dma(uint32_t chanerr) +{ + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_SRC_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_DST_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + return RTE_DMA_STATUS_INVALID_LENGTH; + else if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + return RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR; + else + return RTE_DMA_STATUS_ERROR_UNKNOWN; +} + +/* Returns details of operations that have been completed. */ +static uint16_t +ioat_completed(struct rte_dma_dev *dev, uint16_t qid __rte_unused, const uint16_t max_ops, + uint16_t *last_idx, bool *has_error) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short last_completed, count; + int state, fails = 0; + + /* Do not do any work if there is an uncleared error. */ + if (ioat->failure != 0) { + *has_error = true; + *last_idx = ioat->next_read - 2; + return 0; + } + + last_completed = __get_last_completed(ioat, &state); + count = (last_completed + 1 - read) & mask; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || state != IOAT_CHANSTS_HALTED) { + ioat->next_read = read + count; + *last_idx = ioat->next_read - 1; + } else { + *has_error = true; + rte_errno = EIO; + ioat->failure = ioat->regs->chanerr; + ioat->next_read = read + count + 1; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + ioat_dev_dump(dev, stdout); + return 0; + } + __submit(ioat); + fails++; + *last_idx = ioat->next_read - 2; + } + + return count; +} + +/* Returns detailed status information about operations that have been completed. */ +static uint16_t +ioat_completed_status(struct rte_dma_dev *dev, uint16_t qid __rte_unused, + uint16_t max_ops, uint16_t *last_idx, enum rte_dma_status_code *status) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short count, last_completed; + uint64_t fails = 0; + int state, i; + + last_completed = __get_last_completed(ioat, &state); + count = (last_completed + 1 - read) & mask; + + for (i = 0; i < RTE_MIN(count + 1, max_ops); i++) + status[i] = RTE_DMA_STATUS_SUCCESSFUL; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || state != IOAT_CHANSTS_HALTED) + ioat->next_read = read + count; + else { + rte_errno = EIO; + status[count] = __translate_status_ioat_to_dma(ioat->regs->chanerr); + count++; + ioat->next_read = read + count; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + ioat_dev_dump(dev, stdout); + return 0; + } + __submit(ioat); + fails++; + } + + if (ioat->failure > 0) { + status[0] = __translate_status_ioat_to_dma(ioat->failure); + count = RTE_MIN(count + 1, max_ops); + ioat->failure = 0; + } + + *last_idx = ioat->next_read - 1; + + return count; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -391,6 +530,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->completed = ioat_completed; + dmadev->completed_status = ioat_completed_status; dmadev->copy = ioat_enqueue_copy; dmadev->fill = ioat_enqueue_fill; dmadev->submit = ioat_submit; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v5 08/12] dma/ioat: add statistics 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (6 preceding siblings ...) 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 07/12] dma/ioat: add data path completion functions Conor Walsh @ 2021-09-24 14:33 ` Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 09/12] dma/ioat: add support for vchan status function Conor Walsh ` (3 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-24 14:33 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add statistic tracking for operations in IOAT. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 43 ++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index b2b7ebb3db..20ae364318 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -77,6 +77,9 @@ ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, ioat->offset = 0; ioat->failure = 0; + /* Reset Stats. */ + ioat->stats = (struct rte_dma_stats){0}; + /* Configure descriptor ring - each one points to next. */ for (i = 0; i < ioat->qcfg.nb_desc; i++) { ioat->desc_ring[i].next = ioat->ring_addr + @@ -222,6 +225,8 @@ __submit(struct ioat_dmadev *ioat) { *ioat->doorbell = ioat->next_write - ioat->offset; + ioat->stats.submitted += (uint16_t)(ioat->next_write - ioat->last_write); + ioat->last_write = ioat->next_write; } @@ -352,6 +357,10 @@ ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) fprintf(f, " Dest: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); fprintf(f, " Next: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].next); fprintf(f, " }\n"); + fprintf(f, " Key Stats { submitted: %"PRIu64", comp: %"PRIu64", failed: %"PRIu64" }\n", + ioat->stats.submitted, + ioat->stats.completed, + ioat->stats.errors); return 0; } @@ -441,6 +450,9 @@ ioat_completed(struct rte_dma_dev *dev, uint16_t qid __rte_unused, const uint16_ *last_idx = ioat->next_read - 2; } + ioat->stats.completed += count; + ioat->stats.errors += fails; + return count; } @@ -491,9 +503,38 @@ ioat_completed_status(struct rte_dma_dev *dev, uint16_t qid __rte_unused, *last_idx = ioat->next_read - 1; + ioat->stats.completed += count; + ioat->stats.errors += fails; + return count; } +/* Retrieve the generic stats of a DMA device. */ +static int +ioat_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + struct rte_dma_stats *rte_stats, uint32_t size) +{ + struct rte_dma_stats *stats = (&((struct ioat_dmadev *)dev->dev_private)->stats); + + if (size < sizeof(rte_stats)) + return -EINVAL; + if (rte_stats == NULL) + return -EINVAL; + + *rte_stats = *stats; + return 0; +} + +/* Reset the generic stat counters for the DMA device. */ +static int +ioat_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + ioat->stats = (struct rte_dma_stats){0}; + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -505,6 +546,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_info_get = ioat_dev_info_get, .dev_start = ioat_dev_start, .dev_stop = ioat_dev_stop, + .stats_get = ioat_stats_get, + .stats_reset = ioat_stats_reset, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v5 09/12] dma/ioat: add support for vchan status function 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (7 preceding siblings ...) 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 08/12] dma/ioat: add statistics Conor Walsh @ 2021-09-24 14:33 ` Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 10/12] dma/ioat: add burst capacity function Conor Walsh ` (2 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-24 14:33 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add support for the rte_dmadev_vchan_status API call. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 20ae364318..fe01a3b1db 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -535,6 +535,26 @@ ioat_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused) return 0; } +/* Check if the IOAT device is idle. */ +static int +ioat_vchan_status(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + enum rte_dma_vchan_status *status) +{ + int state = 0; + const struct ioat_dmadev *ioat = dev->dev_private; + const uint16_t mask = ioat->qcfg.nb_desc - 1; + const uint16_t last = __get_last_completed(ioat, &state); + + if (state == IOAT_CHANSTS_HALTED || state == IOAT_CHANSTS_SUSPENDED) + *status = RTE_DMA_VCHAN_HALTED_ERROR; + else if (last == ((ioat->next_write - 1) & mask)) + *status = RTE_DMA_VCHAN_IDLE; + else + *status = RTE_DMA_VCHAN_ACTIVE; + + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -548,6 +568,7 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_stop = ioat_dev_stop, .stats_get = ioat_stats_get, .stats_reset = ioat_stats_reset, + .vchan_status = ioat_vchan_status, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v5 10/12] dma/ioat: add burst capacity function 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (8 preceding siblings ...) 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 09/12] dma/ioat: add support for vchan status function Conor Walsh @ 2021-09-24 14:33 ` Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 11/12] devbind: move ioat device IDs to dmadev category Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-24 14:33 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Adds the ability to find the remaining space in the IOAT ring. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index fe01a3b1db..6c783ec94f 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -509,6 +509,19 @@ ioat_completed_status(struct rte_dma_dev *dev, uint16_t qid __rte_unused, return count; } +/* Get the remaining capacity of the ring. */ +static uint16_t +ioat_burst_capacity(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused) +{ + struct ioat_dmadev *ioat = dev->data->dev_private; + unsigned short size = ioat->qcfg.nb_desc - 1; + unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + unsigned short space = size - (write - read); + + return space; +} + /* Retrieve the generic stats of a DMA device. */ static int ioat_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, @@ -594,6 +607,7 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->burst_capacity = ioat_burst_capacity; dmadev->completed = ioat_completed; dmadev->completed_status = ioat_completed_status; dmadev->copy = ioat_enqueue_copy; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v5 11/12] devbind: move ioat device IDs to dmadev category 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (9 preceding siblings ...) 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 10/12] dma/ioat: add burst capacity function Conor Walsh @ 2021-09-24 14:33 ` Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-24 14:33 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Move Intel IOAT devices from Misc to DMA devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Reviewed-by: Bruce Richardson <bruce.richardson@intel.com> --- usertools/dpdk-devbind.py | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index 15d438715f..4a72229622 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -69,14 +69,12 @@ network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] -dma_devices = [intel_idxd_spr] +dma_devices = [intel_idxd_spr, intel_ioat_bdw, intel_ioat_icx, intel_ioat_skx] eventdev_devices = [cavium_sso, cavium_tim, intel_dlb, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] regex_devices = [octeontx2_ree] -misc_devices = [cnxk_bphy, cnxk_bphy_cgx, intel_ioat_bdw, intel_ioat_skx, intel_ioat_icx, - intel_ntb_skx, intel_ntb_icx, - octeontx2_dma] +misc_devices = [cnxk_bphy, cnxk_bphy_cgx, intel_ntb_skx, intel_ntb_icx, octeontx2_dma] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v5 12/12] raw/ioat: deprecate ioat rawdev driver 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (10 preceding siblings ...) 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 11/12] devbind: move ioat device IDs to dmadev category Conor Walsh @ 2021-09-24 14:33 ` Conor Walsh 2021-09-24 16:21 ` Kevin Laatz 2021-09-24 16:56 ` Bruce Richardson 11 siblings, 2 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-24 14:33 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Deprecate the rawdev IOAT driver as both IOAT and IDXD drivers have moved to dmadev. Signed-off-by: Conor Walsh <conor.walsh@intel.com> --- MAINTAINERS | 2 +- doc/guides/rawdevs/ioat.rst | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/MAINTAINERS b/MAINTAINERS index ccabba9169..a4bcd2d024 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1322,7 +1322,7 @@ T: git://dpdk.org/next/dpdk-next-net-intel F: drivers/raw/ifpga/ F: doc/guides/rawdevs/ifpga.rst -IOAT Rawdev +IOAT Rawdev - DEPRECATED M: Bruce Richardson <bruce.richardson@intel.com> F: drivers/raw/ioat/ F: doc/guides/rawdevs/ioat.rst diff --git a/doc/guides/rawdevs/ioat.rst b/doc/guides/rawdevs/ioat.rst index a65530bd30..98d15dd032 100644 --- a/doc/guides/rawdevs/ioat.rst +++ b/doc/guides/rawdevs/ioat.rst @@ -6,6 +6,10 @@ IOAT Rawdev Driver =================== +.. warning:: + As of DPDK 21.11 the rawdev implementation of the IOAT driver has been deprecated. + Please use the dmadev library instead. + The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg| Data Streaming Accelerator `(Intel DSA) <https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator>`_ and for Intel\ |reg| -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v5 12/12] raw/ioat: deprecate ioat rawdev driver 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh @ 2021-09-24 16:21 ` Kevin Laatz 2021-09-24 16:56 ` Bruce Richardson 1 sibling, 0 replies; 132+ messages in thread From: Kevin Laatz @ 2021-09-24 16:21 UTC (permalink / raw) To: Conor Walsh, bruce.richardson, fengchengwen, jerinj; +Cc: dev On 24/09/2021 15:33, Conor Walsh wrote: > Deprecate the rawdev IOAT driver as both IOAT and IDXD drivers have > moved to dmadev. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > --- > MAINTAINERS | 2 +- > doc/guides/rawdevs/ioat.rst | 4 ++++ > 2 files changed, 5 insertions(+), 1 deletion(-) Acked-by: Kevin Laatz <kevin.laatz@intel.com> ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v5 12/12] raw/ioat: deprecate ioat rawdev driver 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh 2021-09-24 16:21 ` Kevin Laatz @ 2021-09-24 16:56 ` Bruce Richardson 2021-09-27 9:36 ` Walsh, Conor 1 sibling, 1 reply; 132+ messages in thread From: Bruce Richardson @ 2021-09-24 16:56 UTC (permalink / raw) To: Conor Walsh; +Cc: fengchengwen, jerinj, kevin.laatz, dev On Fri, Sep 24, 2021 at 02:33:35PM +0000, Conor Walsh wrote: > Deprecate the rawdev IOAT driver as both IOAT and IDXD drivers have > moved to dmadev. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > --- This probably needs a release note entry for it too. With such a RN addition, feel free to add my acked-by. > MAINTAINERS | 2 +- > doc/guides/rawdevs/ioat.rst | 4 ++++ > 2 files changed, 5 insertions(+), 1 deletion(-) > > diff --git a/MAINTAINERS b/MAINTAINERS > index ccabba9169..a4bcd2d024 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -1322,7 +1322,7 @@ T: git://dpdk.org/next/dpdk-next-net-intel > F: drivers/raw/ifpga/ > F: doc/guides/rawdevs/ifpga.rst > > -IOAT Rawdev > +IOAT Rawdev - DEPRECATED > M: Bruce Richardson <bruce.richardson@intel.com> > F: drivers/raw/ioat/ > F: doc/guides/rawdevs/ioat.rst > diff --git a/doc/guides/rawdevs/ioat.rst b/doc/guides/rawdevs/ioat.rst > index a65530bd30..98d15dd032 100644 > --- a/doc/guides/rawdevs/ioat.rst > +++ b/doc/guides/rawdevs/ioat.rst > @@ -6,6 +6,10 @@ > IOAT Rawdev Driver > =================== > > +.. warning:: > + As of DPDK 21.11 the rawdev implementation of the IOAT driver has been deprecated. > + Please use the dmadev library instead. > + > The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg| > Data Streaming Accelerator `(Intel DSA) > <https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator>`_ and for Intel\ |reg| > -- > 2.25.1 > ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v5 12/12] raw/ioat: deprecate ioat rawdev driver 2021-09-24 16:56 ` Bruce Richardson @ 2021-09-27 9:36 ` Walsh, Conor 0 siblings, 0 replies; 132+ messages in thread From: Walsh, Conor @ 2021-09-27 9:36 UTC (permalink / raw) To: Richardson, Bruce; +Cc: fengchengwen, jerinj, Laatz, Kevin, dev > From: Richardson, Bruce <bruce.richardson@intel.com> > Sent: Friday 24 September 2021 17:57 > To: Walsh, Conor <conor.walsh@intel.com> > Cc: fengchengwen@huawei.com; jerinj@marvell.com; Laatz, Kevin > <kevin.laatz@intel.com>; dev@dpdk.org > Subject: Re: [PATCH v5 12/12] raw/ioat: deprecate ioat rawdev driver > > On Fri, Sep 24, 2021 at 02:33:35PM +0000, Conor Walsh wrote: > > Deprecate the rawdev IOAT driver as both IOAT and IDXD drivers have > > moved to dmadev. > > > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > > --- > > This probably needs a release note entry for it too. > With such a RN addition, feel free to add my acked-by. Thanks Bruce, I will add a deprecation notice in v6. /Conor. > > > MAINTAINERS | 2 +- > > doc/guides/rawdevs/ioat.rst | 4 ++++ > > 2 files changed, 5 insertions(+), 1 deletion(-) > > > > diff --git a/MAINTAINERS b/MAINTAINERS > > index ccabba9169..a4bcd2d024 100644 > > --- a/MAINTAINERS > > +++ b/MAINTAINERS > > @@ -1322,7 +1322,7 @@ T: git://dpdk.org/next/dpdk-next-net-intel > > F: drivers/raw/ifpga/ > > F: doc/guides/rawdevs/ifpga.rst > > > > -IOAT Rawdev > > +IOAT Rawdev - DEPRECATED > > M: Bruce Richardson <bruce.richardson@intel.com> > > F: drivers/raw/ioat/ > > F: doc/guides/rawdevs/ioat.rst > > diff --git a/doc/guides/rawdevs/ioat.rst b/doc/guides/rawdevs/ioat.rst > > index a65530bd30..98d15dd032 100644 > > --- a/doc/guides/rawdevs/ioat.rst > > +++ b/doc/guides/rawdevs/ioat.rst > > @@ -6,6 +6,10 @@ > > IOAT Rawdev Driver > > =================== > > > > +.. warning:: > > + As of DPDK 21.11 the rawdev implementation of the IOAT driver has > been deprecated. > > + Please use the dmadev library instead. > > + > > The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ > |reg| > > Data Streaming Accelerator `(Intel DSA) > > <https://01.org/blogs/2019/introducing-intel-data-streaming- > accelerator>`_ and for Intel\ |reg| > > -- > > 2.25.1 > > ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh ` (10 preceding siblings ...) 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh @ 2021-09-27 10:21 ` Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 01/12] dma/ioat: add device probe and removal functions Conor Walsh ` (11 more replies) 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh 13 siblings, 12 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-27 10:21 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh This patchset adds a dmadev driver and associated documentation to support Intel QuickData Technology devices, part of the Intel I/O Acceleration Technology (Intel I/OAT). This driver is intended to ultimately replace the current IOAT part of the IOAT rawdev driver. This patchset passes all the driver tests added in the dmadev test suite. NOTE: This patchset has several dependencies: - v23 of the dmadev set [1] - v6 of the dmadev test suite [2] - v6 of the IDXD driver [3] [1] http://patches.dpdk.org/project/dpdk/list/?series=19140 [2] http://patches.dpdk.org/project/dpdk/list/?series=19138 [3] http://patches.dpdk.org/project/dpdk/list/?series=19144 --- v6: - Added rawdev IOAT deprecation notice to deprecation.rst v5: - Updated to v23 of the dmadev lib. - Removed experimental tag for driver from MAINTAINERS. - Seperated IOAT and IDXD announcements in release notes. - Added missing check for rte_dma_get_dev_id in destroy. - Fixed memleak in destroy caused by NULL pointer. - Rewrote part of the docs to reduce duplication with DMA and IDXD. - Added patch to deprecate the rawdev IOAT driver. - Reworked destroy and close functions. - Added RTE_DMA_CAPA_HANDLES_ERRORS flag for IOAT versions >=3.4. - Other minor changes to IOAT driver. v4: - Changes needed to update from dmadev v21 to v22. - Fixed 32-bit build. - Made stats reset logic easier to understand. v3: - Added burst capacity function. - Stop function now waits for suspend rather than just using a sleep. - Changed from vchan idle to vchan status function. - Other minor changes to update from dmadev v19 to v21. v2: - Rebased on the above patchsets. - Added support for the vchan idle function. - Stop function now suspends IOAT channel to allow for reconfig. - dmadev_autotest can now be run multiple times using the IOAT driver without errors. - Added devbind updates for DMA devices - Removed some logically dead code found by coverity in the Conor Walsh (12): dma/ioat: add device probe and removal functions dma/ioat: create dmadev instances on PCI probe dma/ioat: add datapath structures dma/ioat: add configuration functions dma/ioat: add start and stop functions dma/ioat: add data path job submission functions dma/ioat: add data path completion functions dma/ioat: add statistics dma/ioat: add support for vchan status function dma/ioat: add burst capacity function devbind: move ioat device IDs to dmadev category raw/ioat: deprecate ioat rawdev driver MAINTAINERS | 8 +- doc/guides/dmadevs/index.rst | 2 + doc/guides/dmadevs/ioat.rst | 131 +++++ doc/guides/rawdevs/ioat.rst | 4 + doc/guides/rel_notes/deprecation.rst | 7 + doc/guides/rel_notes/release_21_11.rst | 6 + drivers/dma/ioat/ioat_dmadev.c | 738 +++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 295 ++++++++++ drivers/dma/ioat/ioat_internal.h | 47 ++ drivers/dma/ioat/meson.build | 7 + drivers/dma/ioat/version.map | 3 + drivers/dma/meson.build | 1 + usertools/dpdk-devbind.py | 6 +- 13 files changed, 1250 insertions(+), 5 deletions(-) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v6 01/12] dma/ioat: add device probe and removal functions 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh @ 2021-09-27 10:21 ` Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 02/12] dma/ioat: create dmadev instances on PCI probe Conor Walsh ` (10 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-27 10:21 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add the basic device probe/remove skeleton code and initial documentation for new IOAT DMA driver. Maintainers update is also included in this patch. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Reviewed-by: Chengwen Feng <fengchengwen@huawei.com> --- MAINTAINERS | 6 +++ doc/guides/dmadevs/index.rst | 2 + doc/guides/dmadevs/ioat.rst | 69 ++++++++++++++++++++++++++ doc/guides/rel_notes/release_21_11.rst | 6 +++ drivers/dma/ioat/ioat_dmadev.c | 69 ++++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 35 +++++++++++++ drivers/dma/ioat/ioat_internal.h | 20 ++++++++ drivers/dma/ioat/meson.build | 7 +++ drivers/dma/ioat/version.map | 3 ++ drivers/dma/meson.build | 1 + 10 files changed, 218 insertions(+) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map diff --git a/MAINTAINERS b/MAINTAINERS index 497219e948..ccabba9169 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1209,6 +1209,12 @@ M: Kevin Laatz <kevin.laatz@intel.com> F: drivers/dma/idxd/ F: doc/guides/dmadevs/idxd.rst +Intel IOAT +M: Bruce Richardson <bruce.richardson@intel.com> +M: Conor Walsh <conor.walsh@intel.com> +F: drivers/dma/ioat/ +F: doc/guides/dmadevs/ioat.rst + RegEx Drivers ------------- diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst index 5d4abf880e..c59f4b5c92 100644 --- a/doc/guides/dmadevs/index.rst +++ b/doc/guides/dmadevs/index.rst @@ -12,3 +12,5 @@ an application through DMA API. :numbered: idxd + ioat + diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst new file mode 100644 index 0000000000..9ae1d8a2ad --- /dev/null +++ b/doc/guides/dmadevs/ioat.rst @@ -0,0 +1,69 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2021 Intel Corporation. + +.. include:: <isonum.txt> + +IOAT DMA Device Driver +======================= + +The ``ioat`` dmadev driver provides a poll-mode driver (PMD) for Intel\ +|reg| QuickData Technology which is part of part of Intel\ |reg| I/O +Acceleration Technology (`Intel I/OAT +<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_). +This PMD, when used on supported hardware, allows data copies, for example, +cloning packet data, to be accelerated by IOAT hardware rather than having to +be done by software, freeing up CPU cycles for other tasks. + +Hardware Requirements +---------------------- + +The ``dpdk-devbind.py`` script, included with DPDK, can be used to show the +presence of supported hardware. Running ``dpdk-devbind.py --status-dev dma`` +will show all the DMA devices on the system, IOAT devices are included in this +list. For Intel\ |reg| IOAT devices, the hardware will often be listed as +"Crystal Beach DMA", or "CBDMA" or on some newer systems '0b00' due to the +absence of pci-id database entries for them at this point. + +.. note:: + Error handling is not supported by this driver on hardware prior to + Intel Ice Lake. Unsupported systems include Broadwell, Skylake and + Cascade Lake. + +Compilation +------------ + +For builds using ``meson`` and ``ninja``, the driver will be built when the +target platform is x86-based. No additional compilation steps are necessary. + +Device Setup +------------- + +Intel\ |reg| IOAT devices will need to be bound to a suitable DPDK-supported +user-space IO driver such as ``vfio-pci`` in order to be used by DPDK. + +The ``dpdk-devbind.py`` script can be used to view the state of the devices using:: + + $ dpdk-devbind.py --status-dev dma + +The ``dpdk-devbind.py`` script can also be used to bind devices to a suitable driver. +For example:: + + $ dpdk-devbind.py -b vfio-pci 00:01.0 00:01.1 + +Device Probing and Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For devices bound to a suitable DPDK-supported driver (``vfio-pci``), the HW +devices will be found as part of the device scan done at application +initialization time without the need to pass parameters to the application. + +If the application does not require all the devices available an allowlist can +be used in the same way that other DPDK devices use them. + +For example:: + + $ dpdk-test -a <b:d:f> + +Once probed successfully, the device will appear as a ``dmadev``, that is a +"DMA device type" inside DPDK, and can be accessed using APIs from the +``rte_dmadev`` library. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index c980e729f8..e34957069f 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -103,6 +103,12 @@ New Features The IDXD dmadev driver provide device drivers for the Intel DSA devices. This device driver can be used through the generic dmadev API. +* **Added IOAT dmadev driver implementation.** + + The Intel I/O Acceleration Technology (IOAT) dmadev driver provides a device + driver for Intel IOAT devices such as Crystal Beach DMA (CBDMA) on Ice Lake, + Skylake and Broadwell. This device driver can be used through the generic dmadev API. + Removed Items ------------- diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c new file mode 100644 index 0000000000..f3491d45b1 --- /dev/null +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#include <rte_bus_pci.h> +#include <rte_dmadev_pmd.h> + +#include "ioat_internal.h" + +static struct rte_pci_driver ioat_pmd_drv; + +RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); + +#define IOAT_PMD_NAME dmadev_ioat +#define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) + +/* Probe DMA device. */ +static int +ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); + + dev->device.driver = &drv->driver; + return 0; +} + +/* Remove DMA device. */ +static int +ioat_dmadev_remove(struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + + IOAT_PMD_INFO("Closing %s on NUMA node %d", + name, dev->device.numa_node); + + return 0; +} + +static const struct rte_pci_id pci_id_ioat_map[] = { + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_SKX) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX0) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX1) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX2) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX3) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX4) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX5) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX6) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX7) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXE) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXF) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_ICX) }, + { .vendor_id = 0, /* sentinel */ }, +}; + +static struct rte_pci_driver ioat_pmd_drv = { + .id_table = pci_id_ioat_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC, + .probe = ioat_dmadev_probe, + .remove = ioat_dmadev_remove, +}; + +RTE_PMD_REGISTER_PCI(IOAT_PMD_NAME, ioat_pmd_drv); +RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_NAME, pci_id_ioat_map); +RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_NAME, "* igb_uio | uio_pci_generic | vfio-pci"); diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h new file mode 100644 index 0000000000..eeabba41ef --- /dev/null +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#ifndef IOAT_HW_DEFS_H +#define IOAT_HW_DEFS_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> + +#define IOAT_VER_3_0 0x30 +#define IOAT_VER_3_3 0x33 + +#define IOAT_VENDOR_ID 0x8086 +#define IOAT_DEVICE_ID_SKX 0x2021 +#define IOAT_DEVICE_ID_BDX0 0x6f20 +#define IOAT_DEVICE_ID_BDX1 0x6f21 +#define IOAT_DEVICE_ID_BDX2 0x6f22 +#define IOAT_DEVICE_ID_BDX3 0x6f23 +#define IOAT_DEVICE_ID_BDX4 0x6f24 +#define IOAT_DEVICE_ID_BDX5 0x6f25 +#define IOAT_DEVICE_ID_BDX6 0x6f26 +#define IOAT_DEVICE_ID_BDX7 0x6f27 +#define IOAT_DEVICE_ID_BDXE 0x6f2E +#define IOAT_DEVICE_ID_BDXF 0x6f2F +#define IOAT_DEVICE_ID_ICX 0x0b00 + +#ifdef __cplusplus +} +#endif + +#endif /* IOAT_HW_DEFS_H */ diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h new file mode 100644 index 0000000000..f1ec12a919 --- /dev/null +++ b/drivers/dma/ioat/ioat_internal.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021 Intel Corporation + */ + +#ifndef _IOAT_INTERNAL_H_ +#define _IOAT_INTERNAL_H_ + +#include "ioat_hw_defs.h" + +extern int ioat_pmd_logtype; + +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + ioat_pmd_logtype, "IOAT: %s(): " fmt "\n", __func__, ##args) + +#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) +#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) +#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) +#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) + +#endif /* _IOAT_INTERNAL_H_ */ diff --git a/drivers/dma/ioat/meson.build b/drivers/dma/ioat/meson.build new file mode 100644 index 0000000000..d67fac96fb --- /dev/null +++ b/drivers/dma/ioat/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021 Intel Corporation + +build = dpdk_conf.has('RTE_ARCH_X86') +reason = 'only supported on x86' +sources = files('ioat_dmadev.c') +deps += ['bus_pci', 'dmadev'] diff --git a/drivers/dma/ioat/version.map b/drivers/dma/ioat/version.map new file mode 100644 index 0000000000..c2e0723b4c --- /dev/null +++ b/drivers/dma/ioat/version.map @@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +}; diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build index 411be7a240..a69418ce9b 100644 --- a/drivers/dma/meson.build +++ b/drivers/dma/meson.build @@ -3,6 +3,7 @@ drivers = [ 'idxd', + 'ioat', 'skeleton', ] std_deps = ['dmadev'] -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v6 02/12] dma/ioat: create dmadev instances on PCI probe 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 01/12] dma/ioat: add device probe and removal functions Conor Walsh @ 2021-09-27 10:21 ` Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 03/12] dma/ioat: add datapath structures Conor Walsh ` (9 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-27 10:21 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh When a suitable device is found during the PCI probe, create a dmadev instance for each channel. Internal structures and HW definitions required for device creation are also included. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 102 ++++++++++++++++++++++++++++++- drivers/dma/ioat/ioat_hw_defs.h | 45 ++++++++++++++ drivers/dma/ioat/ioat_internal.h | 27 ++++++++ 3 files changed, 172 insertions(+), 2 deletions(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index f3491d45b1..df3c72363a 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -4,6 +4,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> +#include <rte_malloc.h> #include "ioat_internal.h" @@ -14,6 +15,103 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Create a DMA device. */ +static int +ioat_dmadev_create(const char *name, struct rte_pci_device *dev) +{ + static const struct rte_dma_dev_ops ioat_dmadev_ops = { }; + + struct rte_dma_dev *dmadev = NULL; + struct ioat_dmadev *ioat = NULL; + int retry = 0; + + if (!name) { + IOAT_PMD_ERR("Invalid name of the device!"); + return -EINVAL; + } + + /* Allocate device structure. */ + dmadev = rte_dma_pmd_allocate(name, dev->device.numa_node, sizeof(*ioat)); + if (dmadev == NULL) { + IOAT_PMD_ERR("Unable to allocate dma device"); + return -ENOMEM; + } + + dmadev->device = &dev->device; + + dmadev->dev_private = dmadev->data->dev_private; + + dmadev->dev_ops = &ioat_dmadev_ops; + + ioat = dmadev->data->dev_private; + ioat->regs = dev->mem_resource[0].addr; + ioat->doorbell = &ioat->regs->dmacount; + ioat->qcfg.nb_desc = 0; + ioat->desc_ring = NULL; + ioat->version = ioat->regs->cbver; + + /* Do device initialization - reset and set error behaviour. */ + if (ioat->regs->chancnt != 1) + IOAT_PMD_WARN("%s: Channel count == %d\n", __func__, + ioat->regs->chancnt); + + /* Locked by someone else. */ + if (ioat->regs->chanctrl & IOAT_CHANCTRL_CHANNEL_IN_USE) { + IOAT_PMD_WARN("%s: Channel appears locked\n", __func__); + ioat->regs->chanctrl = 0; + } + + /* clear any previous errors */ + if (ioat->regs->chanerr != 0) { + uint32_t val = ioat->regs->chanerr; + ioat->regs->chanerr = val; + } + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + rte_delay_ms(1); + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + rte_delay_ms(1); + while (ioat->regs->chancmd & IOAT_CHANCMD_RESET) { + ioat->regs->chainaddr = 0; + rte_delay_ms(1); + if (++retry >= 200) { + IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=%#"PRIx8 + ", CHANSTS=%#"PRIx64", CHANERR=%#"PRIx32"\n", + __func__, + ioat->regs->chancmd, + ioat->regs->chansts, + ioat->regs->chanerr); + rte_dma_pmd_release(name); + return -EIO; + } + } + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + + dmadev->state = RTE_DMA_DEV_READY; + + return 0; + +} + +/* Destroy a DMA device. */ +static int +ioat_dmadev_destroy(const char *name) +{ + int ret; + + if (!name) { + IOAT_PMD_ERR("Invalid device name"); + return -EINVAL; + } + + ret = rte_dma_pmd_release(name); + if (ret) + IOAT_PMD_DEBUG("Device cleanup failed"); + + return 0; +} + /* Probe DMA device. */ static int ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) @@ -24,7 +122,7 @@ ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); dev->device.driver = &drv->driver; - return 0; + return ioat_dmadev_create(name, dev); } /* Remove DMA device. */ @@ -38,7 +136,7 @@ ioat_dmadev_remove(struct rte_pci_device *dev) IOAT_PMD_INFO("Closing %s on NUMA node %d", name, dev->device.numa_node); - return 0; + return ioat_dmadev_destroy(name); } static const struct rte_pci_id pci_id_ioat_map[] = { diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index eeabba41ef..73bdf548b3 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -11,6 +11,8 @@ extern "C" { #include <stdint.h> +#define IOAT_PCI_CHANERR_INT_OFFSET 0x180 + #define IOAT_VER_3_0 0x30 #define IOAT_VER_3_3 0x33 @@ -28,6 +30,49 @@ extern "C" { #define IOAT_DEVICE_ID_BDXF 0x6f2F #define IOAT_DEVICE_ID_ICX 0x0b00 +#define IOAT_COMP_UPDATE_SHIFT 3 +#define IOAT_CMD_OP_SHIFT 24 + +/* DMA Channel Registers */ +#define IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK 0xF000 +#define IOAT_CHANCTRL_COMPL_DCA_EN 0x0200 +#define IOAT_CHANCTRL_CHANNEL_IN_USE 0x0100 +#define IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL 0x0020 +#define IOAT_CHANCTRL_ERR_INT_EN 0x0010 +#define IOAT_CHANCTRL_ANY_ERR_ABORT_EN 0x0008 +#define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 +#define IOAT_CHANCTRL_INT_REARM 0x0001 + +struct ioat_registers { + uint8_t chancnt; + uint8_t xfercap; + uint8_t genctrl; + uint8_t intrctrl; + uint32_t attnstatus; + uint8_t cbver; /* 0x08 */ + uint8_t reserved4[0x3]; /* 0x09 */ + uint16_t intrdelay; /* 0x0C */ + uint16_t cs_status; /* 0x0E */ + uint32_t dmacapability; /* 0x10 */ + uint8_t reserved5[0x6C]; /* 0x14 */ + uint16_t chanctrl; /* 0x80 */ + uint8_t reserved6[0x2]; /* 0x82 */ + uint8_t chancmd; /* 0x84 */ + uint8_t reserved3[1]; /* 0x85 */ + uint16_t dmacount; /* 0x86 */ + uint64_t chansts; /* 0x88 */ + uint64_t chainaddr; /* 0x90 */ + uint64_t chancmp; /* 0x98 */ + uint8_t reserved2[0x8]; /* 0xA0 */ + uint32_t chanerr; /* 0xA8 */ + uint32_t chanerrmask; /* 0xAC */ +} __rte_packed; + +#define IOAT_CHANCMD_RESET 0x20 +#define IOAT_CHANCMD_SUSPEND 0x04 + +#define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ + #ifdef __cplusplus } #endif diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h index f1ec12a919..83ef5973f5 100644 --- a/drivers/dma/ioat/ioat_internal.h +++ b/drivers/dma/ioat/ioat_internal.h @@ -7,6 +7,33 @@ #include "ioat_hw_defs.h" +struct ioat_dmadev { + struct rte_dma_dev_data *dmadev; + struct rte_dma_vchan_conf qcfg; + struct rte_dma_stats stats; + + volatile uint16_t *doorbell __rte_cache_aligned; + phys_addr_t status_addr; + phys_addr_t ring_addr; + + struct ioat_dma_hw_desc *desc_ring; + + unsigned short next_read; + unsigned short next_write; + unsigned short last_write; /* Used to compute submitted count. */ + unsigned short offset; /* Used after a device recovery when counts -> 0. */ + unsigned int failure; /* Used to store chanerr for error handling. */ + + /* To report completions, the device will write status back here. */ + volatile uint64_t status __rte_cache_aligned; + + /* Pointer to the register bar. */ + volatile struct ioat_registers *regs; + + /* Store the IOAT version. */ + uint8_t version; +}; + extern int ioat_pmd_logtype; #define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v6 03/12] dma/ioat: add datapath structures 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 01/12] dma/ioat: add device probe and removal functions Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 02/12] dma/ioat: create dmadev instances on PCI probe Conor Walsh @ 2021-09-27 10:21 ` Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 04/12] dma/ioat: add configuration functions Conor Walsh ` (8 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-27 10:21 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add data structures required for the data path of IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 63 +++++++++- drivers/dma/ioat/ioat_hw_defs.h | 215 ++++++++++++++++++++++++++++++++ 2 files changed, 277 insertions(+), 1 deletion(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index df3c72363a..b132283ba5 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -15,11 +15,72 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Dump DMA device info. */ +static int +ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint64_t chansts_masked = ioat->regs->chansts & IOAT_CHANSTS_STATUS; + uint32_t chanerr = ioat->regs->chanerr; + uint64_t mask = (ioat->qcfg.nb_desc - 1); + char ver = ioat->version; + fprintf(f, "========= IOAT =========\n"); + fprintf(f, " IOAT version: %d.%d\n", ver >> 4, ver & 0xF); + fprintf(f, " Channel status: %s [0x%"PRIx64"]\n", + chansts_readable[chansts_masked], chansts_masked); + fprintf(f, " ChainADDR: 0x%"PRIu64"\n", ioat->regs->chainaddr); + if (chanerr == 0) { + fprintf(f, " No Channel Errors\n"); + } else { + fprintf(f, " ChanERR: 0x%"PRIu32"\n", chanerr); + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + fprintf(f, " Invalid Source Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + fprintf(f, " Invalid Destination Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + fprintf(f, " Invalid Descriptor Length\n"); + if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + fprintf(f, " Descriptor Read Error\n"); + if ((chanerr & ~(IOAT_CHANERR_INVALID_SRC_ADDR_MASK | + IOAT_CHANERR_INVALID_DST_ADDR_MASK | + IOAT_CHANERR_INVALID_LENGTH_MASK | + IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK)) != 0) + fprintf(f, " Unknown Error(s)\n"); + } + fprintf(f, "== Private Data ==\n"); + fprintf(f, " Config: { ring_size: %u }\n", ioat->qcfg.nb_desc); + fprintf(f, " Status: 0x%"PRIx64"\n", ioat->status); + fprintf(f, " Status IOVA: 0x%"PRIx64"\n", ioat->status_addr); + fprintf(f, " Status ADDR: %p\n", &ioat->status); + fprintf(f, " Ring IOVA: 0x%"PRIx64"\n", ioat->ring_addr); + fprintf(f, " Ring ADDR: 0x%"PRIx64"\n", ioat->desc_ring[0].next-64); + fprintf(f, " Next write: %"PRIu16"\n", ioat->next_write); + fprintf(f, " Next read: %"PRIu16"\n", ioat->next_read); + struct ioat_dma_hw_desc *desc_ring = &ioat->desc_ring[(ioat->next_write - 1) & mask]; + fprintf(f, " Last Descriptor Written {\n"); + fprintf(f, " Size: %"PRIu32"\n", desc_ring->size); + fprintf(f, " Control: 0x%"PRIx32"\n", desc_ring->u.control_raw); + fprintf(f, " Src: 0x%"PRIx64"\n", desc_ring->src_addr); + fprintf(f, " Dest: 0x%"PRIx64"\n", desc_ring->dest_addr); + fprintf(f, " Next: 0x%"PRIx64"\n", desc_ring->next); + fprintf(f, " }\n"); + fprintf(f, " Next Descriptor {\n"); + fprintf(f, " Size: %"PRIu32"\n", ioat->desc_ring[ioat->next_read & mask].size); + fprintf(f, " Src: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].src_addr); + fprintf(f, " Dest: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); + fprintf(f, " Next: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].next); + fprintf(f, " }\n"); + + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { - static const struct rte_dma_dev_ops ioat_dmadev_ops = { }; + static const struct rte_dma_dev_ops ioat_dmadev_ops = { + .dev_dump = ioat_dev_dump, + }; struct rte_dma_dev *dmadev = NULL; struct ioat_dmadev *ioat = NULL; diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index 73bdf548b3..dc3493a78f 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -15,6 +15,7 @@ extern "C" { #define IOAT_VER_3_0 0x30 #define IOAT_VER_3_3 0x33 +#define IOAT_VER_3_4 0x34 #define IOAT_VENDOR_ID 0x8086 #define IOAT_DEVICE_ID_SKX 0x2021 @@ -43,6 +44,14 @@ extern "C" { #define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 #define IOAT_CHANCTRL_INT_REARM 0x0001 +/* DMA Channel Capabilities */ +#define IOAT_DMACAP_PB (1 << 0) +#define IOAT_DMACAP_DCA (1 << 4) +#define IOAT_DMACAP_BFILL (1 << 6) +#define IOAT_DMACAP_XOR (1 << 8) +#define IOAT_DMACAP_PQ (1 << 9) +#define IOAT_DMACAP_DMA_DIF (1 << 10) + struct ioat_registers { uint8_t chancnt; uint8_t xfercap; @@ -71,8 +80,214 @@ struct ioat_registers { #define IOAT_CHANCMD_RESET 0x20 #define IOAT_CHANCMD_SUSPEND 0x04 +#define IOAT_CHANSTS_STATUS 0x7ULL +#define IOAT_CHANSTS_ACTIVE 0x0 +#define IOAT_CHANSTS_IDLE 0x1 +#define IOAT_CHANSTS_SUSPENDED 0x2 +#define IOAT_CHANSTS_HALTED 0x3 +#define IOAT_CHANSTS_ARMED 0x4 + +#define IOAT_CHANERR_INVALID_SRC_ADDR_MASK (1 << 0) +#define IOAT_CHANERR_INVALID_DST_ADDR_MASK (1 << 1) +#define IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK (1 << 8) +#define IOAT_CHANERR_INVALID_LENGTH_MASK (1 << 10) + +const char *chansts_readable[] = { + "ACTIVE", /* 0x0 */ + "IDLE", /* 0x1 */ + "SUSPENDED", /* 0x2 */ + "HALTED", /* 0x3 */ + "ARMED" /* 0x4 */ +}; + +#define IOAT_CHANSTS_UNAFFILIATED_ERROR 0x8ULL +#define IOAT_CHANSTS_SOFT_ERROR 0x10ULL + +#define IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK (~0x3FULL) + #define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ +struct ioat_dma_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t null: 1; + uint32_t src_page_break: 1; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_COPY 0x00 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t reserved2; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_fill_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t reserved: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t reserved2: 2; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t reserved3: 15; +#define IOAT_OP_FILL 0x01 + uint32_t op: 8; + } control; + } u; + uint64_t src_data; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t next_dest_addr; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_xor_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_XOR 0x87 +#define IOAT_OP_XOR_VAL 0x88 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint64_t src_addr4; + uint64_t src_addr5; +}; + +struct ioat_xor_ext_hw_desc { + uint64_t src_addr6; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t next; + uint64_t reserved[4]; +}; + +struct ioat_pq_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 11; +#define IOAT_OP_PQ 0x89 +#define IOAT_OP_PQ_VAL 0x8a + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint8_t coef[8]; + uint64_t q_addr; +}; + +struct ioat_pq_ext_hw_desc { + uint64_t src_addr4; + uint64_t src_addr5; + uint64_t src_addr6; + uint64_t next; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t reserved[2]; +}; + +struct ioat_pq_update_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_cnt: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 3; + uint32_t coef: 8; +#define IOAT_OP_PQ_UP 0x8b + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t p_src; + uint64_t q_src; + uint64_t q_addr; +}; + +union ioat_hw_desc { + struct ioat_dma_hw_desc dma; + struct ioat_fill_hw_desc fill; + struct ioat_xor_hw_desc xor_desc; + struct ioat_xor_ext_hw_desc xor_ext; + struct ioat_pq_hw_desc pq; + struct ioat_pq_ext_hw_desc pq_ext; + struct ioat_pq_update_hw_desc pq_update; +}; + +#define GENSTS_DEV_STATE_MASK 0x03 +#define CMDSTATUS_ACTIVE_SHIFT 31 +#define CMDSTATUS_ACTIVE_MASK (1 << 31) +#define CMDSTATUS_ERR_MASK 0xFF + #ifdef __cplusplus } #endif -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v6 04/12] dma/ioat: add configuration functions 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (2 preceding siblings ...) 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 03/12] dma/ioat: add datapath structures Conor Walsh @ 2021-09-27 10:21 ` Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 05/12] dma/ioat: add start and stop functions Conor Walsh ` (7 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-27 10:21 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add functions for device configuration. The info_get and close functions are included here also. info_get can be useful for checking successful configuration and close is used by the dmadev api when releasing a configured device. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 19 ++++++ drivers/dma/ioat/ioat_dmadev.c | 107 +++++++++++++++++++++++++++++++++ 2 files changed, 126 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 9ae1d8a2ad..b1f847d273 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -67,3 +67,22 @@ For example:: Once probed successfully, the device will appear as a ``dmadev``, that is a "DMA device type" inside DPDK, and can be accessed using APIs from the ``rte_dmadev`` library. + +Using IOAT DMAdev Devices +-------------------------- + +To use IOAT devices from an application, the ``dmadev`` API can be used. + +Device Configuration +~~~~~~~~~~~~~~~~~~~~~ + +Refer to the :ref:`Device Configuration <dmadev_device_configuration>` and +:ref:`Configuration of Virtual DMA Channels <dmadev_vchan_configuration>` sections +of the dmadev library documentation for details on device configuration API usage. + +IOAT configuration requirements: + +* ``ring_size`` must be a power of two, between 64 and 4096. +* Only one ``vchan`` is supported per device. +* Silent mode is not supported. +* The transfer direction must be set to ``RTE_DMA_DIR_MEM_TO_MEM`` to copy from memory to memory. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index b132283ba5..92c4e2b04f 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -12,9 +12,112 @@ static struct rte_pci_driver ioat_pmd_drv; RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); +#define DESC_SZ sizeof(struct ioat_dma_hw_desc) + #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Configure a device. */ +static int +ioat_dev_configure(struct rte_dma_dev *dev __rte_unused, const struct rte_dma_conf *dev_conf, + uint32_t conf_sz) +{ + if (sizeof(struct rte_dma_conf) != conf_sz) + return -EINVAL; + + if (dev_conf->nb_vchans != 1) + return -EINVAL; + + return 0; +} + +/* Setup a virtual channel for IOAT, only 1 vchan is supported. */ +static int +ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + const struct rte_dma_vchan_conf *qconf, uint32_t qconf_sz) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint16_t max_desc = qconf->nb_desc; + int i; + + if (sizeof(struct rte_dma_vchan_conf) != qconf_sz) + return -EINVAL; + + ioat->qcfg = *qconf; + + if (!rte_is_power_of_2(max_desc)) { + max_desc = rte_align32pow2(max_desc); + IOAT_PMD_DEBUG("DMA dev %u using %u descriptors", dev->data->dev_id, max_desc); + ioat->qcfg.nb_desc = max_desc; + } + + /* In case we are reconfiguring a device, free any existing memory. */ + rte_free(ioat->desc_ring); + + ioat->desc_ring = rte_zmalloc(NULL, sizeof(*ioat->desc_ring) * max_desc, 0); + if (ioat->desc_ring == NULL) + return -ENOMEM; + + ioat->ring_addr = rte_mem_virt2iova(ioat->desc_ring); + + ioat->status_addr = rte_mem_virt2iova(ioat) + offsetof(struct ioat_dmadev, status); + + /* Ensure all counters are reset, if reconfiguring/restarting device. */ + ioat->next_read = 0; + ioat->next_write = 0; + ioat->last_write = 0; + ioat->offset = 0; + ioat->failure = 0; + + /* Configure descriptor ring - each one points to next. */ + for (i = 0; i < ioat->qcfg.nb_desc; i++) { + ioat->desc_ring[i].next = ioat->ring_addr + + (((i + 1) % ioat->qcfg.nb_desc) * DESC_SZ); + } + + return 0; +} + +/* Get device information of a device. */ +static int +ioat_dev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *info, uint32_t size) +{ + struct ioat_dmadev *ioat = dev->dev_private; + if (size < sizeof(*info)) + return -EINVAL; + info->dev_capa = RTE_DMA_CAPA_MEM_TO_MEM | + RTE_DMA_CAPA_OPS_COPY | + RTE_DMA_CAPA_OPS_FILL; + if (ioat->version >= IOAT_VER_3_4) + info->dev_capa |= RTE_DMA_CAPA_HANDLES_ERRORS; + info->max_vchans = 1; + info->min_desc = 32; + info->max_desc = 4096; + return 0; +} + +/* Close a configured device. */ +static int +ioat_dev_close(struct rte_dma_dev *dev) +{ + struct ioat_dmadev *ioat; + + if (!dev) { + IOAT_PMD_ERR("Invalid device"); + return -EINVAL; + } + + ioat = dev->dev_private; + if (!ioat) { + IOAT_PMD_ERR("Error getting dev_private"); + return -EINVAL; + } + + rte_free(ioat->desc_ring); + + return 0; +} + /* Dump DMA device info. */ static int ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) @@ -79,7 +182,11 @@ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { static const struct rte_dma_dev_ops ioat_dmadev_ops = { + .dev_close = ioat_dev_close, + .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, + .dev_info_get = ioat_dev_info_get, + .vchan_setup = ioat_vchan_setup, }; struct rte_dma_dev *dmadev = NULL; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v6 05/12] dma/ioat: add start and stop functions 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (3 preceding siblings ...) 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 04/12] dma/ioat: add configuration functions Conor Walsh @ 2021-09-27 10:21 ` Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 06/12] dma/ioat: add data path job submission functions Conor Walsh ` (6 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-27 10:21 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add start, stop and recover functions for IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 3 ++ drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index b1f847d273..d93d28023f 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -86,3 +86,6 @@ IOAT configuration requirements: * Only one ``vchan`` is supported per device. * Silent mode is not supported. * The transfer direction must be set to ``RTE_DMA_DIR_MEM_TO_MEM`` to copy from memory to memory. + +Once configured, the device can then be made ready for use by calling the +``rte_dma_start()`` API. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 92c4e2b04f..96bf55135f 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -78,6 +78,96 @@ ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, return 0; } +/* Recover IOAT device. */ +static inline int +__ioat_recover(struct ioat_dmadev *ioat) +{ + uint32_t chanerr, retry = 0; + uint16_t mask = ioat->qcfg.nb_desc - 1; + + /* Clear any channel errors. Reading and writing to chanerr does this. */ + chanerr = ioat->regs->chanerr; + ioat->regs->chanerr = chanerr; + + /* Reset Channel. */ + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + + /* Write new chain address to trigger state change. */ + ioat->regs->chainaddr = ioat->desc_ring[(ioat->next_read - 1) & mask].next; + /* Ensure channel control and status addr are correct. */ + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + ioat->regs->chancmp = ioat->status_addr; + + /* Allow HW time to move to the ARMED state. */ + do { + rte_pause(); + retry++; + } while (ioat->regs->chansts != IOAT_CHANSTS_ARMED && retry < 200); + + /* Exit as failure if device is still HALTED. */ + if (ioat->regs->chansts != IOAT_CHANSTS_ARMED) + return -1; + + /* Store next write as offset as recover will move HW and SW ring out of sync. */ + ioat->offset = ioat->next_read; + + /* Prime status register with previous address. */ + ioat->status = ioat->desc_ring[(ioat->next_read - 2) & mask].next; + + return 0; +} + +/* Start a configured device. */ +static int +ioat_dev_start(struct rte_dma_dev *dev) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + if (ioat->qcfg.nb_desc == 0 || ioat->desc_ring == NULL) + return -EBUSY; + + /* Inform hardware of where the descriptor ring is. */ + ioat->regs->chainaddr = ioat->ring_addr; + /* Inform hardware of where to write the status/completions. */ + ioat->regs->chancmp = ioat->status_addr; + + /* Prime the status register to be set to the last element. */ + ioat->status = ioat->ring_addr + ((ioat->qcfg.nb_desc - 1) * DESC_SZ); + + printf("IOAT.status: %s [0x%"PRIx64"]\n", + chansts_readable[ioat->status & IOAT_CHANSTS_STATUS], + ioat->status); + + if ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_HALTED) { + IOAT_PMD_WARN("Device HALTED on start, attempting to recover\n"); + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device couldn't be recovered"); + return -1; + } + } + + return 0; +} + +/* Stop a configured device. */ +static int +ioat_dev_stop(struct rte_dma_dev *dev) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint32_t retry = 0; + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + + do { + rte_pause(); + retry++; + } while ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) != IOAT_CHANSTS_SUSPENDED + && retry < 200); + + return ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_SUSPENDED) ? 0 : -1; +} + /* Get device information of a device. */ static int ioat_dev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *info, uint32_t size) @@ -186,6 +276,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, .dev_info_get = ioat_dev_info_get, + .dev_start = ioat_dev_start, + .dev_stop = ioat_dev_stop, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v6 06/12] dma/ioat: add data path job submission functions 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (4 preceding siblings ...) 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 05/12] dma/ioat: add start and stop functions Conor Walsh @ 2021-09-27 10:21 ` Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 07/12] dma/ioat: add data path completion functions Conor Walsh ` (5 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-27 10:21 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add data path functions for enqueuing and submitting operations to IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Reviewed-by: Chengwen Feng <fengchengwen@huawei.com> --- doc/guides/dmadevs/ioat.rst | 9 ++++ drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index d93d28023f..ec8ce5a8e5 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -89,3 +89,12 @@ IOAT configuration requirements: Once configured, the device can then be made ready for use by calling the ``rte_dma_start()`` API. + +Performing Data Copies +~~~~~~~~~~~~~~~~~~~~~~~ + +Refer to the :ref:`Enqueue / Dequeue APIs <dmadev_enqueue_dequeue>` section of the dmadev library +documentation for details on operation enqueue and submission API usage. + +It is expected that, for efficiency reasons, a burst of operations will be enqueued to the +device via multiple enqueue calls between calls to the ``rte_dma_submit()`` function. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 96bf55135f..0e92c80fb0 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -5,6 +5,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> #include <rte_malloc.h> +#include <rte_prefetch.h> #include "ioat_internal.h" @@ -17,6 +18,12 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* IOAT operations. */ +enum rte_ioat_ops { + ioat_op_copy = 0, /* Standard DMA Operation */ + ioat_op_fill /* Block Fill */ +}; + /* Configure a device. */ static int ioat_dev_configure(struct rte_dma_dev *dev __rte_unused, const struct rte_dma_conf *dev_conf, @@ -208,6 +215,87 @@ ioat_dev_close(struct rte_dma_dev *dev) return 0; } +/* Trigger hardware to begin performing enqueued operations. */ +static inline void +__submit(struct ioat_dmadev *ioat) +{ + *ioat->doorbell = ioat->next_write - ioat->offset; + + ioat->last_write = ioat->next_write; +} + +/* External submit function wrapper. */ +static int +ioat_submit(struct rte_dma_dev *dev, uint16_t qid __rte_unused) +{ + struct ioat_dmadev *ioat = (struct ioat_dmadev *)dev->dev_private; + + __submit(ioat); + + return 0; +} + +/* Write descriptor for enqueue. */ +static inline int +__write_desc(struct rte_dma_dev *dev, uint32_t op, uint64_t src, phys_addr_t dst, + unsigned int length, uint64_t flags) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint16_t ret; + const unsigned short mask = ioat->qcfg.nb_desc - 1; + const unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + const unsigned short space = mask + read - write; + struct ioat_dma_hw_desc *desc; + + if (space == 0) + return -ENOSPC; + + ioat->next_write = write + 1; + write &= mask; + + desc = &ioat->desc_ring[write]; + desc->size = length; + desc->u.control_raw = (uint32_t)((op << IOAT_CMD_OP_SHIFT) | + (1 << IOAT_COMP_UPDATE_SHIFT)); + + /* In IOAT the fence ensures that all operations including the current one + * are completed before moving on, DMAdev assumes that the fence ensures + * all operations before the current one are completed before starting + * the current one, so in IOAT we set the fence for the previous descriptor. + */ + if (flags & RTE_DMA_OP_FLAG_FENCE) + ioat->desc_ring[(write - 1) & mask].u.control.fence = 1; + + desc->src_addr = src; + desc->dest_addr = dst; + + rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]); + + ret = (uint16_t)(ioat->next_write - 1); + + if (flags & RTE_DMA_OP_FLAG_SUBMIT) + __submit(ioat); + + return ret; +} + +/* Enqueue a fill operation onto the ioat device. */ +static int +ioat_enqueue_fill(struct rte_dma_dev *dev, uint16_t qid __rte_unused, uint64_t pattern, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev, ioat_op_fill, pattern, dst, length, flags); +} + +/* Enqueue a copy operation onto the ioat device. */ +static int +ioat_enqueue_copy(struct rte_dma_dev *dev, uint16_t qid __rte_unused, rte_iova_t src, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev, ioat_op_copy, src, dst, length, flags); +} + /* Dump DMA device info. */ static int ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) @@ -303,6 +391,10 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->copy = ioat_enqueue_copy; + dmadev->fill = ioat_enqueue_fill; + dmadev->submit = ioat_submit; + ioat = dmadev->data->dev_private; ioat->regs = dev->mem_resource[0].addr; ioat->doorbell = &ioat->regs->dmacount; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v6 07/12] dma/ioat: add data path completion functions 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (5 preceding siblings ...) 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 06/12] dma/ioat: add data path job submission functions Conor Walsh @ 2021-09-27 10:21 ` Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 08/12] dma/ioat: add statistics Conor Walsh ` (4 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-27 10:21 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add the data path functions for gathering completed operations from IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- doc/guides/dmadevs/ioat.rst | 33 +++++++- drivers/dma/ioat/ioat_dmadev.c | 141 +++++++++++++++++++++++++++++++++ 2 files changed, 173 insertions(+), 1 deletion(-) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index ec8ce5a8e5..fc1b3131c7 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -94,7 +94,38 @@ Performing Data Copies ~~~~~~~~~~~~~~~~~~~~~~~ Refer to the :ref:`Enqueue / Dequeue APIs <dmadev_enqueue_dequeue>` section of the dmadev library -documentation for details on operation enqueue and submission API usage. +documentation for details on operation enqueue, submission and completion API usage. It is expected that, for efficiency reasons, a burst of operations will be enqueued to the device via multiple enqueue calls between calls to the ``rte_dma_submit()`` function. + +When gathering completions, ``rte_dma_completed()`` should be used, up until the point an error +occurs with an operation. If an error was encountered, ``rte_dma_completed_status()`` must be used +to reset the device and continue processing operations. This function will also gather the status +of each individual operation which is filled in to the ``status`` array provided as parameter +by the application. + +The status codes supported by IOAT are: + +* ``RTE_DMA_STATUS_SUCCESSFUL``: The operation was successful. +* ``RTE_DMA_STATUS_INVALID_SRC_ADDR``: The operation failed due to an invalid source address. +* ``RTE_DMA_STATUS_INVALID_DST_ADDR``: The operation failed due to an invalid destination address. +* ``RTE_DMA_STATUS_INVALID_LENGTH``: The operation failed due to an invalid descriptor length. +* ``RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR``: The device could not read the descriptor. +* ``RTE_DMA_STATUS_ERROR_UNKNOWN``: The operation failed due to an unspecified error. + +The following code shows how to retrieve the number of successfully completed +copies within a burst and then uses ``rte_dma_completed_status()`` to check +which operation failed and reset the device to continue processing operations: + +.. code-block:: C + + enum rte_dma_status_code status[COMP_BURST_SZ]; + uint16_t count, idx, status_count; + bool error = 0; + + count = rte_dma_completed(dev_id, vchan, COMP_BURST_SZ, &idx, &error); + + if (error){ + status_count = rte_dma_completed_status(dev_id, vchan, COMP_BURST_SZ, &idx, status); + } diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 0e92c80fb0..b2b7ebb3db 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -6,6 +6,7 @@ #include <rte_dmadev_pmd.h> #include <rte_malloc.h> #include <rte_prefetch.h> +#include <rte_errno.h> #include "ioat_internal.h" @@ -355,6 +356,144 @@ ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) return 0; } +/* Returns the index of the last completed operation. */ +static inline uint16_t +__get_last_completed(const struct ioat_dmadev *ioat, int *state) +{ + /* Status register contains the address of the completed operation */ + uint64_t status = ioat->status; + + /* lower 3 bits indicate "transfer status" : active, idle, halted. + * We can ignore bit 0. + */ + *state = status & IOAT_CHANSTS_STATUS; + + /* If we are just after recovering from an error the address returned by + * status will be 0, in this case we return the offset - 1 as the last + * completed. If not return the status value minus the chainaddr which + * gives us an offset into the ring. Right shifting by 6 (divide by 64) + * gives the index of the completion from the HW point of view and adding + * the offset translates the ring index from HW to SW point of view. + */ + if ((status & ~IOAT_CHANSTS_STATUS) == 0) + return ioat->offset - 1; + + return (status - ioat->ring_addr) >> 6; +} + +/* Translates IOAT ChanERRs to DMA error codes. */ +static inline enum rte_dma_status_code +__translate_status_ioat_to_dma(uint32_t chanerr) +{ + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_SRC_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_DST_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + return RTE_DMA_STATUS_INVALID_LENGTH; + else if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + return RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR; + else + return RTE_DMA_STATUS_ERROR_UNKNOWN; +} + +/* Returns details of operations that have been completed. */ +static uint16_t +ioat_completed(struct rte_dma_dev *dev, uint16_t qid __rte_unused, const uint16_t max_ops, + uint16_t *last_idx, bool *has_error) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short last_completed, count; + int state, fails = 0; + + /* Do not do any work if there is an uncleared error. */ + if (ioat->failure != 0) { + *has_error = true; + *last_idx = ioat->next_read - 2; + return 0; + } + + last_completed = __get_last_completed(ioat, &state); + count = (last_completed + 1 - read) & mask; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || state != IOAT_CHANSTS_HALTED) { + ioat->next_read = read + count; + *last_idx = ioat->next_read - 1; + } else { + *has_error = true; + rte_errno = EIO; + ioat->failure = ioat->regs->chanerr; + ioat->next_read = read + count + 1; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + ioat_dev_dump(dev, stdout); + return 0; + } + __submit(ioat); + fails++; + *last_idx = ioat->next_read - 2; + } + + return count; +} + +/* Returns detailed status information about operations that have been completed. */ +static uint16_t +ioat_completed_status(struct rte_dma_dev *dev, uint16_t qid __rte_unused, + uint16_t max_ops, uint16_t *last_idx, enum rte_dma_status_code *status) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short count, last_completed; + uint64_t fails = 0; + int state, i; + + last_completed = __get_last_completed(ioat, &state); + count = (last_completed + 1 - read) & mask; + + for (i = 0; i < RTE_MIN(count + 1, max_ops); i++) + status[i] = RTE_DMA_STATUS_SUCCESSFUL; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || state != IOAT_CHANSTS_HALTED) + ioat->next_read = read + count; + else { + rte_errno = EIO; + status[count] = __translate_status_ioat_to_dma(ioat->regs->chanerr); + count++; + ioat->next_read = read + count; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + ioat_dev_dump(dev, stdout); + return 0; + } + __submit(ioat); + fails++; + } + + if (ioat->failure > 0) { + status[0] = __translate_status_ioat_to_dma(ioat->failure); + count = RTE_MIN(count + 1, max_ops); + ioat->failure = 0; + } + + *last_idx = ioat->next_read - 1; + + return count; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -391,6 +530,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->completed = ioat_completed; + dmadev->completed_status = ioat_completed_status; dmadev->copy = ioat_enqueue_copy; dmadev->fill = ioat_enqueue_fill; dmadev->submit = ioat_submit; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v6 08/12] dma/ioat: add statistics 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (6 preceding siblings ...) 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 07/12] dma/ioat: add data path completion functions Conor Walsh @ 2021-09-27 10:21 ` Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 09/12] dma/ioat: add support for vchan status function Conor Walsh ` (3 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-27 10:21 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add statistic tracking for operations in IOAT. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 43 ++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index b2b7ebb3db..20ae364318 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -77,6 +77,9 @@ ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, ioat->offset = 0; ioat->failure = 0; + /* Reset Stats. */ + ioat->stats = (struct rte_dma_stats){0}; + /* Configure descriptor ring - each one points to next. */ for (i = 0; i < ioat->qcfg.nb_desc; i++) { ioat->desc_ring[i].next = ioat->ring_addr + @@ -222,6 +225,8 @@ __submit(struct ioat_dmadev *ioat) { *ioat->doorbell = ioat->next_write - ioat->offset; + ioat->stats.submitted += (uint16_t)(ioat->next_write - ioat->last_write); + ioat->last_write = ioat->next_write; } @@ -352,6 +357,10 @@ ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) fprintf(f, " Dest: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); fprintf(f, " Next: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].next); fprintf(f, " }\n"); + fprintf(f, " Key Stats { submitted: %"PRIu64", comp: %"PRIu64", failed: %"PRIu64" }\n", + ioat->stats.submitted, + ioat->stats.completed, + ioat->stats.errors); return 0; } @@ -441,6 +450,9 @@ ioat_completed(struct rte_dma_dev *dev, uint16_t qid __rte_unused, const uint16_ *last_idx = ioat->next_read - 2; } + ioat->stats.completed += count; + ioat->stats.errors += fails; + return count; } @@ -491,9 +503,38 @@ ioat_completed_status(struct rte_dma_dev *dev, uint16_t qid __rte_unused, *last_idx = ioat->next_read - 1; + ioat->stats.completed += count; + ioat->stats.errors += fails; + return count; } +/* Retrieve the generic stats of a DMA device. */ +static int +ioat_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + struct rte_dma_stats *rte_stats, uint32_t size) +{ + struct rte_dma_stats *stats = (&((struct ioat_dmadev *)dev->dev_private)->stats); + + if (size < sizeof(rte_stats)) + return -EINVAL; + if (rte_stats == NULL) + return -EINVAL; + + *rte_stats = *stats; + return 0; +} + +/* Reset the generic stat counters for the DMA device. */ +static int +ioat_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + ioat->stats = (struct rte_dma_stats){0}; + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -505,6 +546,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_info_get = ioat_dev_info_get, .dev_start = ioat_dev_start, .dev_stop = ioat_dev_stop, + .stats_get = ioat_stats_get, + .stats_reset = ioat_stats_reset, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v6 09/12] dma/ioat: add support for vchan status function 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (7 preceding siblings ...) 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 08/12] dma/ioat: add statistics Conor Walsh @ 2021-09-27 10:21 ` Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 10/12] dma/ioat: add burst capacity function Conor Walsh ` (2 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-27 10:21 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add support for the rte_dmadev_vchan_status API call. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 20ae364318..fe01a3b1db 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -535,6 +535,26 @@ ioat_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused) return 0; } +/* Check if the IOAT device is idle. */ +static int +ioat_vchan_status(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + enum rte_dma_vchan_status *status) +{ + int state = 0; + const struct ioat_dmadev *ioat = dev->dev_private; + const uint16_t mask = ioat->qcfg.nb_desc - 1; + const uint16_t last = __get_last_completed(ioat, &state); + + if (state == IOAT_CHANSTS_HALTED || state == IOAT_CHANSTS_SUSPENDED) + *status = RTE_DMA_VCHAN_HALTED_ERROR; + else if (last == ((ioat->next_write - 1) & mask)) + *status = RTE_DMA_VCHAN_IDLE; + else + *status = RTE_DMA_VCHAN_ACTIVE; + + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -548,6 +568,7 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_stop = ioat_dev_stop, .stats_get = ioat_stats_get, .stats_reset = ioat_stats_reset, + .vchan_status = ioat_vchan_status, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v6 10/12] dma/ioat: add burst capacity function 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (8 preceding siblings ...) 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 09/12] dma/ioat: add support for vchan status function Conor Walsh @ 2021-09-27 10:21 ` Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 11/12] devbind: move ioat device IDs to dmadev category Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-27 10:21 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Adds the ability to find the remaining space in the IOAT ring. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index fe01a3b1db..6c783ec94f 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -509,6 +509,19 @@ ioat_completed_status(struct rte_dma_dev *dev, uint16_t qid __rte_unused, return count; } +/* Get the remaining capacity of the ring. */ +static uint16_t +ioat_burst_capacity(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused) +{ + struct ioat_dmadev *ioat = dev->data->dev_private; + unsigned short size = ioat->qcfg.nb_desc - 1; + unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + unsigned short space = size - (write - read); + + return space; +} + /* Retrieve the generic stats of a DMA device. */ static int ioat_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, @@ -594,6 +607,7 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->burst_capacity = ioat_burst_capacity; dmadev->completed = ioat_completed; dmadev->completed_status = ioat_completed_status; dmadev->copy = ioat_enqueue_copy; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v6 11/12] devbind: move ioat device IDs to dmadev category 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (9 preceding siblings ...) 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 10/12] dma/ioat: add burst capacity function Conor Walsh @ 2021-09-27 10:21 ` Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-27 10:21 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Move Intel IOAT devices from Misc to DMA devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Reviewed-by: Bruce Richardson <bruce.richardson@intel.com> --- usertools/dpdk-devbind.py | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index 15d438715f..4a72229622 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -69,14 +69,12 @@ network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] -dma_devices = [intel_idxd_spr] +dma_devices = [intel_idxd_spr, intel_ioat_bdw, intel_ioat_icx, intel_ioat_skx] eventdev_devices = [cavium_sso, cavium_tim, intel_dlb, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] regex_devices = [octeontx2_ree] -misc_devices = [cnxk_bphy, cnxk_bphy_cgx, intel_ioat_bdw, intel_ioat_skx, intel_ioat_icx, - intel_ntb_skx, intel_ntb_icx, - octeontx2_dma] +misc_devices = [cnxk_bphy, cnxk_bphy_cgx, intel_ntb_skx, intel_ntb_icx, octeontx2_dma] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v6 12/12] raw/ioat: deprecate ioat rawdev driver 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (10 preceding siblings ...) 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 11/12] devbind: move ioat device IDs to dmadev category Conor Walsh @ 2021-09-27 10:21 ` Conor Walsh 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-09-27 10:21 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Deprecate the rawdev IOAT driver as both IOAT and IDXD drivers have moved to dmadev. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Acked-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- MAINTAINERS | 2 +- doc/guides/rawdevs/ioat.rst | 4 ++++ doc/guides/rel_notes/deprecation.rst | 7 +++++++ 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/MAINTAINERS b/MAINTAINERS index ccabba9169..a4bcd2d024 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1322,7 +1322,7 @@ T: git://dpdk.org/next/dpdk-next-net-intel F: drivers/raw/ifpga/ F: doc/guides/rawdevs/ifpga.rst -IOAT Rawdev +IOAT Rawdev - DEPRECATED M: Bruce Richardson <bruce.richardson@intel.com> F: drivers/raw/ioat/ F: doc/guides/rawdevs/ioat.rst diff --git a/doc/guides/rawdevs/ioat.rst b/doc/guides/rawdevs/ioat.rst index a65530bd30..98d15dd032 100644 --- a/doc/guides/rawdevs/ioat.rst +++ b/doc/guides/rawdevs/ioat.rst @@ -6,6 +6,10 @@ IOAT Rawdev Driver =================== +.. warning:: + As of DPDK 21.11 the rawdev implementation of the IOAT driver has been deprecated. + Please use the dmadev library instead. + The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg| Data Streaming Accelerator `(Intel DSA) <https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator>`_ and for Intel\ |reg| diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst index 59445a6f42..b1dbb45802 100644 --- a/doc/guides/rel_notes/deprecation.rst +++ b/doc/guides/rel_notes/deprecation.rst @@ -284,3 +284,10 @@ Deprecation Notices reserved bytes to 2 (from 3), and use 1 byte to indicate warnings and other information from the crypto/security operation. This field will be used to communicate events such as soft expiry with IPsec in lookaside mode. + +* raw/ioat: The ``ioat`` rawdev driver has been deprecated, since it's + functionality is provided through the new ``dmadev`` infrastructure. + To continue to use hardware previously supported by the ``ioat`` rawdev driver, + applications should be updated to use the ``dmadev`` library instead, + with the underlying HW-functionality being provided by the ``ioat`` or + ``idxd`` dma drivers -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh ` (11 preceding siblings ...) 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh @ 2021-10-14 9:48 ` Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 01/12] dma/ioat: add device probe and removal functions Conor Walsh ` (11 more replies) 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh 13 siblings, 12 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-14 9:48 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh This patchset adds a dmadev driver and associated documentation to support Intel QuickData Technology devices, part of the Intel I/O Acceleration Technology (Intel I/OAT). This driver is intended to ultimately replace the current IOAT part of the IOAT rawdev driver. This patchset passes all the driver tests added in the dmadev test suite. NOTE: This patchset has several dependencies: - v26 of the dmadev set [1] - v7 of the dmadev test suite [2] - v7 of the IDXD driver [3] [1] http://patches.dpdk.org/project/dpdk/list/?series=19594 [2] http://patches.dpdk.org/project/dpdk/list/?series=19599 [3] http://patches.dpdk.org/project/dpdk/list/?series=19603 --- v7: - Minor rework to update from v23 to v26 of the dmadev lib. v6: - Added rawdev IOAT deprecation notice to deprecation.rst. v5: - Updated to v23 of the dmadev lib. - Removed experimental tag for driver from MAINTAINERS. - Seperated IOAT and IDXD announcements in release notes. - Added missing check for rte_dma_get_dev_id in destroy. - Fixed memleak in destroy caused by NULL pointer. - Rewrote part of the docs to reduce duplication with DMA and IDXD. - Added patch to deprecate the rawdev IOAT driver. - Reworked destroy and close functions. - Added RTE_DMA_CAPA_HANDLES_ERRORS flag for IOAT versions >=3.4. - Other minor changes to IOAT driver. v4: - Changes needed to update from dmadev v21 to v22. - Fixed 32-bit build. - Made stats reset logic easier to understand. v3: - Added burst capacity function. - Stop function now waits for suspend rather than just using a sleep. - Changed from vchan idle to vchan status function. - Other minor changes to update from dmadev v19 to v21. v2: - Rebased on the above patchsets. Conor Walsh (12): dma/ioat: add device probe and removal functions dma/ioat: create dmadev instances on PCI probe dma/ioat: add datapath structures dma/ioat: add configuration functions dma/ioat: add start and stop functions dma/ioat: add data path job submission functions dma/ioat: add data path completion functions dma/ioat: add statistics dma/ioat: add support for vchan status function dma/ioat: add burst capacity function devbind: move ioat device IDs to dmadev category raw/ioat: deprecate ioat rawdev driver MAINTAINERS | 8 +- doc/guides/dmadevs/index.rst | 2 + doc/guides/dmadevs/ioat.rst | 127 +++++ doc/guides/rawdevs/ioat.rst | 4 + doc/guides/rel_notes/deprecation.rst | 7 + doc/guides/rel_notes/release_21_11.rst | 6 + drivers/dma/ioat/ioat_dmadev.c | 748 +++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 295 ++++++++++ drivers/dma/ioat/ioat_internal.h | 47 ++ drivers/dma/ioat/meson.build | 7 + drivers/dma/ioat/version.map | 3 + drivers/dma/meson.build | 1 + usertools/dpdk-devbind.py | 7 +- 13 files changed, 1257 insertions(+), 5 deletions(-) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v7 01/12] dma/ioat: add device probe and removal functions 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh @ 2021-10-14 9:48 ` Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 02/12] dma/ioat: create dmadev instances on PCI probe Conor Walsh ` (10 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-14 9:48 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add the basic device probe/remove skeleton code and initial documentation for new IOAT DMA driver. Maintainers update is also included in this patch. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Reviewed-by: Chengwen Feng <fengchengwen@huawei.com> --- MAINTAINERS | 6 +++ doc/guides/dmadevs/index.rst | 2 + doc/guides/dmadevs/ioat.rst | 69 ++++++++++++++++++++++++++ doc/guides/rel_notes/release_21_11.rst | 6 +++ drivers/dma/ioat/ioat_dmadev.c | 69 ++++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 35 +++++++++++++ drivers/dma/ioat/ioat_internal.h | 20 ++++++++ drivers/dma/ioat/meson.build | 7 +++ drivers/dma/ioat/version.map | 3 ++ drivers/dma/meson.build | 1 + 10 files changed, 218 insertions(+) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map diff --git a/MAINTAINERS b/MAINTAINERS index 423d8a73ce..283c70f7d7 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1209,6 +1209,12 @@ M: Kevin Laatz <kevin.laatz@intel.com> F: drivers/dma/idxd/ F: doc/guides/dmadevs/idxd.rst +Intel IOAT +M: Bruce Richardson <bruce.richardson@intel.com> +M: Conor Walsh <conor.walsh@intel.com> +F: drivers/dma/ioat/ +F: doc/guides/dmadevs/ioat.rst + RegEx Drivers ------------- diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst index 5d4abf880e..c59f4b5c92 100644 --- a/doc/guides/dmadevs/index.rst +++ b/doc/guides/dmadevs/index.rst @@ -12,3 +12,5 @@ an application through DMA API. :numbered: idxd + ioat + diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst new file mode 100644 index 0000000000..9ae1d8a2ad --- /dev/null +++ b/doc/guides/dmadevs/ioat.rst @@ -0,0 +1,69 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2021 Intel Corporation. + +.. include:: <isonum.txt> + +IOAT DMA Device Driver +======================= + +The ``ioat`` dmadev driver provides a poll-mode driver (PMD) for Intel\ +|reg| QuickData Technology which is part of part of Intel\ |reg| I/O +Acceleration Technology (`Intel I/OAT +<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_). +This PMD, when used on supported hardware, allows data copies, for example, +cloning packet data, to be accelerated by IOAT hardware rather than having to +be done by software, freeing up CPU cycles for other tasks. + +Hardware Requirements +---------------------- + +The ``dpdk-devbind.py`` script, included with DPDK, can be used to show the +presence of supported hardware. Running ``dpdk-devbind.py --status-dev dma`` +will show all the DMA devices on the system, IOAT devices are included in this +list. For Intel\ |reg| IOAT devices, the hardware will often be listed as +"Crystal Beach DMA", or "CBDMA" or on some newer systems '0b00' due to the +absence of pci-id database entries for them at this point. + +.. note:: + Error handling is not supported by this driver on hardware prior to + Intel Ice Lake. Unsupported systems include Broadwell, Skylake and + Cascade Lake. + +Compilation +------------ + +For builds using ``meson`` and ``ninja``, the driver will be built when the +target platform is x86-based. No additional compilation steps are necessary. + +Device Setup +------------- + +Intel\ |reg| IOAT devices will need to be bound to a suitable DPDK-supported +user-space IO driver such as ``vfio-pci`` in order to be used by DPDK. + +The ``dpdk-devbind.py`` script can be used to view the state of the devices using:: + + $ dpdk-devbind.py --status-dev dma + +The ``dpdk-devbind.py`` script can also be used to bind devices to a suitable driver. +For example:: + + $ dpdk-devbind.py -b vfio-pci 00:01.0 00:01.1 + +Device Probing and Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For devices bound to a suitable DPDK-supported driver (``vfio-pci``), the HW +devices will be found as part of the device scan done at application +initialization time without the need to pass parameters to the application. + +If the application does not require all the devices available an allowlist can +be used in the same way that other DPDK devices use them. + +For example:: + + $ dpdk-test -a <b:d:f> + +Once probed successfully, the device will appear as a ``dmadev``, that is a +"DMA device type" inside DPDK, and can be accessed using APIs from the +``rte_dmadev`` library. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 65868a730a..9f99a5b380 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -85,6 +85,12 @@ New Features The IDXD dmadev driver provide device drivers for the Intel DSA devices. This device driver can be used through the generic dmadev API. +* **Added IOAT dmadev driver implementation.** + + The Intel I/O Acceleration Technology (IOAT) dmadev driver provides a device + driver for Intel IOAT devices such as Crystal Beach DMA (CBDMA) on Ice Lake, + Skylake and Broadwell. This device driver can be used through the generic dmadev API. + * **Updated af_packet ethdev driver.** * Default VLAN strip behavior was changed. VLAN tag won't be stripped diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c new file mode 100644 index 0000000000..f3491d45b1 --- /dev/null +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#include <rte_bus_pci.h> +#include <rte_dmadev_pmd.h> + +#include "ioat_internal.h" + +static struct rte_pci_driver ioat_pmd_drv; + +RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); + +#define IOAT_PMD_NAME dmadev_ioat +#define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) + +/* Probe DMA device. */ +static int +ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); + + dev->device.driver = &drv->driver; + return 0; +} + +/* Remove DMA device. */ +static int +ioat_dmadev_remove(struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + + IOAT_PMD_INFO("Closing %s on NUMA node %d", + name, dev->device.numa_node); + + return 0; +} + +static const struct rte_pci_id pci_id_ioat_map[] = { + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_SKX) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX0) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX1) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX2) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX3) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX4) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX5) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX6) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX7) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXE) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXF) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_ICX) }, + { .vendor_id = 0, /* sentinel */ }, +}; + +static struct rte_pci_driver ioat_pmd_drv = { + .id_table = pci_id_ioat_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC, + .probe = ioat_dmadev_probe, + .remove = ioat_dmadev_remove, +}; + +RTE_PMD_REGISTER_PCI(IOAT_PMD_NAME, ioat_pmd_drv); +RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_NAME, pci_id_ioat_map); +RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_NAME, "* igb_uio | uio_pci_generic | vfio-pci"); diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h new file mode 100644 index 0000000000..eeabba41ef --- /dev/null +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#ifndef IOAT_HW_DEFS_H +#define IOAT_HW_DEFS_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> + +#define IOAT_VER_3_0 0x30 +#define IOAT_VER_3_3 0x33 + +#define IOAT_VENDOR_ID 0x8086 +#define IOAT_DEVICE_ID_SKX 0x2021 +#define IOAT_DEVICE_ID_BDX0 0x6f20 +#define IOAT_DEVICE_ID_BDX1 0x6f21 +#define IOAT_DEVICE_ID_BDX2 0x6f22 +#define IOAT_DEVICE_ID_BDX3 0x6f23 +#define IOAT_DEVICE_ID_BDX4 0x6f24 +#define IOAT_DEVICE_ID_BDX5 0x6f25 +#define IOAT_DEVICE_ID_BDX6 0x6f26 +#define IOAT_DEVICE_ID_BDX7 0x6f27 +#define IOAT_DEVICE_ID_BDXE 0x6f2E +#define IOAT_DEVICE_ID_BDXF 0x6f2F +#define IOAT_DEVICE_ID_ICX 0x0b00 + +#ifdef __cplusplus +} +#endif + +#endif /* IOAT_HW_DEFS_H */ diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h new file mode 100644 index 0000000000..f1ec12a919 --- /dev/null +++ b/drivers/dma/ioat/ioat_internal.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021 Intel Corporation + */ + +#ifndef _IOAT_INTERNAL_H_ +#define _IOAT_INTERNAL_H_ + +#include "ioat_hw_defs.h" + +extern int ioat_pmd_logtype; + +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + ioat_pmd_logtype, "IOAT: %s(): " fmt "\n", __func__, ##args) + +#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) +#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) +#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) +#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) + +#endif /* _IOAT_INTERNAL_H_ */ diff --git a/drivers/dma/ioat/meson.build b/drivers/dma/ioat/meson.build new file mode 100644 index 0000000000..d67fac96fb --- /dev/null +++ b/drivers/dma/ioat/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021 Intel Corporation + +build = dpdk_conf.has('RTE_ARCH_X86') +reason = 'only supported on x86' +sources = files('ioat_dmadev.c') +deps += ['bus_pci', 'dmadev'] diff --git a/drivers/dma/ioat/version.map b/drivers/dma/ioat/version.map new file mode 100644 index 0000000000..c2e0723b4c --- /dev/null +++ b/drivers/dma/ioat/version.map @@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +}; diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build index 411be7a240..a69418ce9b 100644 --- a/drivers/dma/meson.build +++ b/drivers/dma/meson.build @@ -3,6 +3,7 @@ drivers = [ 'idxd', + 'ioat', 'skeleton', ] std_deps = ['dmadev'] -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v7 02/12] dma/ioat: create dmadev instances on PCI probe 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 01/12] dma/ioat: add device probe and removal functions Conor Walsh @ 2021-10-14 9:48 ` Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 03/12] dma/ioat: add datapath structures Conor Walsh ` (9 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-14 9:48 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh When a suitable device is found during the PCI probe, create a dmadev instance for each channel. Internal structures and HW definitions required for device creation are also included. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 105 ++++++++++++++++++++++++++++++- drivers/dma/ioat/ioat_hw_defs.h | 45 +++++++++++++ drivers/dma/ioat/ioat_internal.h | 27 ++++++++ 3 files changed, 175 insertions(+), 2 deletions(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index f3491d45b1..90f54567a4 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -4,6 +4,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> +#include <rte_malloc.h> #include "ioat_internal.h" @@ -14,6 +15,106 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Create a DMA device. */ +static int +ioat_dmadev_create(const char *name, struct rte_pci_device *dev) +{ + static const struct rte_dma_dev_ops ioat_dmadev_ops = { }; + + struct rte_dma_dev *dmadev = NULL; + struct ioat_dmadev *ioat = NULL; + int retry = 0; + + if (!name) { + IOAT_PMD_ERR("Invalid name of the device!"); + return -EINVAL; + } + + /* Allocate device structure. */ + dmadev = rte_dma_pmd_allocate(name, dev->device.numa_node, sizeof(struct ioat_dmadev)); + if (dmadev == NULL) { + IOAT_PMD_ERR("Unable to allocate dma device"); + return -ENOMEM; + } + + dmadev->device = &dev->device; + + dmadev->fp_obj->dev_private = dmadev->data->dev_private; + + dmadev->dev_ops = &ioat_dmadev_ops; + + ioat = dmadev->data->dev_private; + ioat->dmadev = dmadev; + ioat->regs = dev->mem_resource[0].addr; + ioat->doorbell = &ioat->regs->dmacount; + ioat->qcfg.nb_desc = 0; + ioat->desc_ring = NULL; + ioat->version = ioat->regs->cbver; + + /* Do device initialization - reset and set error behaviour. */ + if (ioat->regs->chancnt != 1) + IOAT_PMD_WARN("%s: Channel count == %d\n", __func__, + ioat->regs->chancnt); + + /* Locked by someone else. */ + if (ioat->regs->chanctrl & IOAT_CHANCTRL_CHANNEL_IN_USE) { + IOAT_PMD_WARN("%s: Channel appears locked\n", __func__); + ioat->regs->chanctrl = 0; + } + + /* clear any previous errors */ + if (ioat->regs->chanerr != 0) { + uint32_t val = ioat->regs->chanerr; + ioat->regs->chanerr = val; + } + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + rte_delay_ms(1); + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + rte_delay_ms(1); + while (ioat->regs->chancmd & IOAT_CHANCMD_RESET) { + ioat->regs->chainaddr = 0; + rte_delay_ms(1); + if (++retry >= 200) { + IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=%#"PRIx8 + ", CHANSTS=%#"PRIx64", CHANERR=%#"PRIx32"\n", + __func__, + ioat->regs->chancmd, + ioat->regs->chansts, + ioat->regs->chanerr); + rte_dma_pmd_release(name); + return -EIO; + } + } + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + + dmadev->fp_obj->dev_private = ioat; + + dmadev->state = RTE_DMA_DEV_READY; + + return 0; + +} + +/* Destroy a DMA device. */ +static int +ioat_dmadev_destroy(const char *name) +{ + int ret; + + if (!name) { + IOAT_PMD_ERR("Invalid device name"); + return -EINVAL; + } + + ret = rte_dma_pmd_release(name); + if (ret) + IOAT_PMD_DEBUG("Device cleanup failed"); + + return 0; +} + /* Probe DMA device. */ static int ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) @@ -24,7 +125,7 @@ ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); dev->device.driver = &drv->driver; - return 0; + return ioat_dmadev_create(name, dev); } /* Remove DMA device. */ @@ -38,7 +139,7 @@ ioat_dmadev_remove(struct rte_pci_device *dev) IOAT_PMD_INFO("Closing %s on NUMA node %d", name, dev->device.numa_node); - return 0; + return ioat_dmadev_destroy(name); } static const struct rte_pci_id pci_id_ioat_map[] = { diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index eeabba41ef..73bdf548b3 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -11,6 +11,8 @@ extern "C" { #include <stdint.h> +#define IOAT_PCI_CHANERR_INT_OFFSET 0x180 + #define IOAT_VER_3_0 0x30 #define IOAT_VER_3_3 0x33 @@ -28,6 +30,49 @@ extern "C" { #define IOAT_DEVICE_ID_BDXF 0x6f2F #define IOAT_DEVICE_ID_ICX 0x0b00 +#define IOAT_COMP_UPDATE_SHIFT 3 +#define IOAT_CMD_OP_SHIFT 24 + +/* DMA Channel Registers */ +#define IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK 0xF000 +#define IOAT_CHANCTRL_COMPL_DCA_EN 0x0200 +#define IOAT_CHANCTRL_CHANNEL_IN_USE 0x0100 +#define IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL 0x0020 +#define IOAT_CHANCTRL_ERR_INT_EN 0x0010 +#define IOAT_CHANCTRL_ANY_ERR_ABORT_EN 0x0008 +#define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 +#define IOAT_CHANCTRL_INT_REARM 0x0001 + +struct ioat_registers { + uint8_t chancnt; + uint8_t xfercap; + uint8_t genctrl; + uint8_t intrctrl; + uint32_t attnstatus; + uint8_t cbver; /* 0x08 */ + uint8_t reserved4[0x3]; /* 0x09 */ + uint16_t intrdelay; /* 0x0C */ + uint16_t cs_status; /* 0x0E */ + uint32_t dmacapability; /* 0x10 */ + uint8_t reserved5[0x6C]; /* 0x14 */ + uint16_t chanctrl; /* 0x80 */ + uint8_t reserved6[0x2]; /* 0x82 */ + uint8_t chancmd; /* 0x84 */ + uint8_t reserved3[1]; /* 0x85 */ + uint16_t dmacount; /* 0x86 */ + uint64_t chansts; /* 0x88 */ + uint64_t chainaddr; /* 0x90 */ + uint64_t chancmp; /* 0x98 */ + uint8_t reserved2[0x8]; /* 0xA0 */ + uint32_t chanerr; /* 0xA8 */ + uint32_t chanerrmask; /* 0xAC */ +} __rte_packed; + +#define IOAT_CHANCMD_RESET 0x20 +#define IOAT_CHANCMD_SUSPEND 0x04 + +#define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ + #ifdef __cplusplus } #endif diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h index f1ec12a919..4fa19eb811 100644 --- a/drivers/dma/ioat/ioat_internal.h +++ b/drivers/dma/ioat/ioat_internal.h @@ -7,6 +7,33 @@ #include "ioat_hw_defs.h" +struct ioat_dmadev { + struct rte_dma_dev *dmadev; + struct rte_dma_vchan_conf qcfg; + struct rte_dma_stats stats; + + volatile uint16_t *doorbell __rte_cache_aligned; + phys_addr_t status_addr; + phys_addr_t ring_addr; + + struct ioat_dma_hw_desc *desc_ring; + + unsigned short next_read; + unsigned short next_write; + unsigned short last_write; /* Used to compute submitted count. */ + unsigned short offset; /* Used after a device recovery when counts -> 0. */ + unsigned int failure; /* Used to store chanerr for error handling. */ + + /* To report completions, the device will write status back here. */ + volatile uint64_t status __rte_cache_aligned; + + /* Pointer to the register bar. */ + volatile struct ioat_registers *regs; + + /* Store the IOAT version. */ + uint8_t version; +}; + extern int ioat_pmd_logtype; #define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v7 03/12] dma/ioat: add datapath structures 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 01/12] dma/ioat: add device probe and removal functions Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 02/12] dma/ioat: create dmadev instances on PCI probe Conor Walsh @ 2021-10-14 9:48 ` Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 04/12] dma/ioat: add configuration functions Conor Walsh ` (8 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-14 9:48 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add data structures required for the data path of IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 70 ++++++++++- drivers/dma/ioat/ioat_hw_defs.h | 215 ++++++++++++++++++++++++++++++++ 2 files changed, 284 insertions(+), 1 deletion(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 90f54567a4..876e17f320 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -15,11 +15,79 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Dump DMA device info. */ +static int +__dev_dump(void *dev_private, FILE *f) +{ + struct ioat_dmadev *ioat = dev_private; + uint64_t chansts_masked = ioat->regs->chansts & IOAT_CHANSTS_STATUS; + uint32_t chanerr = ioat->regs->chanerr; + uint64_t mask = (ioat->qcfg.nb_desc - 1); + char ver = ioat->version; + fprintf(f, "========= IOAT =========\n"); + fprintf(f, " IOAT version: %d.%d\n", ver >> 4, ver & 0xF); + fprintf(f, " Channel status: %s [0x%"PRIx64"]\n", + chansts_readable[chansts_masked], chansts_masked); + fprintf(f, " ChainADDR: 0x%"PRIu64"\n", ioat->regs->chainaddr); + if (chanerr == 0) { + fprintf(f, " No Channel Errors\n"); + } else { + fprintf(f, " ChanERR: 0x%"PRIu32"\n", chanerr); + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + fprintf(f, " Invalid Source Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + fprintf(f, " Invalid Destination Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + fprintf(f, " Invalid Descriptor Length\n"); + if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + fprintf(f, " Descriptor Read Error\n"); + if ((chanerr & ~(IOAT_CHANERR_INVALID_SRC_ADDR_MASK | + IOAT_CHANERR_INVALID_DST_ADDR_MASK | + IOAT_CHANERR_INVALID_LENGTH_MASK | + IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK)) != 0) + fprintf(f, " Unknown Error(s)\n"); + } + fprintf(f, "== Private Data ==\n"); + fprintf(f, " Config: { ring_size: %u }\n", ioat->qcfg.nb_desc); + fprintf(f, " Status: 0x%"PRIx64"\n", ioat->status); + fprintf(f, " Status IOVA: 0x%"PRIx64"\n", ioat->status_addr); + fprintf(f, " Status ADDR: %p\n", &ioat->status); + fprintf(f, " Ring IOVA: 0x%"PRIx64"\n", ioat->ring_addr); + fprintf(f, " Ring ADDR: 0x%"PRIx64"\n", ioat->desc_ring[0].next-64); + fprintf(f, " Next write: %"PRIu16"\n", ioat->next_write); + fprintf(f, " Next read: %"PRIu16"\n", ioat->next_read); + struct ioat_dma_hw_desc *desc_ring = &ioat->desc_ring[(ioat->next_write - 1) & mask]; + fprintf(f, " Last Descriptor Written {\n"); + fprintf(f, " Size: %"PRIu32"\n", desc_ring->size); + fprintf(f, " Control: 0x%"PRIx32"\n", desc_ring->u.control_raw); + fprintf(f, " Src: 0x%"PRIx64"\n", desc_ring->src_addr); + fprintf(f, " Dest: 0x%"PRIx64"\n", desc_ring->dest_addr); + fprintf(f, " Next: 0x%"PRIx64"\n", desc_ring->next); + fprintf(f, " }\n"); + fprintf(f, " Next Descriptor {\n"); + fprintf(f, " Size: %"PRIu32"\n", ioat->desc_ring[ioat->next_read & mask].size); + fprintf(f, " Src: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].src_addr); + fprintf(f, " Dest: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); + fprintf(f, " Next: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].next); + fprintf(f, " }\n"); + + return 0; +} + +/* Public wrapper for dump. */ +static int +ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) +{ + return __dev_dump(dev->fp_obj->dev_private, f); +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { - static const struct rte_dma_dev_ops ioat_dmadev_ops = { }; + static const struct rte_dma_dev_ops ioat_dmadev_ops = { + .dev_dump = ioat_dev_dump, + }; struct rte_dma_dev *dmadev = NULL; struct ioat_dmadev *ioat = NULL; diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index 73bdf548b3..dc3493a78f 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -15,6 +15,7 @@ extern "C" { #define IOAT_VER_3_0 0x30 #define IOAT_VER_3_3 0x33 +#define IOAT_VER_3_4 0x34 #define IOAT_VENDOR_ID 0x8086 #define IOAT_DEVICE_ID_SKX 0x2021 @@ -43,6 +44,14 @@ extern "C" { #define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 #define IOAT_CHANCTRL_INT_REARM 0x0001 +/* DMA Channel Capabilities */ +#define IOAT_DMACAP_PB (1 << 0) +#define IOAT_DMACAP_DCA (1 << 4) +#define IOAT_DMACAP_BFILL (1 << 6) +#define IOAT_DMACAP_XOR (1 << 8) +#define IOAT_DMACAP_PQ (1 << 9) +#define IOAT_DMACAP_DMA_DIF (1 << 10) + struct ioat_registers { uint8_t chancnt; uint8_t xfercap; @@ -71,8 +80,214 @@ struct ioat_registers { #define IOAT_CHANCMD_RESET 0x20 #define IOAT_CHANCMD_SUSPEND 0x04 +#define IOAT_CHANSTS_STATUS 0x7ULL +#define IOAT_CHANSTS_ACTIVE 0x0 +#define IOAT_CHANSTS_IDLE 0x1 +#define IOAT_CHANSTS_SUSPENDED 0x2 +#define IOAT_CHANSTS_HALTED 0x3 +#define IOAT_CHANSTS_ARMED 0x4 + +#define IOAT_CHANERR_INVALID_SRC_ADDR_MASK (1 << 0) +#define IOAT_CHANERR_INVALID_DST_ADDR_MASK (1 << 1) +#define IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK (1 << 8) +#define IOAT_CHANERR_INVALID_LENGTH_MASK (1 << 10) + +const char *chansts_readable[] = { + "ACTIVE", /* 0x0 */ + "IDLE", /* 0x1 */ + "SUSPENDED", /* 0x2 */ + "HALTED", /* 0x3 */ + "ARMED" /* 0x4 */ +}; + +#define IOAT_CHANSTS_UNAFFILIATED_ERROR 0x8ULL +#define IOAT_CHANSTS_SOFT_ERROR 0x10ULL + +#define IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK (~0x3FULL) + #define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ +struct ioat_dma_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t null: 1; + uint32_t src_page_break: 1; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_COPY 0x00 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t reserved2; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_fill_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t reserved: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t reserved2: 2; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t reserved3: 15; +#define IOAT_OP_FILL 0x01 + uint32_t op: 8; + } control; + } u; + uint64_t src_data; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t next_dest_addr; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_xor_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_XOR 0x87 +#define IOAT_OP_XOR_VAL 0x88 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint64_t src_addr4; + uint64_t src_addr5; +}; + +struct ioat_xor_ext_hw_desc { + uint64_t src_addr6; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t next; + uint64_t reserved[4]; +}; + +struct ioat_pq_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 11; +#define IOAT_OP_PQ 0x89 +#define IOAT_OP_PQ_VAL 0x8a + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint8_t coef[8]; + uint64_t q_addr; +}; + +struct ioat_pq_ext_hw_desc { + uint64_t src_addr4; + uint64_t src_addr5; + uint64_t src_addr6; + uint64_t next; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t reserved[2]; +}; + +struct ioat_pq_update_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_cnt: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 3; + uint32_t coef: 8; +#define IOAT_OP_PQ_UP 0x8b + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t p_src; + uint64_t q_src; + uint64_t q_addr; +}; + +union ioat_hw_desc { + struct ioat_dma_hw_desc dma; + struct ioat_fill_hw_desc fill; + struct ioat_xor_hw_desc xor_desc; + struct ioat_xor_ext_hw_desc xor_ext; + struct ioat_pq_hw_desc pq; + struct ioat_pq_ext_hw_desc pq_ext; + struct ioat_pq_update_hw_desc pq_update; +}; + +#define GENSTS_DEV_STATE_MASK 0x03 +#define CMDSTATUS_ACTIVE_SHIFT 31 +#define CMDSTATUS_ACTIVE_MASK (1 << 31) +#define CMDSTATUS_ERR_MASK 0xFF + #ifdef __cplusplus } #endif -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v7 04/12] dma/ioat: add configuration functions 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (2 preceding siblings ...) 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 03/12] dma/ioat: add datapath structures Conor Walsh @ 2021-10-14 9:48 ` Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 05/12] dma/ioat: add start and stop functions Conor Walsh ` (7 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-14 9:48 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add functions for device configuration. The info_get and close functions are included here also. info_get can be useful for checking successful configuration and close is used by the dmadev api when releasing a configured device. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 15 +++++ drivers/dma/ioat/ioat_dmadev.c | 107 +++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 9ae1d8a2ad..af69556241 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -67,3 +67,18 @@ For example:: Once probed successfully, the device will appear as a ``dmadev``, that is a "DMA device type" inside DPDK, and can be accessed using APIs from the ``rte_dmadev`` library. + +Using IOAT DMAdev Devices +-------------------------- + +To use IOAT devices from an application, the ``dmadev`` API can be used. + +Device Configuration +~~~~~~~~~~~~~~~~~~~~~ + +IOAT configuration requirements: + +* ``ring_size`` must be a power of two, between 64 and 4096. +* Only one ``vchan`` is supported per device. +* Silent mode is not supported. +* The transfer direction must be set to ``RTE_DMA_DIR_MEM_TO_MEM`` to copy from memory to memory. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 876e17f320..ada57c5814 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -12,9 +12,112 @@ static struct rte_pci_driver ioat_pmd_drv; RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); +#define DESC_SZ sizeof(struct ioat_dma_hw_desc) + #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Configure a device. */ +static int +ioat_dev_configure(struct rte_dma_dev *dev __rte_unused, const struct rte_dma_conf *dev_conf, + uint32_t conf_sz) +{ + if (sizeof(struct rte_dma_conf) != conf_sz) + return -EINVAL; + + if (dev_conf->nb_vchans != 1) + return -EINVAL; + + return 0; +} + +/* Setup a virtual channel for IOAT, only 1 vchan is supported. */ +static int +ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + const struct rte_dma_vchan_conf *qconf, uint32_t qconf_sz) +{ + struct ioat_dmadev *ioat = dev->fp_obj->dev_private; + uint16_t max_desc = qconf->nb_desc; + int i; + + if (sizeof(struct rte_dma_vchan_conf) != qconf_sz) + return -EINVAL; + + ioat->qcfg = *qconf; + + if (!rte_is_power_of_2(max_desc)) { + max_desc = rte_align32pow2(max_desc); + IOAT_PMD_DEBUG("DMA dev %u using %u descriptors", dev->data->dev_id, max_desc); + ioat->qcfg.nb_desc = max_desc; + } + + /* In case we are reconfiguring a device, free any existing memory. */ + rte_free(ioat->desc_ring); + + ioat->desc_ring = rte_zmalloc(NULL, sizeof(*ioat->desc_ring) * max_desc, 0); + if (ioat->desc_ring == NULL) + return -ENOMEM; + + ioat->ring_addr = rte_mem_virt2iova(ioat->desc_ring); + + ioat->status_addr = rte_mem_virt2iova(ioat) + offsetof(struct ioat_dmadev, status); + + /* Ensure all counters are reset, if reconfiguring/restarting device. */ + ioat->next_read = 0; + ioat->next_write = 0; + ioat->last_write = 0; + ioat->offset = 0; + ioat->failure = 0; + + /* Configure descriptor ring - each one points to next. */ + for (i = 0; i < ioat->qcfg.nb_desc; i++) { + ioat->desc_ring[i].next = ioat->ring_addr + + (((i + 1) % ioat->qcfg.nb_desc) * DESC_SZ); + } + + return 0; +} + +/* Get device information of a device. */ +static int +ioat_dev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *info, uint32_t size) +{ + struct ioat_dmadev *ioat = dev->fp_obj->dev_private; + if (size < sizeof(*info)) + return -EINVAL; + info->dev_capa = RTE_DMA_CAPA_MEM_TO_MEM | + RTE_DMA_CAPA_OPS_COPY | + RTE_DMA_CAPA_OPS_FILL; + if (ioat->version >= IOAT_VER_3_4) + info->dev_capa |= RTE_DMA_CAPA_HANDLES_ERRORS; + info->max_vchans = 1; + info->min_desc = 32; + info->max_desc = 4096; + return 0; +} + +/* Close a configured device. */ +static int +ioat_dev_close(struct rte_dma_dev *dev) +{ + struct ioat_dmadev *ioat; + + if (!dev) { + IOAT_PMD_ERR("Invalid device"); + return -EINVAL; + } + + ioat = dev->fp_obj->dev_private; + if (!ioat) { + IOAT_PMD_ERR("Error getting dev_private"); + return -EINVAL; + } + + rte_free(ioat->desc_ring); + + return 0; +} + /* Dump DMA device info. */ static int __dev_dump(void *dev_private, FILE *f) @@ -86,7 +189,11 @@ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { static const struct rte_dma_dev_ops ioat_dmadev_ops = { + .dev_close = ioat_dev_close, + .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, + .dev_info_get = ioat_dev_info_get, + .vchan_setup = ioat_vchan_setup, }; struct rte_dma_dev *dmadev = NULL; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v7 05/12] dma/ioat: add start and stop functions 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (3 preceding siblings ...) 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 04/12] dma/ioat: add configuration functions Conor Walsh @ 2021-10-14 9:48 ` Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 06/12] dma/ioat: add data path job submission functions Conor Walsh ` (6 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-14 9:48 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add start, stop and recover functions for IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 3 ++ drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index af69556241..df159f9957 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -82,3 +82,6 @@ IOAT configuration requirements: * Only one ``vchan`` is supported per device. * Silent mode is not supported. * The transfer direction must be set to ``RTE_DMA_DIR_MEM_TO_MEM`` to copy from memory to memory. + +Once configured, the device can then be made ready for use by calling the +``rte_dma_start()`` API. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index ada57c5814..cf28f4a7e6 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -78,6 +78,96 @@ ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, return 0; } +/* Recover IOAT device. */ +static inline int +__ioat_recover(struct ioat_dmadev *ioat) +{ + uint32_t chanerr, retry = 0; + uint16_t mask = ioat->qcfg.nb_desc - 1; + + /* Clear any channel errors. Reading and writing to chanerr does this. */ + chanerr = ioat->regs->chanerr; + ioat->regs->chanerr = chanerr; + + /* Reset Channel. */ + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + + /* Write new chain address to trigger state change. */ + ioat->regs->chainaddr = ioat->desc_ring[(ioat->next_read - 1) & mask].next; + /* Ensure channel control and status addr are correct. */ + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + ioat->regs->chancmp = ioat->status_addr; + + /* Allow HW time to move to the ARMED state. */ + do { + rte_pause(); + retry++; + } while (ioat->regs->chansts != IOAT_CHANSTS_ARMED && retry < 200); + + /* Exit as failure if device is still HALTED. */ + if (ioat->regs->chansts != IOAT_CHANSTS_ARMED) + return -1; + + /* Store next write as offset as recover will move HW and SW ring out of sync. */ + ioat->offset = ioat->next_read; + + /* Prime status register with previous address. */ + ioat->status = ioat->desc_ring[(ioat->next_read - 2) & mask].next; + + return 0; +} + +/* Start a configured device. */ +static int +ioat_dev_start(struct rte_dma_dev *dev) +{ + struct ioat_dmadev *ioat = dev->fp_obj->dev_private; + + if (ioat->qcfg.nb_desc == 0 || ioat->desc_ring == NULL) + return -EBUSY; + + /* Inform hardware of where the descriptor ring is. */ + ioat->regs->chainaddr = ioat->ring_addr; + /* Inform hardware of where to write the status/completions. */ + ioat->regs->chancmp = ioat->status_addr; + + /* Prime the status register to be set to the last element. */ + ioat->status = ioat->ring_addr + ((ioat->qcfg.nb_desc - 1) * DESC_SZ); + + printf("IOAT.status: %s [0x%"PRIx64"]\n", + chansts_readable[ioat->status & IOAT_CHANSTS_STATUS], + ioat->status); + + if ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_HALTED) { + IOAT_PMD_WARN("Device HALTED on start, attempting to recover\n"); + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device couldn't be recovered"); + return -1; + } + } + + return 0; +} + +/* Stop a configured device. */ +static int +ioat_dev_stop(struct rte_dma_dev *dev) +{ + struct ioat_dmadev *ioat = dev->fp_obj->dev_private; + uint32_t retry = 0; + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + + do { + rte_pause(); + retry++; + } while ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) != IOAT_CHANSTS_SUSPENDED + && retry < 200); + + return ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_SUSPENDED) ? 0 : -1; +} + /* Get device information of a device. */ static int ioat_dev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *info, uint32_t size) @@ -193,6 +283,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, .dev_info_get = ioat_dev_info_get, + .dev_start = ioat_dev_start, + .dev_stop = ioat_dev_stop, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v7 06/12] dma/ioat: add data path job submission functions 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (4 preceding siblings ...) 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 05/12] dma/ioat: add start and stop functions Conor Walsh @ 2021-10-14 9:48 ` Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 07/12] dma/ioat: add data path completion functions Conor Walsh ` (5 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-14 9:48 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add data path functions for enqueuing and submitting operations to IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Reviewed-by: Chengwen Feng <fengchengwen@huawei.com> --- doc/guides/dmadevs/ioat.rst | 9 ++++ drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index df159f9957..9ee4e372a8 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -85,3 +85,12 @@ IOAT configuration requirements: Once configured, the device can then be made ready for use by calling the ``rte_dma_start()`` API. + +Performing Data Copies +~~~~~~~~~~~~~~~~~~~~~~~ + +Refer to the :ref:`Enqueue / Dequeue APIs <dmadev_enqueue_dequeue>` section of the dmadev library +documentation for details on operation enqueue and submission API usage. + +It is expected that, for efficiency reasons, a burst of operations will be enqueued to the +device via multiple enqueue calls between calls to the ``rte_dma_submit()`` function. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index cf28f4a7e6..4d00fec5c8 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -5,6 +5,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> #include <rte_malloc.h> +#include <rte_prefetch.h> #include "ioat_internal.h" @@ -17,6 +18,12 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* IOAT operations. */ +enum rte_ioat_ops { + ioat_op_copy = 0, /* Standard DMA Operation */ + ioat_op_fill /* Block Fill */ +}; + /* Configure a device. */ static int ioat_dev_configure(struct rte_dma_dev *dev __rte_unused, const struct rte_dma_conf *dev_conf, @@ -208,6 +215,87 @@ ioat_dev_close(struct rte_dma_dev *dev) return 0; } +/* Trigger hardware to begin performing enqueued operations. */ +static inline void +__submit(struct ioat_dmadev *ioat) +{ + *ioat->doorbell = ioat->next_write - ioat->offset; + + ioat->last_write = ioat->next_write; +} + +/* External submit function wrapper. */ +static int +ioat_submit(void *dev_private, uint16_t qid __rte_unused) +{ + struct ioat_dmadev *ioat = dev_private; + + __submit(ioat); + + return 0; +} + +/* Write descriptor for enqueue. */ +static inline int +__write_desc(void *dev_private, uint32_t op, uint64_t src, phys_addr_t dst, + unsigned int length, uint64_t flags) +{ + struct ioat_dmadev *ioat = dev_private; + uint16_t ret; + const unsigned short mask = ioat->qcfg.nb_desc - 1; + const unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + const unsigned short space = mask + read - write; + struct ioat_dma_hw_desc *desc; + + if (space == 0) + return -ENOSPC; + + ioat->next_write = write + 1; + write &= mask; + + desc = &ioat->desc_ring[write]; + desc->size = length; + desc->u.control_raw = (uint32_t)((op << IOAT_CMD_OP_SHIFT) | + (1 << IOAT_COMP_UPDATE_SHIFT)); + + /* In IOAT the fence ensures that all operations including the current one + * are completed before moving on, DMAdev assumes that the fence ensures + * all operations before the current one are completed before starting + * the current one, so in IOAT we set the fence for the previous descriptor. + */ + if (flags & RTE_DMA_OP_FLAG_FENCE) + ioat->desc_ring[(write - 1) & mask].u.control.fence = 1; + + desc->src_addr = src; + desc->dest_addr = dst; + + rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]); + + ret = (uint16_t)(ioat->next_write - 1); + + if (flags & RTE_DMA_OP_FLAG_SUBMIT) + __submit(ioat); + + return ret; +} + +/* Enqueue a fill operation onto the ioat device. */ +static int +ioat_enqueue_fill(void *dev_private, uint16_t qid __rte_unused, uint64_t pattern, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev_private, ioat_op_fill, pattern, dst, length, flags); +} + +/* Enqueue a copy operation onto the ioat device. */ +static int +ioat_enqueue_copy(void *dev_private, uint16_t qid __rte_unused, rte_iova_t src, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev_private, ioat_op_copy, src, dst, length, flags); +} + /* Dump DMA device info. */ static int __dev_dump(void *dev_private, FILE *f) @@ -310,6 +398,10 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->fp_obj->copy = ioat_enqueue_copy; + dmadev->fp_obj->fill = ioat_enqueue_fill; + dmadev->fp_obj->submit = ioat_submit; + ioat = dmadev->data->dev_private; ioat->dmadev = dmadev; ioat->regs = dev->mem_resource[0].addr; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v7 07/12] dma/ioat: add data path completion functions 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (5 preceding siblings ...) 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 06/12] dma/ioat: add data path job submission functions Conor Walsh @ 2021-10-14 9:48 ` Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 08/12] dma/ioat: add statistics Conor Walsh ` (4 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-14 9:48 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add the data path functions for gathering completed operations from IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- doc/guides/dmadevs/ioat.rst | 33 +++++++- drivers/dma/ioat/ioat_dmadev.c | 141 +++++++++++++++++++++++++++++++++ 2 files changed, 173 insertions(+), 1 deletion(-) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 9ee4e372a8..9ac90e3108 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -90,7 +90,38 @@ Performing Data Copies ~~~~~~~~~~~~~~~~~~~~~~~ Refer to the :ref:`Enqueue / Dequeue APIs <dmadev_enqueue_dequeue>` section of the dmadev library -documentation for details on operation enqueue and submission API usage. +documentation for details on operation enqueue, submission and completion API usage. It is expected that, for efficiency reasons, a burst of operations will be enqueued to the device via multiple enqueue calls between calls to the ``rte_dma_submit()`` function. + +When gathering completions, ``rte_dma_completed()`` should be used, up until the point an error +occurs with an operation. If an error was encountered, ``rte_dma_completed_status()`` must be used +to reset the device and continue processing operations. This function will also gather the status +of each individual operation which is filled in to the ``status`` array provided as parameter +by the application. + +The status codes supported by IOAT are: + +* ``RTE_DMA_STATUS_SUCCESSFUL``: The operation was successful. +* ``RTE_DMA_STATUS_INVALID_SRC_ADDR``: The operation failed due to an invalid source address. +* ``RTE_DMA_STATUS_INVALID_DST_ADDR``: The operation failed due to an invalid destination address. +* ``RTE_DMA_STATUS_INVALID_LENGTH``: The operation failed due to an invalid descriptor length. +* ``RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR``: The device could not read the descriptor. +* ``RTE_DMA_STATUS_ERROR_UNKNOWN``: The operation failed due to an unspecified error. + +The following code shows how to retrieve the number of successfully completed +copies within a burst and then uses ``rte_dma_completed_status()`` to check +which operation failed and reset the device to continue processing operations: + +.. code-block:: C + + enum rte_dma_status_code status[COMP_BURST_SZ]; + uint16_t count, idx, status_count; + bool error = 0; + + count = rte_dma_completed(dev_id, vchan, COMP_BURST_SZ, &idx, &error); + + if (error){ + status_count = rte_dma_completed_status(dev_id, vchan, COMP_BURST_SZ, &idx, status); + } diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 4d00fec5c8..0318f67772 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -6,6 +6,7 @@ #include <rte_dmadev_pmd.h> #include <rte_malloc.h> #include <rte_prefetch.h> +#include <rte_errno.h> #include "ioat_internal.h" @@ -362,6 +363,144 @@ ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) return __dev_dump(dev->fp_obj->dev_private, f); } +/* Returns the index of the last completed operation. */ +static inline uint16_t +__get_last_completed(const struct ioat_dmadev *ioat, int *state) +{ + /* Status register contains the address of the completed operation */ + uint64_t status = ioat->status; + + /* lower 3 bits indicate "transfer status" : active, idle, halted. + * We can ignore bit 0. + */ + *state = status & IOAT_CHANSTS_STATUS; + + /* If we are just after recovering from an error the address returned by + * status will be 0, in this case we return the offset - 1 as the last + * completed. If not return the status value minus the chainaddr which + * gives us an offset into the ring. Right shifting by 6 (divide by 64) + * gives the index of the completion from the HW point of view and adding + * the offset translates the ring index from HW to SW point of view. + */ + if ((status & ~IOAT_CHANSTS_STATUS) == 0) + return ioat->offset - 1; + + return (status - ioat->ring_addr) >> 6; +} + +/* Translates IOAT ChanERRs to DMA error codes. */ +static inline enum rte_dma_status_code +__translate_status_ioat_to_dma(uint32_t chanerr) +{ + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_SRC_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_DST_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + return RTE_DMA_STATUS_INVALID_LENGTH; + else if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + return RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR; + else + return RTE_DMA_STATUS_ERROR_UNKNOWN; +} + +/* Returns details of operations that have been completed. */ +static uint16_t +ioat_completed(void *dev_private, uint16_t qid __rte_unused, const uint16_t max_ops, + uint16_t *last_idx, bool *has_error) +{ + struct ioat_dmadev *ioat = dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short last_completed, count; + int state, fails = 0; + + /* Do not do any work if there is an uncleared error. */ + if (ioat->failure != 0) { + *has_error = true; + *last_idx = ioat->next_read - 2; + return 0; + } + + last_completed = __get_last_completed(ioat, &state); + count = (last_completed + 1 - read) & mask; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || state != IOAT_CHANSTS_HALTED) { + ioat->next_read = read + count; + *last_idx = ioat->next_read - 1; + } else { + *has_error = true; + rte_errno = EIO; + ioat->failure = ioat->regs->chanerr; + ioat->next_read = read + count + 1; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + __dev_dump(dev_private, stdout); + return 0; + } + __submit(ioat); + fails++; + *last_idx = ioat->next_read - 2; + } + + return count; +} + +/* Returns detailed status information about operations that have been completed. */ +static uint16_t +ioat_completed_status(void *dev_private, uint16_t qid __rte_unused, + uint16_t max_ops, uint16_t *last_idx, enum rte_dma_status_code *status) +{ + struct ioat_dmadev *ioat = dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short count, last_completed; + uint64_t fails = 0; + int state, i; + + last_completed = __get_last_completed(ioat, &state); + count = (last_completed + 1 - read) & mask; + + for (i = 0; i < RTE_MIN(count + 1, max_ops); i++) + status[i] = RTE_DMA_STATUS_SUCCESSFUL; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || state != IOAT_CHANSTS_HALTED) + ioat->next_read = read + count; + else { + rte_errno = EIO; + status[count] = __translate_status_ioat_to_dma(ioat->regs->chanerr); + count++; + ioat->next_read = read + count; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + __dev_dump(dev_private, stdout); + return 0; + } + __submit(ioat); + fails++; + } + + if (ioat->failure > 0) { + status[0] = __translate_status_ioat_to_dma(ioat->failure); + count = RTE_MIN(count + 1, max_ops); + ioat->failure = 0; + } + + *last_idx = ioat->next_read - 1; + + return count; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -398,6 +537,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->fp_obj->completed = ioat_completed; + dmadev->fp_obj->completed_status = ioat_completed_status; dmadev->fp_obj->copy = ioat_enqueue_copy; dmadev->fp_obj->fill = ioat_enqueue_fill; dmadev->fp_obj->submit = ioat_submit; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v7 08/12] dma/ioat: add statistics 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (6 preceding siblings ...) 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 07/12] dma/ioat: add data path completion functions Conor Walsh @ 2021-10-14 9:48 ` Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 09/12] dma/ioat: add support for vchan status function Conor Walsh ` (3 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-14 9:48 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add statistic tracking for operations in IOAT. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 43 ++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 0318f67772..b731361f9a 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -77,6 +77,9 @@ ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, ioat->offset = 0; ioat->failure = 0; + /* Reset Stats. */ + ioat->stats = (struct rte_dma_stats){0}; + /* Configure descriptor ring - each one points to next. */ for (i = 0; i < ioat->qcfg.nb_desc; i++) { ioat->desc_ring[i].next = ioat->ring_addr + @@ -222,6 +225,8 @@ __submit(struct ioat_dmadev *ioat) { *ioat->doorbell = ioat->next_write - ioat->offset; + ioat->stats.submitted += (uint16_t)(ioat->next_write - ioat->last_write); + ioat->last_write = ioat->next_write; } @@ -352,6 +357,10 @@ __dev_dump(void *dev_private, FILE *f) fprintf(f, " Dest: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); fprintf(f, " Next: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].next); fprintf(f, " }\n"); + fprintf(f, " Key Stats { submitted: %"PRIu64", comp: %"PRIu64", failed: %"PRIu64" }\n", + ioat->stats.submitted, + ioat->stats.completed, + ioat->stats.errors); return 0; } @@ -448,6 +457,9 @@ ioat_completed(void *dev_private, uint16_t qid __rte_unused, const uint16_t max_ *last_idx = ioat->next_read - 2; } + ioat->stats.completed += count; + ioat->stats.errors += fails; + return count; } @@ -498,9 +510,38 @@ ioat_completed_status(void *dev_private, uint16_t qid __rte_unused, *last_idx = ioat->next_read - 1; + ioat->stats.completed += count; + ioat->stats.errors += fails; + return count; } +/* Retrieve the generic stats of a DMA device. */ +static int +ioat_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + struct rte_dma_stats *rte_stats, uint32_t size) +{ + struct rte_dma_stats *stats = (&((struct ioat_dmadev *)dev->dev_private)->stats); + + if (size < sizeof(rte_stats)) + return -EINVAL; + if (rte_stats == NULL) + return -EINVAL; + + *rte_stats = *stats; + return 0; +} + +/* Reset the generic stat counters for the DMA device. */ +static int +ioat_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused) +{ + struct ioat_dmadev *ioat = dev->dev_private; + + ioat->stats = (struct rte_dma_stats){0}; + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -512,6 +553,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_info_get = ioat_dev_info_get, .dev_start = ioat_dev_start, .dev_stop = ioat_dev_stop, + .stats_get = ioat_stats_get, + .stats_reset = ioat_stats_reset, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v7 09/12] dma/ioat: add support for vchan status function 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (7 preceding siblings ...) 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 08/12] dma/ioat: add statistics Conor Walsh @ 2021-10-14 9:48 ` Conor Walsh 2021-10-14 9:49 ` [dpdk-dev] [PATCH v7 10/12] dma/ioat: add burst capacity function Conor Walsh ` (2 subsequent siblings) 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-14 9:48 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Add support for the rte_dmadev_vchan_status API call. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index b731361f9a..17ac3217c7 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -542,6 +542,26 @@ ioat_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused) return 0; } +/* Check if the IOAT device is idle. */ +static int +ioat_vchan_status(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + enum rte_dma_vchan_status *status) +{ + int state = 0; + const struct ioat_dmadev *ioat = dev->dev_private; + const uint16_t mask = ioat->qcfg.nb_desc - 1; + const uint16_t last = __get_last_completed(ioat, &state); + + if (state == IOAT_CHANSTS_HALTED || state == IOAT_CHANSTS_SUSPENDED) + *status = RTE_DMA_VCHAN_HALTED_ERROR; + else if (last == ((ioat->next_write - 1) & mask)) + *status = RTE_DMA_VCHAN_IDLE; + else + *status = RTE_DMA_VCHAN_ACTIVE; + + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -555,6 +575,7 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_stop = ioat_dev_stop, .stats_get = ioat_stats_get, .stats_reset = ioat_stats_reset, + .vchan_status = ioat_vchan_status, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v7 10/12] dma/ioat: add burst capacity function 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (8 preceding siblings ...) 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 09/12] dma/ioat: add support for vchan status function Conor Walsh @ 2021-10-14 9:49 ` Conor Walsh 2021-10-14 9:49 ` [dpdk-dev] [PATCH v7 11/12] devbind: move ioat device IDs to dmadev category Conor Walsh 2021-10-14 9:49 ` [dpdk-dev] [PATCH v7 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-14 9:49 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Adds the ability to find the remaining space in the IOAT ring. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 17ac3217c7..a230496b11 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -516,12 +516,25 @@ ioat_completed_status(void *dev_private, uint16_t qid __rte_unused, return count; } +/* Get the remaining capacity of the ring. */ +static uint16_t +ioat_burst_capacity(const void *dev_private, uint16_t vchan __rte_unused) +{ + const struct ioat_dmadev *ioat = dev_private; + unsigned short size = ioat->qcfg.nb_desc - 1; + unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + unsigned short space = size - (write - read); + + return space; +} + /* Retrieve the generic stats of a DMA device. */ static int ioat_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, struct rte_dma_stats *rte_stats, uint32_t size) { - struct rte_dma_stats *stats = (&((struct ioat_dmadev *)dev->dev_private)->stats); + struct rte_dma_stats *stats = (&((struct ioat_dmadev *)dev->fp_obj->dev_private)->stats); if (size < sizeof(rte_stats)) return -EINVAL; @@ -536,7 +549,7 @@ ioat_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, static int ioat_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused) { - struct ioat_dmadev *ioat = dev->dev_private; + struct ioat_dmadev *ioat = dev->fp_obj->dev_private; ioat->stats = (struct rte_dma_stats){0}; return 0; @@ -548,7 +561,7 @@ ioat_vchan_status(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, enum rte_dma_vchan_status *status) { int state = 0; - const struct ioat_dmadev *ioat = dev->dev_private; + const struct ioat_dmadev *ioat = dev->fp_obj->dev_private; const uint16_t mask = ioat->qcfg.nb_desc - 1; const uint16_t last = __get_last_completed(ioat, &state); @@ -601,6 +614,7 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->fp_obj->burst_capacity = ioat_burst_capacity; dmadev->fp_obj->completed = ioat_completed; dmadev->fp_obj->completed_status = ioat_completed_status; dmadev->fp_obj->copy = ioat_enqueue_copy; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v7 11/12] devbind: move ioat device IDs to dmadev category 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (9 preceding siblings ...) 2021-10-14 9:49 ` [dpdk-dev] [PATCH v7 10/12] dma/ioat: add burst capacity function Conor Walsh @ 2021-10-14 9:49 ` Conor Walsh 2021-10-14 9:49 ` [dpdk-dev] [PATCH v7 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh 11 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-14 9:49 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Move Intel IOAT devices from Misc to DMA devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Reviewed-by: Bruce Richardson <bruce.richardson@intel.com> --- usertools/dpdk-devbind.py | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index ba18e2a487..91f1b16bde 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -71,14 +71,13 @@ network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] -dma_devices = [intel_idxd_spr] +dma_devices = [intel_idxd_spr, intel_ioat_bdw, intel_ioat_icx, intel_ioat_skx] eventdev_devices = [cavium_sso, cavium_tim, intel_dlb, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] regex_devices = [octeontx2_ree] -misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, intel_ioat_bdw, - intel_ioat_skx, intel_ioat_icx, intel_ntb_skx, - intel_ntb_icx, octeontx2_dma] +misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, intel_ntb_skx, + intel_ntb_icx, octeontx2_dma] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v7 12/12] raw/ioat: deprecate ioat rawdev driver 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (10 preceding siblings ...) 2021-10-14 9:49 ` [dpdk-dev] [PATCH v7 11/12] devbind: move ioat device IDs to dmadev category Conor Walsh @ 2021-10-14 9:49 ` Conor Walsh 2021-10-18 7:54 ` Thomas Monjalon 11 siblings, 1 reply; 132+ messages in thread From: Conor Walsh @ 2021-10-14 9:49 UTC (permalink / raw) To: bruce.richardson, fengchengwen, jerinj, kevin.laatz; +Cc: dev, Conor Walsh Deprecate the rawdev IOAT driver as both IOAT and IDXD drivers have moved to dmadev. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Acked-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- MAINTAINERS | 2 +- doc/guides/rawdevs/ioat.rst | 4 ++++ doc/guides/rel_notes/deprecation.rst | 7 +++++++ 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/MAINTAINERS b/MAINTAINERS index 283c70f7d7..b9f7746dc4 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1322,7 +1322,7 @@ T: git://dpdk.org/next/dpdk-next-net-intel F: drivers/raw/ifpga/ F: doc/guides/rawdevs/ifpga.rst -IOAT Rawdev +IOAT Rawdev - DEPRECATED M: Bruce Richardson <bruce.richardson@intel.com> F: drivers/raw/ioat/ F: doc/guides/rawdevs/ioat.rst diff --git a/doc/guides/rawdevs/ioat.rst b/doc/guides/rawdevs/ioat.rst index a65530bd30..98d15dd032 100644 --- a/doc/guides/rawdevs/ioat.rst +++ b/doc/guides/rawdevs/ioat.rst @@ -6,6 +6,10 @@ IOAT Rawdev Driver =================== +.. warning:: + As of DPDK 21.11 the rawdev implementation of the IOAT driver has been deprecated. + Please use the dmadev library instead. + The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg| Data Streaming Accelerator `(Intel DSA) <https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator>`_ and for Intel\ |reg| diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst index a4e86b31f5..fcdd24c296 100644 --- a/doc/guides/rel_notes/deprecation.rst +++ b/doc/guides/rel_notes/deprecation.rst @@ -241,3 +241,10 @@ Deprecation Notices * cmdline: ``cmdline`` structure will be made opaque to hide platform-specific content. On Linux and FreeBSD, supported prior to DPDK 20.11, original structure will be kept until DPDK 21.11. + +* raw/ioat: The ``ioat`` rawdev driver has been deprecated, since it's + functionality is provided through the new ``dmadev`` infrastructure. + To continue to use hardware previously supported by the ``ioat`` rawdev driver, + applications should be updated to use the ``dmadev`` library instead, + with the underlying HW-functionality being provided by the ``ioat`` or + ``idxd`` dma drivers -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v7 12/12] raw/ioat: deprecate ioat rawdev driver 2021-10-14 9:49 ` [dpdk-dev] [PATCH v7 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh @ 2021-10-18 7:54 ` Thomas Monjalon 0 siblings, 0 replies; 132+ messages in thread From: Thomas Monjalon @ 2021-10-18 7:54 UTC (permalink / raw) To: bruce.richardson, kevin.laatz, Conor Walsh; +Cc: fengchengwen, jerinj, dev 14/10/2021 11:49, Conor Walsh: > Deprecate the rawdev IOAT driver as both IOAT and IDXD drivers have > moved to dmadev. > > Signed-off-by: Conor Walsh <conor.walsh@intel.com> > Acked-by: Kevin Laatz <kevin.laatz@intel.com> > Acked-by: Bruce Richardson <bruce.richardson@intel.com> > --- > --- a/doc/guides/rel_notes/deprecation.rst > +++ b/doc/guides/rel_notes/deprecation.rst > +* raw/ioat: The ``ioat`` rawdev driver has been deprecated, since it's > + functionality is provided through the new ``dmadev`` infrastructure. > + To continue to use hardware previously supported by the ``ioat`` rawdev driver, > + applications should be updated to use the ``dmadev`` library instead, > + with the underlying HW-functionality being provided by the ``ioat`` or > + ``idxd`` dma drivers Acked-by: Thomas Monjalon <thomas@monjalon.net> ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh ` (12 preceding siblings ...) 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh @ 2021-10-18 12:38 ` Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 01/12] dma/ioat: add device probe and removal functions Conor Walsh ` (13 more replies) 13 siblings, 14 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-18 12:38 UTC (permalink / raw) To: bruce.richardson, thomas, fengchengwen, jerinj, kevin.laatz Cc: dev, Conor Walsh This patchset adds a dmadev driver and associated documentation to support Intel QuickData Technology devices, part of the Intel I/O Acceleration Technology (Intel I/OAT). This driver is intended to ultimately replace the current IOAT part of the IOAT rawdev driver. This patchset passes all the driver tests added in the dmadev test suite. Depends-on: series-19737 ("add dmadev driver for idxd devices") --- v8: - Fixed compliation issue for individual patches. - Rebased on main and v8 of IDXD. v7: - Minor rework to update from v23 to v26 of the dmadev lib. v6: - Added rawdev IOAT deprecation notice to deprecation.rst. v5: - Updated to v23 of the dmadev lib. - Removed experimental tag for driver from MAINTAINERS. - Seperated IOAT and IDXD announcements in release notes. - Added missing check for rte_dma_get_dev_id in destroy. - Fixed memleak in destroy caused by NULL pointer. - Rewrote part of the docs to reduce duplication with DMA and IDXD. - Added patch to deprecate the rawdev IOAT driver. - Reworked destroy and close functions. - Added RTE_DMA_CAPA_HANDLES_ERRORS flag for IOAT versions >=3.4. - Other minor changes to IOAT driver. v4: - Changes needed to update from dmadev v21 to v22. - Fixed 32-bit build. - Made stats reset logic easier to understand. v3: - Added burst capacity function. - Stop function now waits for suspend rather than just using a sleep. - Changed from vchan idle to vchan status function. - Other minor changes to update from dmadev v19 to v21. v2: - Rebased on the above patchsets. Conor Walsh (12): dma/ioat: add device probe and removal functions dma/ioat: create dmadev instances on PCI probe dma/ioat: add datapath structures dma/ioat: add configuration functions dma/ioat: add start and stop functions dma/ioat: add data path job submission functions dma/ioat: add data path completion functions dma/ioat: add statistics dma/ioat: add support for vchan status function dma/ioat: add burst capacity function devbind: move ioat device IDs to dmadev category raw/ioat: deprecate ioat rawdev driver MAINTAINERS | 8 +- doc/guides/dmadevs/index.rst | 2 + doc/guides/dmadevs/ioat.rst | 127 +++++ doc/guides/rawdevs/ioat.rst | 4 + doc/guides/rel_notes/deprecation.rst | 7 + doc/guides/rel_notes/release_21_11.rst | 6 + drivers/dma/ioat/ioat_dmadev.c | 748 +++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 295 ++++++++++ drivers/dma/ioat/ioat_internal.h | 47 ++ drivers/dma/ioat/meson.build | 7 + drivers/dma/ioat/version.map | 3 + drivers/dma/meson.build | 1 + usertools/dpdk-devbind.py | 7 +- 13 files changed, 1257 insertions(+), 5 deletions(-) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v8 01/12] dma/ioat: add device probe and removal functions 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh @ 2021-10-18 12:38 ` Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 02/12] dma/ioat: create dmadev instances on PCI probe Conor Walsh ` (12 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-18 12:38 UTC (permalink / raw) To: bruce.richardson, thomas, fengchengwen, jerinj, kevin.laatz Cc: dev, Conor Walsh Add the basic device probe/remove skeleton code and initial documentation for new IOAT DMA driver. Maintainers update is also included in this patch. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Reviewed-by: Chengwen Feng <fengchengwen@huawei.com> --- MAINTAINERS | 6 +++ doc/guides/dmadevs/index.rst | 2 + doc/guides/dmadevs/ioat.rst | 69 ++++++++++++++++++++++++++ doc/guides/rel_notes/release_21_11.rst | 6 +++ drivers/dma/ioat/ioat_dmadev.c | 69 ++++++++++++++++++++++++++ drivers/dma/ioat/ioat_hw_defs.h | 35 +++++++++++++ drivers/dma/ioat/ioat_internal.h | 20 ++++++++ drivers/dma/ioat/meson.build | 7 +++ drivers/dma/ioat/version.map | 3 ++ drivers/dma/meson.build | 1 + 10 files changed, 218 insertions(+) create mode 100644 doc/guides/dmadevs/ioat.rst create mode 100644 drivers/dma/ioat/ioat_dmadev.c create mode 100644 drivers/dma/ioat/ioat_hw_defs.h create mode 100644 drivers/dma/ioat/ioat_internal.h create mode 100644 drivers/dma/ioat/meson.build create mode 100644 drivers/dma/ioat/version.map diff --git a/MAINTAINERS b/MAINTAINERS index 423d8a73ce..283c70f7d7 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1209,6 +1209,12 @@ M: Kevin Laatz <kevin.laatz@intel.com> F: drivers/dma/idxd/ F: doc/guides/dmadevs/idxd.rst +Intel IOAT +M: Bruce Richardson <bruce.richardson@intel.com> +M: Conor Walsh <conor.walsh@intel.com> +F: drivers/dma/ioat/ +F: doc/guides/dmadevs/ioat.rst + RegEx Drivers ------------- diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst index 5d4abf880e..c59f4b5c92 100644 --- a/doc/guides/dmadevs/index.rst +++ b/doc/guides/dmadevs/index.rst @@ -12,3 +12,5 @@ an application through DMA API. :numbered: idxd + ioat + diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst new file mode 100644 index 0000000000..9ae1d8a2ad --- /dev/null +++ b/doc/guides/dmadevs/ioat.rst @@ -0,0 +1,69 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2021 Intel Corporation. + +.. include:: <isonum.txt> + +IOAT DMA Device Driver +======================= + +The ``ioat`` dmadev driver provides a poll-mode driver (PMD) for Intel\ +|reg| QuickData Technology which is part of part of Intel\ |reg| I/O +Acceleration Technology (`Intel I/OAT +<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_). +This PMD, when used on supported hardware, allows data copies, for example, +cloning packet data, to be accelerated by IOAT hardware rather than having to +be done by software, freeing up CPU cycles for other tasks. + +Hardware Requirements +---------------------- + +The ``dpdk-devbind.py`` script, included with DPDK, can be used to show the +presence of supported hardware. Running ``dpdk-devbind.py --status-dev dma`` +will show all the DMA devices on the system, IOAT devices are included in this +list. For Intel\ |reg| IOAT devices, the hardware will often be listed as +"Crystal Beach DMA", or "CBDMA" or on some newer systems '0b00' due to the +absence of pci-id database entries for them at this point. + +.. note:: + Error handling is not supported by this driver on hardware prior to + Intel Ice Lake. Unsupported systems include Broadwell, Skylake and + Cascade Lake. + +Compilation +------------ + +For builds using ``meson`` and ``ninja``, the driver will be built when the +target platform is x86-based. No additional compilation steps are necessary. + +Device Setup +------------- + +Intel\ |reg| IOAT devices will need to be bound to a suitable DPDK-supported +user-space IO driver such as ``vfio-pci`` in order to be used by DPDK. + +The ``dpdk-devbind.py`` script can be used to view the state of the devices using:: + + $ dpdk-devbind.py --status-dev dma + +The ``dpdk-devbind.py`` script can also be used to bind devices to a suitable driver. +For example:: + + $ dpdk-devbind.py -b vfio-pci 00:01.0 00:01.1 + +Device Probing and Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For devices bound to a suitable DPDK-supported driver (``vfio-pci``), the HW +devices will be found as part of the device scan done at application +initialization time without the need to pass parameters to the application. + +If the application does not require all the devices available an allowlist can +be used in the same way that other DPDK devices use them. + +For example:: + + $ dpdk-test -a <b:d:f> + +Once probed successfully, the device will appear as a ``dmadev``, that is a +"DMA device type" inside DPDK, and can be accessed using APIs from the +``rte_dmadev`` library. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index f8678efa94..3a85f2b33e 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -80,6 +80,12 @@ New Features The IDXD dmadev driver provide device drivers for the Intel DSA devices. This device driver can be used through the generic dmadev API. +* **Added IOAT dmadev driver implementation.** + + The Intel I/O Acceleration Technology (IOAT) dmadev driver provides a device + driver for Intel IOAT devices such as Crystal Beach DMA (CBDMA) on Ice Lake, + Skylake and Broadwell. This device driver can be used through the generic dmadev API. + * **Added new RSS offload types for IPv4/L4 checksum in RSS flow.** Added macros ETH_RSS_IPV4_CHKSUM and ETH_RSS_L4_CHKSUM, now IPv4 and diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c new file mode 100644 index 0000000000..f3491d45b1 --- /dev/null +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#include <rte_bus_pci.h> +#include <rte_dmadev_pmd.h> + +#include "ioat_internal.h" + +static struct rte_pci_driver ioat_pmd_drv; + +RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); + +#define IOAT_PMD_NAME dmadev_ioat +#define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) + +/* Probe DMA device. */ +static int +ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); + + dev->device.driver = &drv->driver; + return 0; +} + +/* Remove DMA device. */ +static int +ioat_dmadev_remove(struct rte_pci_device *dev) +{ + char name[32]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + + IOAT_PMD_INFO("Closing %s on NUMA node %d", + name, dev->device.numa_node); + + return 0; +} + +static const struct rte_pci_id pci_id_ioat_map[] = { + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_SKX) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX0) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX1) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX2) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX3) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX4) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX5) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX6) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX7) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXE) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXF) }, + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_ICX) }, + { .vendor_id = 0, /* sentinel */ }, +}; + +static struct rte_pci_driver ioat_pmd_drv = { + .id_table = pci_id_ioat_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC, + .probe = ioat_dmadev_probe, + .remove = ioat_dmadev_remove, +}; + +RTE_PMD_REGISTER_PCI(IOAT_PMD_NAME, ioat_pmd_drv); +RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_NAME, pci_id_ioat_map); +RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_NAME, "* igb_uio | uio_pci_generic | vfio-pci"); diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h new file mode 100644 index 0000000000..eeabba41ef --- /dev/null +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#ifndef IOAT_HW_DEFS_H +#define IOAT_HW_DEFS_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> + +#define IOAT_VER_3_0 0x30 +#define IOAT_VER_3_3 0x33 + +#define IOAT_VENDOR_ID 0x8086 +#define IOAT_DEVICE_ID_SKX 0x2021 +#define IOAT_DEVICE_ID_BDX0 0x6f20 +#define IOAT_DEVICE_ID_BDX1 0x6f21 +#define IOAT_DEVICE_ID_BDX2 0x6f22 +#define IOAT_DEVICE_ID_BDX3 0x6f23 +#define IOAT_DEVICE_ID_BDX4 0x6f24 +#define IOAT_DEVICE_ID_BDX5 0x6f25 +#define IOAT_DEVICE_ID_BDX6 0x6f26 +#define IOAT_DEVICE_ID_BDX7 0x6f27 +#define IOAT_DEVICE_ID_BDXE 0x6f2E +#define IOAT_DEVICE_ID_BDXF 0x6f2F +#define IOAT_DEVICE_ID_ICX 0x0b00 + +#ifdef __cplusplus +} +#endif + +#endif /* IOAT_HW_DEFS_H */ diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h new file mode 100644 index 0000000000..f1ec12a919 --- /dev/null +++ b/drivers/dma/ioat/ioat_internal.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021 Intel Corporation + */ + +#ifndef _IOAT_INTERNAL_H_ +#define _IOAT_INTERNAL_H_ + +#include "ioat_hw_defs.h" + +extern int ioat_pmd_logtype; + +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + ioat_pmd_logtype, "IOAT: %s(): " fmt "\n", __func__, ##args) + +#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) +#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) +#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) +#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) + +#endif /* _IOAT_INTERNAL_H_ */ diff --git a/drivers/dma/ioat/meson.build b/drivers/dma/ioat/meson.build new file mode 100644 index 0000000000..d67fac96fb --- /dev/null +++ b/drivers/dma/ioat/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021 Intel Corporation + +build = dpdk_conf.has('RTE_ARCH_X86') +reason = 'only supported on x86' +sources = files('ioat_dmadev.c') +deps += ['bus_pci', 'dmadev'] diff --git a/drivers/dma/ioat/version.map b/drivers/dma/ioat/version.map new file mode 100644 index 0000000000..c2e0723b4c --- /dev/null +++ b/drivers/dma/ioat/version.map @@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +}; diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build index 411be7a240..a69418ce9b 100644 --- a/drivers/dma/meson.build +++ b/drivers/dma/meson.build @@ -3,6 +3,7 @@ drivers = [ 'idxd', + 'ioat', 'skeleton', ] std_deps = ['dmadev'] -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v8 02/12] dma/ioat: create dmadev instances on PCI probe 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 01/12] dma/ioat: add device probe and removal functions Conor Walsh @ 2021-10-18 12:38 ` Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 03/12] dma/ioat: add datapath structures Conor Walsh ` (11 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-18 12:38 UTC (permalink / raw) To: bruce.richardson, thomas, fengchengwen, jerinj, kevin.laatz Cc: dev, Conor Walsh When a suitable device is found during the PCI probe, create a dmadev instance for each channel. Internal structures and HW definitions required for device creation are also included. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 105 ++++++++++++++++++++++++++++++- drivers/dma/ioat/ioat_hw_defs.h | 45 +++++++++++++ drivers/dma/ioat/ioat_internal.h | 27 ++++++++ 3 files changed, 175 insertions(+), 2 deletions(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index f3491d45b1..90f54567a4 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -4,6 +4,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> +#include <rte_malloc.h> #include "ioat_internal.h" @@ -14,6 +15,106 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Create a DMA device. */ +static int +ioat_dmadev_create(const char *name, struct rte_pci_device *dev) +{ + static const struct rte_dma_dev_ops ioat_dmadev_ops = { }; + + struct rte_dma_dev *dmadev = NULL; + struct ioat_dmadev *ioat = NULL; + int retry = 0; + + if (!name) { + IOAT_PMD_ERR("Invalid name of the device!"); + return -EINVAL; + } + + /* Allocate device structure. */ + dmadev = rte_dma_pmd_allocate(name, dev->device.numa_node, sizeof(struct ioat_dmadev)); + if (dmadev == NULL) { + IOAT_PMD_ERR("Unable to allocate dma device"); + return -ENOMEM; + } + + dmadev->device = &dev->device; + + dmadev->fp_obj->dev_private = dmadev->data->dev_private; + + dmadev->dev_ops = &ioat_dmadev_ops; + + ioat = dmadev->data->dev_private; + ioat->dmadev = dmadev; + ioat->regs = dev->mem_resource[0].addr; + ioat->doorbell = &ioat->regs->dmacount; + ioat->qcfg.nb_desc = 0; + ioat->desc_ring = NULL; + ioat->version = ioat->regs->cbver; + + /* Do device initialization - reset and set error behaviour. */ + if (ioat->regs->chancnt != 1) + IOAT_PMD_WARN("%s: Channel count == %d\n", __func__, + ioat->regs->chancnt); + + /* Locked by someone else. */ + if (ioat->regs->chanctrl & IOAT_CHANCTRL_CHANNEL_IN_USE) { + IOAT_PMD_WARN("%s: Channel appears locked\n", __func__); + ioat->regs->chanctrl = 0; + } + + /* clear any previous errors */ + if (ioat->regs->chanerr != 0) { + uint32_t val = ioat->regs->chanerr; + ioat->regs->chanerr = val; + } + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + rte_delay_ms(1); + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + rte_delay_ms(1); + while (ioat->regs->chancmd & IOAT_CHANCMD_RESET) { + ioat->regs->chainaddr = 0; + rte_delay_ms(1); + if (++retry >= 200) { + IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=%#"PRIx8 + ", CHANSTS=%#"PRIx64", CHANERR=%#"PRIx32"\n", + __func__, + ioat->regs->chancmd, + ioat->regs->chansts, + ioat->regs->chanerr); + rte_dma_pmd_release(name); + return -EIO; + } + } + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + + dmadev->fp_obj->dev_private = ioat; + + dmadev->state = RTE_DMA_DEV_READY; + + return 0; + +} + +/* Destroy a DMA device. */ +static int +ioat_dmadev_destroy(const char *name) +{ + int ret; + + if (!name) { + IOAT_PMD_ERR("Invalid device name"); + return -EINVAL; + } + + ret = rte_dma_pmd_release(name); + if (ret) + IOAT_PMD_DEBUG("Device cleanup failed"); + + return 0; +} + /* Probe DMA device. */ static int ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) @@ -24,7 +125,7 @@ ioat_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); dev->device.driver = &drv->driver; - return 0; + return ioat_dmadev_create(name, dev); } /* Remove DMA device. */ @@ -38,7 +139,7 @@ ioat_dmadev_remove(struct rte_pci_device *dev) IOAT_PMD_INFO("Closing %s on NUMA node %d", name, dev->device.numa_node); - return 0; + return ioat_dmadev_destroy(name); } static const struct rte_pci_id pci_id_ioat_map[] = { diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index eeabba41ef..73bdf548b3 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -11,6 +11,8 @@ extern "C" { #include <stdint.h> +#define IOAT_PCI_CHANERR_INT_OFFSET 0x180 + #define IOAT_VER_3_0 0x30 #define IOAT_VER_3_3 0x33 @@ -28,6 +30,49 @@ extern "C" { #define IOAT_DEVICE_ID_BDXF 0x6f2F #define IOAT_DEVICE_ID_ICX 0x0b00 +#define IOAT_COMP_UPDATE_SHIFT 3 +#define IOAT_CMD_OP_SHIFT 24 + +/* DMA Channel Registers */ +#define IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK 0xF000 +#define IOAT_CHANCTRL_COMPL_DCA_EN 0x0200 +#define IOAT_CHANCTRL_CHANNEL_IN_USE 0x0100 +#define IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL 0x0020 +#define IOAT_CHANCTRL_ERR_INT_EN 0x0010 +#define IOAT_CHANCTRL_ANY_ERR_ABORT_EN 0x0008 +#define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 +#define IOAT_CHANCTRL_INT_REARM 0x0001 + +struct ioat_registers { + uint8_t chancnt; + uint8_t xfercap; + uint8_t genctrl; + uint8_t intrctrl; + uint32_t attnstatus; + uint8_t cbver; /* 0x08 */ + uint8_t reserved4[0x3]; /* 0x09 */ + uint16_t intrdelay; /* 0x0C */ + uint16_t cs_status; /* 0x0E */ + uint32_t dmacapability; /* 0x10 */ + uint8_t reserved5[0x6C]; /* 0x14 */ + uint16_t chanctrl; /* 0x80 */ + uint8_t reserved6[0x2]; /* 0x82 */ + uint8_t chancmd; /* 0x84 */ + uint8_t reserved3[1]; /* 0x85 */ + uint16_t dmacount; /* 0x86 */ + uint64_t chansts; /* 0x88 */ + uint64_t chainaddr; /* 0x90 */ + uint64_t chancmp; /* 0x98 */ + uint8_t reserved2[0x8]; /* 0xA0 */ + uint32_t chanerr; /* 0xA8 */ + uint32_t chanerrmask; /* 0xAC */ +} __rte_packed; + +#define IOAT_CHANCMD_RESET 0x20 +#define IOAT_CHANCMD_SUSPEND 0x04 + +#define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ + #ifdef __cplusplus } #endif diff --git a/drivers/dma/ioat/ioat_internal.h b/drivers/dma/ioat/ioat_internal.h index f1ec12a919..4fa19eb811 100644 --- a/drivers/dma/ioat/ioat_internal.h +++ b/drivers/dma/ioat/ioat_internal.h @@ -7,6 +7,33 @@ #include "ioat_hw_defs.h" +struct ioat_dmadev { + struct rte_dma_dev *dmadev; + struct rte_dma_vchan_conf qcfg; + struct rte_dma_stats stats; + + volatile uint16_t *doorbell __rte_cache_aligned; + phys_addr_t status_addr; + phys_addr_t ring_addr; + + struct ioat_dma_hw_desc *desc_ring; + + unsigned short next_read; + unsigned short next_write; + unsigned short last_write; /* Used to compute submitted count. */ + unsigned short offset; /* Used after a device recovery when counts -> 0. */ + unsigned int failure; /* Used to store chanerr for error handling. */ + + /* To report completions, the device will write status back here. */ + volatile uint64_t status __rte_cache_aligned; + + /* Pointer to the register bar. */ + volatile struct ioat_registers *regs; + + /* Store the IOAT version. */ + uint8_t version; +}; + extern int ioat_pmd_logtype; #define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v8 03/12] dma/ioat: add datapath structures 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 01/12] dma/ioat: add device probe and removal functions Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 02/12] dma/ioat: create dmadev instances on PCI probe Conor Walsh @ 2021-10-18 12:38 ` Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 04/12] dma/ioat: add configuration functions Conor Walsh ` (10 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-18 12:38 UTC (permalink / raw) To: bruce.richardson, thomas, fengchengwen, jerinj, kevin.laatz Cc: dev, Conor Walsh Add data structures required for the data path of IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 70 ++++++++++- drivers/dma/ioat/ioat_hw_defs.h | 215 ++++++++++++++++++++++++++++++++ 2 files changed, 284 insertions(+), 1 deletion(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 90f54567a4..876e17f320 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -15,11 +15,79 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Dump DMA device info. */ +static int +__dev_dump(void *dev_private, FILE *f) +{ + struct ioat_dmadev *ioat = dev_private; + uint64_t chansts_masked = ioat->regs->chansts & IOAT_CHANSTS_STATUS; + uint32_t chanerr = ioat->regs->chanerr; + uint64_t mask = (ioat->qcfg.nb_desc - 1); + char ver = ioat->version; + fprintf(f, "========= IOAT =========\n"); + fprintf(f, " IOAT version: %d.%d\n", ver >> 4, ver & 0xF); + fprintf(f, " Channel status: %s [0x%"PRIx64"]\n", + chansts_readable[chansts_masked], chansts_masked); + fprintf(f, " ChainADDR: 0x%"PRIu64"\n", ioat->regs->chainaddr); + if (chanerr == 0) { + fprintf(f, " No Channel Errors\n"); + } else { + fprintf(f, " ChanERR: 0x%"PRIu32"\n", chanerr); + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + fprintf(f, " Invalid Source Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + fprintf(f, " Invalid Destination Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + fprintf(f, " Invalid Descriptor Length\n"); + if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + fprintf(f, " Descriptor Read Error\n"); + if ((chanerr & ~(IOAT_CHANERR_INVALID_SRC_ADDR_MASK | + IOAT_CHANERR_INVALID_DST_ADDR_MASK | + IOAT_CHANERR_INVALID_LENGTH_MASK | + IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK)) != 0) + fprintf(f, " Unknown Error(s)\n"); + } + fprintf(f, "== Private Data ==\n"); + fprintf(f, " Config: { ring_size: %u }\n", ioat->qcfg.nb_desc); + fprintf(f, " Status: 0x%"PRIx64"\n", ioat->status); + fprintf(f, " Status IOVA: 0x%"PRIx64"\n", ioat->status_addr); + fprintf(f, " Status ADDR: %p\n", &ioat->status); + fprintf(f, " Ring IOVA: 0x%"PRIx64"\n", ioat->ring_addr); + fprintf(f, " Ring ADDR: 0x%"PRIx64"\n", ioat->desc_ring[0].next-64); + fprintf(f, " Next write: %"PRIu16"\n", ioat->next_write); + fprintf(f, " Next read: %"PRIu16"\n", ioat->next_read); + struct ioat_dma_hw_desc *desc_ring = &ioat->desc_ring[(ioat->next_write - 1) & mask]; + fprintf(f, " Last Descriptor Written {\n"); + fprintf(f, " Size: %"PRIu32"\n", desc_ring->size); + fprintf(f, " Control: 0x%"PRIx32"\n", desc_ring->u.control_raw); + fprintf(f, " Src: 0x%"PRIx64"\n", desc_ring->src_addr); + fprintf(f, " Dest: 0x%"PRIx64"\n", desc_ring->dest_addr); + fprintf(f, " Next: 0x%"PRIx64"\n", desc_ring->next); + fprintf(f, " }\n"); + fprintf(f, " Next Descriptor {\n"); + fprintf(f, " Size: %"PRIu32"\n", ioat->desc_ring[ioat->next_read & mask].size); + fprintf(f, " Src: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].src_addr); + fprintf(f, " Dest: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); + fprintf(f, " Next: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].next); + fprintf(f, " }\n"); + + return 0; +} + +/* Public wrapper for dump. */ +static int +ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) +{ + return __dev_dump(dev->fp_obj->dev_private, f); +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { - static const struct rte_dma_dev_ops ioat_dmadev_ops = { }; + static const struct rte_dma_dev_ops ioat_dmadev_ops = { + .dev_dump = ioat_dev_dump, + }; struct rte_dma_dev *dmadev = NULL; struct ioat_dmadev *ioat = NULL; diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index 73bdf548b3..dc3493a78f 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -15,6 +15,7 @@ extern "C" { #define IOAT_VER_3_0 0x30 #define IOAT_VER_3_3 0x33 +#define IOAT_VER_3_4 0x34 #define IOAT_VENDOR_ID 0x8086 #define IOAT_DEVICE_ID_SKX 0x2021 @@ -43,6 +44,14 @@ extern "C" { #define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 #define IOAT_CHANCTRL_INT_REARM 0x0001 +/* DMA Channel Capabilities */ +#define IOAT_DMACAP_PB (1 << 0) +#define IOAT_DMACAP_DCA (1 << 4) +#define IOAT_DMACAP_BFILL (1 << 6) +#define IOAT_DMACAP_XOR (1 << 8) +#define IOAT_DMACAP_PQ (1 << 9) +#define IOAT_DMACAP_DMA_DIF (1 << 10) + struct ioat_registers { uint8_t chancnt; uint8_t xfercap; @@ -71,8 +80,214 @@ struct ioat_registers { #define IOAT_CHANCMD_RESET 0x20 #define IOAT_CHANCMD_SUSPEND 0x04 +#define IOAT_CHANSTS_STATUS 0x7ULL +#define IOAT_CHANSTS_ACTIVE 0x0 +#define IOAT_CHANSTS_IDLE 0x1 +#define IOAT_CHANSTS_SUSPENDED 0x2 +#define IOAT_CHANSTS_HALTED 0x3 +#define IOAT_CHANSTS_ARMED 0x4 + +#define IOAT_CHANERR_INVALID_SRC_ADDR_MASK (1 << 0) +#define IOAT_CHANERR_INVALID_DST_ADDR_MASK (1 << 1) +#define IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK (1 << 8) +#define IOAT_CHANERR_INVALID_LENGTH_MASK (1 << 10) + +const char *chansts_readable[] = { + "ACTIVE", /* 0x0 */ + "IDLE", /* 0x1 */ + "SUSPENDED", /* 0x2 */ + "HALTED", /* 0x3 */ + "ARMED" /* 0x4 */ +}; + +#define IOAT_CHANSTS_UNAFFILIATED_ERROR 0x8ULL +#define IOAT_CHANSTS_SOFT_ERROR 0x10ULL + +#define IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK (~0x3FULL) + #define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ +struct ioat_dma_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t null: 1; + uint32_t src_page_break: 1; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_COPY 0x00 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t reserved2; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_fill_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t reserved: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t reserved2: 2; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t reserved3: 15; +#define IOAT_OP_FILL 0x01 + uint32_t op: 8; + } control; + } u; + uint64_t src_data; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t next_dest_addr; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_xor_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_XOR 0x87 +#define IOAT_OP_XOR_VAL 0x88 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint64_t src_addr4; + uint64_t src_addr5; +}; + +struct ioat_xor_ext_hw_desc { + uint64_t src_addr6; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t next; + uint64_t reserved[4]; +}; + +struct ioat_pq_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 11; +#define IOAT_OP_PQ 0x89 +#define IOAT_OP_PQ_VAL 0x8a + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint8_t coef[8]; + uint64_t q_addr; +}; + +struct ioat_pq_ext_hw_desc { + uint64_t src_addr4; + uint64_t src_addr5; + uint64_t src_addr6; + uint64_t next; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t reserved[2]; +}; + +struct ioat_pq_update_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_cnt: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 3; + uint32_t coef: 8; +#define IOAT_OP_PQ_UP 0x8b + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t p_src; + uint64_t q_src; + uint64_t q_addr; +}; + +union ioat_hw_desc { + struct ioat_dma_hw_desc dma; + struct ioat_fill_hw_desc fill; + struct ioat_xor_hw_desc xor_desc; + struct ioat_xor_ext_hw_desc xor_ext; + struct ioat_pq_hw_desc pq; + struct ioat_pq_ext_hw_desc pq_ext; + struct ioat_pq_update_hw_desc pq_update; +}; + +#define GENSTS_DEV_STATE_MASK 0x03 +#define CMDSTATUS_ACTIVE_SHIFT 31 +#define CMDSTATUS_ACTIVE_MASK (1 << 31) +#define CMDSTATUS_ERR_MASK 0xFF + #ifdef __cplusplus } #endif -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v8 04/12] dma/ioat: add configuration functions 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (2 preceding siblings ...) 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 03/12] dma/ioat: add datapath structures Conor Walsh @ 2021-10-18 12:38 ` Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 05/12] dma/ioat: add start and stop functions Conor Walsh ` (9 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-18 12:38 UTC (permalink / raw) To: bruce.richardson, thomas, fengchengwen, jerinj, kevin.laatz Cc: dev, Conor Walsh Add functions for device configuration. The info_get and close functions are included here also. info_get can be useful for checking successful configuration and close is used by the dmadev api when releasing a configured device. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 15 +++++ drivers/dma/ioat/ioat_dmadev.c | 107 +++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 9ae1d8a2ad..af69556241 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -67,3 +67,18 @@ For example:: Once probed successfully, the device will appear as a ``dmadev``, that is a "DMA device type" inside DPDK, and can be accessed using APIs from the ``rte_dmadev`` library. + +Using IOAT DMAdev Devices +-------------------------- + +To use IOAT devices from an application, the ``dmadev`` API can be used. + +Device Configuration +~~~~~~~~~~~~~~~~~~~~~ + +IOAT configuration requirements: + +* ``ring_size`` must be a power of two, between 64 and 4096. +* Only one ``vchan`` is supported per device. +* Silent mode is not supported. +* The transfer direction must be set to ``RTE_DMA_DIR_MEM_TO_MEM`` to copy from memory to memory. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 876e17f320..ada57c5814 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -12,9 +12,112 @@ static struct rte_pci_driver ioat_pmd_drv; RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); +#define DESC_SZ sizeof(struct ioat_dma_hw_desc) + #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Configure a device. */ +static int +ioat_dev_configure(struct rte_dma_dev *dev __rte_unused, const struct rte_dma_conf *dev_conf, + uint32_t conf_sz) +{ + if (sizeof(struct rte_dma_conf) != conf_sz) + return -EINVAL; + + if (dev_conf->nb_vchans != 1) + return -EINVAL; + + return 0; +} + +/* Setup a virtual channel for IOAT, only 1 vchan is supported. */ +static int +ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + const struct rte_dma_vchan_conf *qconf, uint32_t qconf_sz) +{ + struct ioat_dmadev *ioat = dev->fp_obj->dev_private; + uint16_t max_desc = qconf->nb_desc; + int i; + + if (sizeof(struct rte_dma_vchan_conf) != qconf_sz) + return -EINVAL; + + ioat->qcfg = *qconf; + + if (!rte_is_power_of_2(max_desc)) { + max_desc = rte_align32pow2(max_desc); + IOAT_PMD_DEBUG("DMA dev %u using %u descriptors", dev->data->dev_id, max_desc); + ioat->qcfg.nb_desc = max_desc; + } + + /* In case we are reconfiguring a device, free any existing memory. */ + rte_free(ioat->desc_ring); + + ioat->desc_ring = rte_zmalloc(NULL, sizeof(*ioat->desc_ring) * max_desc, 0); + if (ioat->desc_ring == NULL) + return -ENOMEM; + + ioat->ring_addr = rte_mem_virt2iova(ioat->desc_ring); + + ioat->status_addr = rte_mem_virt2iova(ioat) + offsetof(struct ioat_dmadev, status); + + /* Ensure all counters are reset, if reconfiguring/restarting device. */ + ioat->next_read = 0; + ioat->next_write = 0; + ioat->last_write = 0; + ioat->offset = 0; + ioat->failure = 0; + + /* Configure descriptor ring - each one points to next. */ + for (i = 0; i < ioat->qcfg.nb_desc; i++) { + ioat->desc_ring[i].next = ioat->ring_addr + + (((i + 1) % ioat->qcfg.nb_desc) * DESC_SZ); + } + + return 0; +} + +/* Get device information of a device. */ +static int +ioat_dev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *info, uint32_t size) +{ + struct ioat_dmadev *ioat = dev->fp_obj->dev_private; + if (size < sizeof(*info)) + return -EINVAL; + info->dev_capa = RTE_DMA_CAPA_MEM_TO_MEM | + RTE_DMA_CAPA_OPS_COPY | + RTE_DMA_CAPA_OPS_FILL; + if (ioat->version >= IOAT_VER_3_4) + info->dev_capa |= RTE_DMA_CAPA_HANDLES_ERRORS; + info->max_vchans = 1; + info->min_desc = 32; + info->max_desc = 4096; + return 0; +} + +/* Close a configured device. */ +static int +ioat_dev_close(struct rte_dma_dev *dev) +{ + struct ioat_dmadev *ioat; + + if (!dev) { + IOAT_PMD_ERR("Invalid device"); + return -EINVAL; + } + + ioat = dev->fp_obj->dev_private; + if (!ioat) { + IOAT_PMD_ERR("Error getting dev_private"); + return -EINVAL; + } + + rte_free(ioat->desc_ring); + + return 0; +} + /* Dump DMA device info. */ static int __dev_dump(void *dev_private, FILE *f) @@ -86,7 +189,11 @@ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { static const struct rte_dma_dev_ops ioat_dmadev_ops = { + .dev_close = ioat_dev_close, + .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, + .dev_info_get = ioat_dev_info_get, + .vchan_setup = ioat_vchan_setup, }; struct rte_dma_dev *dmadev = NULL; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v8 05/12] dma/ioat: add start and stop functions 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (3 preceding siblings ...) 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 04/12] dma/ioat: add configuration functions Conor Walsh @ 2021-10-18 12:38 ` Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 06/12] dma/ioat: add data path job submission functions Conor Walsh ` (8 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-18 12:38 UTC (permalink / raw) To: bruce.richardson, thomas, fengchengwen, jerinj, kevin.laatz Cc: dev, Conor Walsh Add start, stop and recover functions for IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> --- doc/guides/dmadevs/ioat.rst | 3 ++ drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index af69556241..df159f9957 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -82,3 +82,6 @@ IOAT configuration requirements: * Only one ``vchan`` is supported per device. * Silent mode is not supported. * The transfer direction must be set to ``RTE_DMA_DIR_MEM_TO_MEM`` to copy from memory to memory. + +Once configured, the device can then be made ready for use by calling the +``rte_dma_start()`` API. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index ada57c5814..cf28f4a7e6 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -78,6 +78,96 @@ ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, return 0; } +/* Recover IOAT device. */ +static inline int +__ioat_recover(struct ioat_dmadev *ioat) +{ + uint32_t chanerr, retry = 0; + uint16_t mask = ioat->qcfg.nb_desc - 1; + + /* Clear any channel errors. Reading and writing to chanerr does this. */ + chanerr = ioat->regs->chanerr; + ioat->regs->chanerr = chanerr; + + /* Reset Channel. */ + ioat->regs->chancmd = IOAT_CHANCMD_RESET; + + /* Write new chain address to trigger state change. */ + ioat->regs->chainaddr = ioat->desc_ring[(ioat->next_read - 1) & mask].next; + /* Ensure channel control and status addr are correct. */ + ioat->regs->chanctrl = IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + IOAT_CHANCTRL_ERR_COMPLETION_EN; + ioat->regs->chancmp = ioat->status_addr; + + /* Allow HW time to move to the ARMED state. */ + do { + rte_pause(); + retry++; + } while (ioat->regs->chansts != IOAT_CHANSTS_ARMED && retry < 200); + + /* Exit as failure if device is still HALTED. */ + if (ioat->regs->chansts != IOAT_CHANSTS_ARMED) + return -1; + + /* Store next write as offset as recover will move HW and SW ring out of sync. */ + ioat->offset = ioat->next_read; + + /* Prime status register with previous address. */ + ioat->status = ioat->desc_ring[(ioat->next_read - 2) & mask].next; + + return 0; +} + +/* Start a configured device. */ +static int +ioat_dev_start(struct rte_dma_dev *dev) +{ + struct ioat_dmadev *ioat = dev->fp_obj->dev_private; + + if (ioat->qcfg.nb_desc == 0 || ioat->desc_ring == NULL) + return -EBUSY; + + /* Inform hardware of where the descriptor ring is. */ + ioat->regs->chainaddr = ioat->ring_addr; + /* Inform hardware of where to write the status/completions. */ + ioat->regs->chancmp = ioat->status_addr; + + /* Prime the status register to be set to the last element. */ + ioat->status = ioat->ring_addr + ((ioat->qcfg.nb_desc - 1) * DESC_SZ); + + printf("IOAT.status: %s [0x%"PRIx64"]\n", + chansts_readable[ioat->status & IOAT_CHANSTS_STATUS], + ioat->status); + + if ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_HALTED) { + IOAT_PMD_WARN("Device HALTED on start, attempting to recover\n"); + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device couldn't be recovered"); + return -1; + } + } + + return 0; +} + +/* Stop a configured device. */ +static int +ioat_dev_stop(struct rte_dma_dev *dev) +{ + struct ioat_dmadev *ioat = dev->fp_obj->dev_private; + uint32_t retry = 0; + + ioat->regs->chancmd = IOAT_CHANCMD_SUSPEND; + + do { + rte_pause(); + retry++; + } while ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) != IOAT_CHANSTS_SUSPENDED + && retry < 200); + + return ((ioat->regs->chansts & IOAT_CHANSTS_STATUS) == IOAT_CHANSTS_SUSPENDED) ? 0 : -1; +} + /* Get device information of a device. */ static int ioat_dev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *info, uint32_t size) @@ -193,6 +283,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_configure = ioat_dev_configure, .dev_dump = ioat_dev_dump, .dev_info_get = ioat_dev_info_get, + .dev_start = ioat_dev_start, + .dev_stop = ioat_dev_stop, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v8 06/12] dma/ioat: add data path job submission functions 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (4 preceding siblings ...) 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 05/12] dma/ioat: add start and stop functions Conor Walsh @ 2021-10-18 12:38 ` Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 07/12] dma/ioat: add data path completion functions Conor Walsh ` (7 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-18 12:38 UTC (permalink / raw) To: bruce.richardson, thomas, fengchengwen, jerinj, kevin.laatz Cc: dev, Conor Walsh Add data path functions for enqueuing and submitting operations to IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Reviewed-by: Chengwen Feng <fengchengwen@huawei.com> --- doc/guides/dmadevs/ioat.rst | 9 ++++ drivers/dma/ioat/ioat_dmadev.c | 92 ++++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index df159f9957..9ee4e372a8 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -85,3 +85,12 @@ IOAT configuration requirements: Once configured, the device can then be made ready for use by calling the ``rte_dma_start()`` API. + +Performing Data Copies +~~~~~~~~~~~~~~~~~~~~~~~ + +Refer to the :ref:`Enqueue / Dequeue APIs <dmadev_enqueue_dequeue>` section of the dmadev library +documentation for details on operation enqueue and submission API usage. + +It is expected that, for efficiency reasons, a burst of operations will be enqueued to the +device via multiple enqueue calls between calls to the ``rte_dma_submit()`` function. diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index cf28f4a7e6..4d00fec5c8 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -5,6 +5,7 @@ #include <rte_bus_pci.h> #include <rte_dmadev_pmd.h> #include <rte_malloc.h> +#include <rte_prefetch.h> #include "ioat_internal.h" @@ -17,6 +18,12 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* IOAT operations. */ +enum rte_ioat_ops { + ioat_op_copy = 0, /* Standard DMA Operation */ + ioat_op_fill /* Block Fill */ +}; + /* Configure a device. */ static int ioat_dev_configure(struct rte_dma_dev *dev __rte_unused, const struct rte_dma_conf *dev_conf, @@ -208,6 +215,87 @@ ioat_dev_close(struct rte_dma_dev *dev) return 0; } +/* Trigger hardware to begin performing enqueued operations. */ +static inline void +__submit(struct ioat_dmadev *ioat) +{ + *ioat->doorbell = ioat->next_write - ioat->offset; + + ioat->last_write = ioat->next_write; +} + +/* External submit function wrapper. */ +static int +ioat_submit(void *dev_private, uint16_t qid __rte_unused) +{ + struct ioat_dmadev *ioat = dev_private; + + __submit(ioat); + + return 0; +} + +/* Write descriptor for enqueue. */ +static inline int +__write_desc(void *dev_private, uint32_t op, uint64_t src, phys_addr_t dst, + unsigned int length, uint64_t flags) +{ + struct ioat_dmadev *ioat = dev_private; + uint16_t ret; + const unsigned short mask = ioat->qcfg.nb_desc - 1; + const unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + const unsigned short space = mask + read - write; + struct ioat_dma_hw_desc *desc; + + if (space == 0) + return -ENOSPC; + + ioat->next_write = write + 1; + write &= mask; + + desc = &ioat->desc_ring[write]; + desc->size = length; + desc->u.control_raw = (uint32_t)((op << IOAT_CMD_OP_SHIFT) | + (1 << IOAT_COMP_UPDATE_SHIFT)); + + /* In IOAT the fence ensures that all operations including the current one + * are completed before moving on, DMAdev assumes that the fence ensures + * all operations before the current one are completed before starting + * the current one, so in IOAT we set the fence for the previous descriptor. + */ + if (flags & RTE_DMA_OP_FLAG_FENCE) + ioat->desc_ring[(write - 1) & mask].u.control.fence = 1; + + desc->src_addr = src; + desc->dest_addr = dst; + + rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]); + + ret = (uint16_t)(ioat->next_write - 1); + + if (flags & RTE_DMA_OP_FLAG_SUBMIT) + __submit(ioat); + + return ret; +} + +/* Enqueue a fill operation onto the ioat device. */ +static int +ioat_enqueue_fill(void *dev_private, uint16_t qid __rte_unused, uint64_t pattern, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev_private, ioat_op_fill, pattern, dst, length, flags); +} + +/* Enqueue a copy operation onto the ioat device. */ +static int +ioat_enqueue_copy(void *dev_private, uint16_t qid __rte_unused, rte_iova_t src, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + return __write_desc(dev_private, ioat_op_copy, src, dst, length, flags); +} + /* Dump DMA device info. */ static int __dev_dump(void *dev_private, FILE *f) @@ -310,6 +398,10 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->fp_obj->copy = ioat_enqueue_copy; + dmadev->fp_obj->fill = ioat_enqueue_fill; + dmadev->fp_obj->submit = ioat_submit; + ioat = dmadev->data->dev_private; ioat->dmadev = dmadev; ioat->regs = dev->mem_resource[0].addr; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v8 07/12] dma/ioat: add data path completion functions 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (5 preceding siblings ...) 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 06/12] dma/ioat: add data path job submission functions Conor Walsh @ 2021-10-18 12:38 ` Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 08/12] dma/ioat: add statistics Conor Walsh ` (6 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-18 12:38 UTC (permalink / raw) To: bruce.richardson, thomas, fengchengwen, jerinj, kevin.laatz Cc: dev, Conor Walsh Add the data path functions for gathering completed operations from IOAT devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- doc/guides/dmadevs/ioat.rst | 33 +++++++- drivers/dma/ioat/ioat_dmadev.c | 141 +++++++++++++++++++++++++++++++++ 2 files changed, 173 insertions(+), 1 deletion(-) diff --git a/doc/guides/dmadevs/ioat.rst b/doc/guides/dmadevs/ioat.rst index 9ee4e372a8..9ac90e3108 100644 --- a/doc/guides/dmadevs/ioat.rst +++ b/doc/guides/dmadevs/ioat.rst @@ -90,7 +90,38 @@ Performing Data Copies ~~~~~~~~~~~~~~~~~~~~~~~ Refer to the :ref:`Enqueue / Dequeue APIs <dmadev_enqueue_dequeue>` section of the dmadev library -documentation for details on operation enqueue and submission API usage. +documentation for details on operation enqueue, submission and completion API usage. It is expected that, for efficiency reasons, a burst of operations will be enqueued to the device via multiple enqueue calls between calls to the ``rte_dma_submit()`` function. + +When gathering completions, ``rte_dma_completed()`` should be used, up until the point an error +occurs with an operation. If an error was encountered, ``rte_dma_completed_status()`` must be used +to reset the device and continue processing operations. This function will also gather the status +of each individual operation which is filled in to the ``status`` array provided as parameter +by the application. + +The status codes supported by IOAT are: + +* ``RTE_DMA_STATUS_SUCCESSFUL``: The operation was successful. +* ``RTE_DMA_STATUS_INVALID_SRC_ADDR``: The operation failed due to an invalid source address. +* ``RTE_DMA_STATUS_INVALID_DST_ADDR``: The operation failed due to an invalid destination address. +* ``RTE_DMA_STATUS_INVALID_LENGTH``: The operation failed due to an invalid descriptor length. +* ``RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR``: The device could not read the descriptor. +* ``RTE_DMA_STATUS_ERROR_UNKNOWN``: The operation failed due to an unspecified error. + +The following code shows how to retrieve the number of successfully completed +copies within a burst and then uses ``rte_dma_completed_status()`` to check +which operation failed and reset the device to continue processing operations: + +.. code-block:: C + + enum rte_dma_status_code status[COMP_BURST_SZ]; + uint16_t count, idx, status_count; + bool error = 0; + + count = rte_dma_completed(dev_id, vchan, COMP_BURST_SZ, &idx, &error); + + if (error){ + status_count = rte_dma_completed_status(dev_id, vchan, COMP_BURST_SZ, &idx, status); + } diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 4d00fec5c8..0318f67772 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -6,6 +6,7 @@ #include <rte_dmadev_pmd.h> #include <rte_malloc.h> #include <rte_prefetch.h> +#include <rte_errno.h> #include "ioat_internal.h" @@ -362,6 +363,144 @@ ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) return __dev_dump(dev->fp_obj->dev_private, f); } +/* Returns the index of the last completed operation. */ +static inline uint16_t +__get_last_completed(const struct ioat_dmadev *ioat, int *state) +{ + /* Status register contains the address of the completed operation */ + uint64_t status = ioat->status; + + /* lower 3 bits indicate "transfer status" : active, idle, halted. + * We can ignore bit 0. + */ + *state = status & IOAT_CHANSTS_STATUS; + + /* If we are just after recovering from an error the address returned by + * status will be 0, in this case we return the offset - 1 as the last + * completed. If not return the status value minus the chainaddr which + * gives us an offset into the ring. Right shifting by 6 (divide by 64) + * gives the index of the completion from the HW point of view and adding + * the offset translates the ring index from HW to SW point of view. + */ + if ((status & ~IOAT_CHANSTS_STATUS) == 0) + return ioat->offset - 1; + + return (status - ioat->ring_addr) >> 6; +} + +/* Translates IOAT ChanERRs to DMA error codes. */ +static inline enum rte_dma_status_code +__translate_status_ioat_to_dma(uint32_t chanerr) +{ + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_SRC_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + return RTE_DMA_STATUS_INVALID_DST_ADDR; + else if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + return RTE_DMA_STATUS_INVALID_LENGTH; + else if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + return RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR; + else + return RTE_DMA_STATUS_ERROR_UNKNOWN; +} + +/* Returns details of operations that have been completed. */ +static uint16_t +ioat_completed(void *dev_private, uint16_t qid __rte_unused, const uint16_t max_ops, + uint16_t *last_idx, bool *has_error) +{ + struct ioat_dmadev *ioat = dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short last_completed, count; + int state, fails = 0; + + /* Do not do any work if there is an uncleared error. */ + if (ioat->failure != 0) { + *has_error = true; + *last_idx = ioat->next_read - 2; + return 0; + } + + last_completed = __get_last_completed(ioat, &state); + count = (last_completed + 1 - read) & mask; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || state != IOAT_CHANSTS_HALTED) { + ioat->next_read = read + count; + *last_idx = ioat->next_read - 1; + } else { + *has_error = true; + rte_errno = EIO; + ioat->failure = ioat->regs->chanerr; + ioat->next_read = read + count + 1; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + __dev_dump(dev_private, stdout); + return 0; + } + __submit(ioat); + fails++; + *last_idx = ioat->next_read - 2; + } + + return count; +} + +/* Returns detailed status information about operations that have been completed. */ +static uint16_t +ioat_completed_status(void *dev_private, uint16_t qid __rte_unused, + uint16_t max_ops, uint16_t *last_idx, enum rte_dma_status_code *status) +{ + struct ioat_dmadev *ioat = dev_private; + + const unsigned short mask = (ioat->qcfg.nb_desc - 1); + const unsigned short read = ioat->next_read; + unsigned short count, last_completed; + uint64_t fails = 0; + int state, i; + + last_completed = __get_last_completed(ioat, &state); + count = (last_completed + 1 - read) & mask; + + for (i = 0; i < RTE_MIN(count + 1, max_ops); i++) + status[i] = RTE_DMA_STATUS_SUCCESSFUL; + + /* Cap count at max_ops or set as last run in batch. */ + if (count > max_ops) + count = max_ops; + + if (count == max_ops || state != IOAT_CHANSTS_HALTED) + ioat->next_read = read + count; + else { + rte_errno = EIO; + status[count] = __translate_status_ioat_to_dma(ioat->regs->chanerr); + count++; + ioat->next_read = read + count; + if (__ioat_recover(ioat) != 0) { + IOAT_PMD_ERR("Device HALTED and could not be recovered\n"); + __dev_dump(dev_private, stdout); + return 0; + } + __submit(ioat); + fails++; + } + + if (ioat->failure > 0) { + status[0] = __translate_status_ioat_to_dma(ioat->failure); + count = RTE_MIN(count + 1, max_ops); + ioat->failure = 0; + } + + *last_idx = ioat->next_read - 1; + + return count; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -398,6 +537,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->fp_obj->completed = ioat_completed; + dmadev->fp_obj->completed_status = ioat_completed_status; dmadev->fp_obj->copy = ioat_enqueue_copy; dmadev->fp_obj->fill = ioat_enqueue_fill; dmadev->fp_obj->submit = ioat_submit; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v8 08/12] dma/ioat: add statistics 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (6 preceding siblings ...) 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 07/12] dma/ioat: add data path completion functions Conor Walsh @ 2021-10-18 12:38 ` Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 09/12] dma/ioat: add support for vchan status function Conor Walsh ` (5 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-18 12:38 UTC (permalink / raw) To: bruce.richardson, thomas, fengchengwen, jerinj, kevin.laatz Cc: dev, Conor Walsh Add statistic tracking for operations in IOAT. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 43 ++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 0318f67772..48126a1bcb 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -77,6 +77,9 @@ ioat_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused, ioat->offset = 0; ioat->failure = 0; + /* Reset Stats. */ + ioat->stats = (struct rte_dma_stats){0}; + /* Configure descriptor ring - each one points to next. */ for (i = 0; i < ioat->qcfg.nb_desc; i++) { ioat->desc_ring[i].next = ioat->ring_addr + @@ -222,6 +225,8 @@ __submit(struct ioat_dmadev *ioat) { *ioat->doorbell = ioat->next_write - ioat->offset; + ioat->stats.submitted += (uint16_t)(ioat->next_write - ioat->last_write); + ioat->last_write = ioat->next_write; } @@ -352,6 +357,10 @@ __dev_dump(void *dev_private, FILE *f) fprintf(f, " Dest: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); fprintf(f, " Next: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].next); fprintf(f, " }\n"); + fprintf(f, " Key Stats { submitted: %"PRIu64", comp: %"PRIu64", failed: %"PRIu64" }\n", + ioat->stats.submitted, + ioat->stats.completed, + ioat->stats.errors); return 0; } @@ -448,6 +457,9 @@ ioat_completed(void *dev_private, uint16_t qid __rte_unused, const uint16_t max_ *last_idx = ioat->next_read - 2; } + ioat->stats.completed += count; + ioat->stats.errors += fails; + return count; } @@ -498,9 +510,38 @@ ioat_completed_status(void *dev_private, uint16_t qid __rte_unused, *last_idx = ioat->next_read - 1; + ioat->stats.completed += count; + ioat->stats.errors += fails; + return count; } +/* Retrieve the generic stats of a DMA device. */ +static int +ioat_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + struct rte_dma_stats *rte_stats, uint32_t size) +{ + struct rte_dma_stats *stats = (&((struct ioat_dmadev *)dev->fp_obj->dev_private)->stats); + + if (size < sizeof(rte_stats)) + return -EINVAL; + if (rte_stats == NULL) + return -EINVAL; + + *rte_stats = *stats; + return 0; +} + +/* Reset the generic stat counters for the DMA device. */ +static int +ioat_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused) +{ + struct ioat_dmadev *ioat = dev->fp_obj->dev_private; + + ioat->stats = (struct rte_dma_stats){0}; + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -512,6 +553,8 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_info_get = ioat_dev_info_get, .dev_start = ioat_dev_start, .dev_stop = ioat_dev_stop, + .stats_get = ioat_stats_get, + .stats_reset = ioat_stats_reset, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v8 09/12] dma/ioat: add support for vchan status function 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (7 preceding siblings ...) 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 08/12] dma/ioat: add statistics Conor Walsh @ 2021-10-18 12:38 ` Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 10/12] dma/ioat: add burst capacity function Conor Walsh ` (4 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-18 12:38 UTC (permalink / raw) To: bruce.richardson, thomas, fengchengwen, jerinj, kevin.laatz Cc: dev, Conor Walsh Add support for the rte_dmadev_vchan_status API call. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 48126a1bcb..3d4d7b66e2 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -542,6 +542,26 @@ ioat_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused) return 0; } +/* Check if the IOAT device is idle. */ +static int +ioat_vchan_status(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, + enum rte_dma_vchan_status *status) +{ + int state = 0; + const struct ioat_dmadev *ioat = dev->fp_obj->dev_private; + const uint16_t mask = ioat->qcfg.nb_desc - 1; + const uint16_t last = __get_last_completed(ioat, &state); + + if (state == IOAT_CHANSTS_HALTED || state == IOAT_CHANSTS_SUSPENDED) + *status = RTE_DMA_VCHAN_HALTED_ERROR; + else if (last == ((ioat->next_write - 1) & mask)) + *status = RTE_DMA_VCHAN_IDLE; + else + *status = RTE_DMA_VCHAN_ACTIVE; + + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) @@ -555,6 +575,7 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) .dev_stop = ioat_dev_stop, .stats_get = ioat_stats_get, .stats_reset = ioat_stats_reset, + .vchan_status = ioat_vchan_status, .vchan_setup = ioat_vchan_setup, }; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v8 10/12] dma/ioat: add burst capacity function 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (8 preceding siblings ...) 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 09/12] dma/ioat: add support for vchan status function Conor Walsh @ 2021-10-18 12:38 ` Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 11/12] devbind: move ioat device IDs to dmadev category Conor Walsh ` (3 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-18 12:38 UTC (permalink / raw) To: bruce.richardson, thomas, fengchengwen, jerinj, kevin.laatz Cc: dev, Conor Walsh Adds the ability to find the remaining space in the IOAT ring. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Signed-off-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> --- drivers/dma/ioat/ioat_dmadev.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index 3d4d7b66e2..a230496b11 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -516,6 +516,19 @@ ioat_completed_status(void *dev_private, uint16_t qid __rte_unused, return count; } +/* Get the remaining capacity of the ring. */ +static uint16_t +ioat_burst_capacity(const void *dev_private, uint16_t vchan __rte_unused) +{ + const struct ioat_dmadev *ioat = dev_private; + unsigned short size = ioat->qcfg.nb_desc - 1; + unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + unsigned short space = size - (write - read); + + return space; +} + /* Retrieve the generic stats of a DMA device. */ static int ioat_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused, @@ -601,6 +614,7 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) dmadev->dev_ops = &ioat_dmadev_ops; + dmadev->fp_obj->burst_capacity = ioat_burst_capacity; dmadev->fp_obj->completed = ioat_completed; dmadev->fp_obj->completed_status = ioat_completed_status; dmadev->fp_obj->copy = ioat_enqueue_copy; -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v8 11/12] devbind: move ioat device IDs to dmadev category 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (9 preceding siblings ...) 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 10/12] dma/ioat: add burst capacity function Conor Walsh @ 2021-10-18 12:38 ` Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh ` (2 subsequent siblings) 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-18 12:38 UTC (permalink / raw) To: bruce.richardson, thomas, fengchengwen, jerinj, kevin.laatz Cc: dev, Conor Walsh Move Intel IOAT devices from Misc to DMA devices. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Reviewed-by: Kevin Laatz <kevin.laatz@intel.com> Reviewed-by: Bruce Richardson <bruce.richardson@intel.com> --- usertools/dpdk-devbind.py | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index ba18e2a487..91f1b16bde 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -71,14 +71,13 @@ network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] -dma_devices = [intel_idxd_spr] +dma_devices = [intel_idxd_spr, intel_ioat_bdw, intel_ioat_icx, intel_ioat_skx] eventdev_devices = [cavium_sso, cavium_tim, intel_dlb, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] regex_devices = [octeontx2_ree] -misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, intel_ioat_bdw, - intel_ioat_skx, intel_ioat_icx, intel_ntb_skx, - intel_ntb_icx, octeontx2_dma] +misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, intel_ntb_skx, + intel_ntb_icx, octeontx2_dma] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* [dpdk-dev] [PATCH v8 12/12] raw/ioat: deprecate ioat rawdev driver 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (10 preceding siblings ...) 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 11/12] devbind: move ioat device IDs to dmadev category Conor Walsh @ 2021-10-18 12:38 ` Conor Walsh 2021-10-19 11:28 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Walsh, Conor 2021-10-22 19:25 ` Thomas Monjalon 13 siblings, 0 replies; 132+ messages in thread From: Conor Walsh @ 2021-10-18 12:38 UTC (permalink / raw) To: bruce.richardson, thomas, fengchengwen, jerinj, kevin.laatz Cc: dev, Conor Walsh Deprecate the rawdev IOAT driver as both IOAT and IDXD drivers have moved to dmadev. Signed-off-by: Conor Walsh <conor.walsh@intel.com> Acked-by: Kevin Laatz <kevin.laatz@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Thomas Monjalon <thomas@monjalon.net> --- MAINTAINERS | 2 +- doc/guides/rawdevs/ioat.rst | 4 ++++ doc/guides/rel_notes/deprecation.rst | 7 +++++++ 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/MAINTAINERS b/MAINTAINERS index 283c70f7d7..b9f7746dc4 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1322,7 +1322,7 @@ T: git://dpdk.org/next/dpdk-next-net-intel F: drivers/raw/ifpga/ F: doc/guides/rawdevs/ifpga.rst -IOAT Rawdev +IOAT Rawdev - DEPRECATED M: Bruce Richardson <bruce.richardson@intel.com> F: drivers/raw/ioat/ F: doc/guides/rawdevs/ioat.rst diff --git a/doc/guides/rawdevs/ioat.rst b/doc/guides/rawdevs/ioat.rst index a65530bd30..98d15dd032 100644 --- a/doc/guides/rawdevs/ioat.rst +++ b/doc/guides/rawdevs/ioat.rst @@ -6,6 +6,10 @@ IOAT Rawdev Driver =================== +.. warning:: + As of DPDK 21.11 the rawdev implementation of the IOAT driver has been deprecated. + Please use the dmadev library instead. + The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg| Data Streaming Accelerator `(Intel DSA) <https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator>`_ and for Intel\ |reg| diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst index 45239ca56e..1014c45bc5 100644 --- a/doc/guides/rel_notes/deprecation.rst +++ b/doc/guides/rel_notes/deprecation.rst @@ -236,3 +236,10 @@ Deprecation Notices * cmdline: ``cmdline`` structure will be made opaque to hide platform-specific content. On Linux and FreeBSD, supported prior to DPDK 20.11, original structure will be kept until DPDK 21.11. + +* raw/ioat: The ``ioat`` rawdev driver has been deprecated, since it's + functionality is provided through the new ``dmadev`` infrastructure. + To continue to use hardware previously supported by the ``ioat`` rawdev driver, + applications should be updated to use the ``dmadev`` library instead, + with the underlying HW-functionality being provided by the ``ioat`` or + ``idxd`` dma drivers -- 2.25.1 ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (11 preceding siblings ...) 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh @ 2021-10-19 11:28 ` Walsh, Conor 2021-10-19 14:23 ` Walsh, Conor 2021-10-22 19:25 ` Thomas Monjalon 13 siblings, 1 reply; 132+ messages in thread From: Walsh, Conor @ 2021-10-19 11:28 UTC (permalink / raw) To: Richardson, Bruce, thomas, fengchengwen, jerinj, Laatz, Kevin; +Cc: dev This patchset is compatible with v9 of the "add dmadev driver for idxd devices" patchset (series-19786) Thanks, Conor. > This patchset adds a dmadev driver and associated documentation to > support > Intel QuickData Technology devices, part of the Intel I/O Acceleration > Technology (Intel I/OAT). This driver is intended to ultimately replace > the current IOAT part of the IOAT rawdev driver. > This patchset passes all the driver tests added in the dmadev test suite. > > Depends-on: series-19737 ("add dmadev driver for idxd devices") ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices 2021-10-19 11:28 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Walsh, Conor @ 2021-10-19 14:23 ` Walsh, Conor 2021-10-20 16:35 ` Walsh, Conor 0 siblings, 1 reply; 132+ messages in thread From: Walsh, Conor @ 2021-10-19 14:23 UTC (permalink / raw) To: Richardson, Bruce, thomas, fengchengwen, jerinj, Laatz, Kevin; +Cc: dev This patchset is compatible with v10 of the "add dmadev driver for idxd devices" patchset (series-19797) Thanks, Conor. > This patchset is compatible with v9 of the "add dmadev driver for idxd > devices" patchset (series-19786) > > Thanks, > Conor. > > > This patchset adds a dmadev driver and associated documentation to > > support > > Intel QuickData Technology devices, part of the Intel I/O Acceleration > > Technology (Intel I/OAT). This driver is intended to ultimately replace > > the current IOAT part of the IOAT rawdev driver. > > This patchset passes all the driver tests added in the dmadev test suite. > > > > Depends-on: series-19737 ("add dmadev driver for idxd devices") ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices 2021-10-19 14:23 ` Walsh, Conor @ 2021-10-20 16:35 ` Walsh, Conor 0 siblings, 0 replies; 132+ messages in thread From: Walsh, Conor @ 2021-10-20 16:35 UTC (permalink / raw) To: Richardson, Bruce, thomas, fengchengwen, jerinj, Laatz, Kevin; +Cc: dev This patchset is compatible with v11 of the "add dmadev driver for idxd devices" patchset (series-19848) > This patchset is compatible with v10 of the "add dmadev driver for idxd > devices" patchset (series-19797) > > Thanks, > Conor. > > > This patchset is compatible with v9 of the "add dmadev driver for idxd > > devices" patchset (series-19786) > > > > Thanks, > > Conor. > > > > > This patchset adds a dmadev driver and associated documentation to > > > support > > > Intel QuickData Technology devices, part of the Intel I/O Acceleration > > > Technology (Intel I/OAT). This driver is intended to ultimately replace > > > the current IOAT part of the IOAT rawdev driver. > > > This patchset passes all the driver tests added in the dmadev test suite. > > > > > > Depends-on: series-19737 ("add dmadev driver for idxd devices") ^ permalink raw reply [flat|nested] 132+ messages in thread
* Re: [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh ` (12 preceding siblings ...) 2021-10-19 11:28 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Walsh, Conor @ 2021-10-22 19:25 ` Thomas Monjalon 13 siblings, 0 replies; 132+ messages in thread From: Thomas Monjalon @ 2021-10-22 19:25 UTC (permalink / raw) To: kevin.laatz, Conor Walsh; +Cc: bruce.richardson, fengchengwen, jerinj, dev 18/10/2021 14:38, Conor Walsh: > This patchset adds a dmadev driver and associated documentation to support > Intel QuickData Technology devices, part of the Intel I/O Acceleration > Technology (Intel I/OAT). This driver is intended to ultimately replace > the current IOAT part of the IOAT rawdev driver. > This patchset passes all the driver tests added in the dmadev test suite. Applied, thanks ^ permalink raw reply [flat|nested] 132+ messages in thread
end of thread, other threads:[~2021-10-22 19:25 UTC | newest] Thread overview: 132+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-08-27 17:25 [dpdk-dev] [PATCH 0/8] dma: add dmadev driver for ioat devices Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 1/8] dma/ioat: add device probe and removal functions Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 2/8] dma/ioat: create dmadev instances on PCI probe Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 3/8] dma/ioat: add datapath structures Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 4/8] dma/ioat: add configuration functions Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 5/8] dma/ioat: add start and stop functions Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 6/8] dma/ioat: add data path job submission functions Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 7/8] dma/ioat: add data path completion functions Conor Walsh 2021-08-27 17:25 ` [dpdk-dev] [PATCH 8/8] dma/ioat: add statistics Conor Walsh 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 00/10] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 01/10] dma/ioat: add device probe and removal functions Conor Walsh 2021-09-07 10:10 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 02/10] dma/ioat: create dmadev instances on PCI probe Conor Walsh 2021-09-07 10:11 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 03/10] dma/ioat: add datapath structures Conor Walsh 2021-09-07 10:11 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 04/10] dma/ioat: add configuration functions Conor Walsh 2021-09-07 10:11 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 05/10] dma/ioat: add start and stop functions Conor Walsh 2021-09-07 10:11 ` Kevin Laatz 2021-09-07 10:35 ` Walsh, Conor 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 06/10] dma/ioat: add data path job submission functions Conor Walsh 2021-09-07 10:12 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 07/10] dma/ioat: add data path completion functions Conor Walsh 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 08/10] dma/ioat: add statistics Conor Walsh 2021-09-07 10:12 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 09/10] dma/ioat: add support for vchan idle function Conor Walsh 2021-09-07 10:12 ` Kevin Laatz 2021-09-03 11:17 ` [dpdk-dev] [PATCH v2 10/10] devbind: move ioat device ID for ICX to dmadev category Conor Walsh 2021-09-07 10:10 ` Kevin Laatz 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 00/11] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 01/11] dma/ioat: add device probe and removal functions Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 02/11] dma/ioat: create dmadev instances on PCI probe Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 03/11] dma/ioat: add datapath structures Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 04/11] dma/ioat: add configuration functions Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 05/11] dma/ioat: add start and stop functions Conor Walsh 2021-09-08 10:39 ` [dpdk-dev] [PATCH v3 06/11] dma/ioat: add data path job submission functions Conor Walsh 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 07/11] dma/ioat: add data path completion functions Conor Walsh 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 08/11] dma/ioat: add statistics Conor Walsh 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 09/11] dma/ioat: add support for vchan status function Conor Walsh 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 10/11] dma/ioat: add burst capacity function Conor Walsh 2021-09-08 10:40 ` [dpdk-dev] [PATCH v3 11/11] devbind: move ioat device ID for ICX to dmadev category Conor Walsh 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 00/11] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 01/11] dma/ioat: add device probe and removal functions Conor Walsh 2021-09-20 11:15 ` Bruce Richardson 2021-09-21 16:24 ` Conor Walsh 2021-09-22 3:59 ` fengchengwen 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 02/11] dma/ioat: create dmadev instances on PCI probe Conor Walsh 2021-09-20 13:31 ` Bruce Richardson 2021-09-21 16:25 ` Conor Walsh 2021-09-22 8:04 ` fengchengwen 2021-09-22 16:40 ` Conor Walsh 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 03/11] dma/ioat: add datapath structures Conor Walsh 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 04/11] dma/ioat: add configuration functions Conor Walsh 2021-09-22 8:08 ` fengchengwen 2021-09-22 16:41 ` Conor Walsh 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 05/11] dma/ioat: add start and stop functions Conor Walsh 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 06/11] dma/ioat: add data path job submission functions Conor Walsh 2021-09-20 13:36 ` Bruce Richardson 2021-09-21 16:25 ` Conor Walsh 2021-09-22 8:18 ` fengchengwen 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 07/11] dma/ioat: add data path completion functions Conor Walsh 2021-09-20 13:39 ` Bruce Richardson 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 08/11] dma/ioat: add statistics Conor Walsh 2021-09-20 13:48 ` Bruce Richardson 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 09/11] dma/ioat: add support for vchan status function Conor Walsh 2021-09-20 13:49 ` Bruce Richardson 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 10/11] dma/ioat: add burst capacity function Conor Walsh 2021-09-20 13:49 ` Bruce Richardson 2021-09-17 15:42 ` [dpdk-dev] [PATCH v4 11/11] devbind: move ioat device ID for ICX to dmadev category Conor Walsh 2021-09-20 13:53 ` Bruce Richardson 2021-09-21 16:26 ` Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 00/12] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 01/12] dma/ioat: add device probe and removal functions Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 02/12] dma/ioat: create dmadev instances on PCI probe Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 03/12] dma/ioat: add datapath structures Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 04/12] dma/ioat: add configuration functions Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 05/12] dma/ioat: add start and stop functions Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 06/12] dma/ioat: add data path job submission functions Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 07/12] dma/ioat: add data path completion functions Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 08/12] dma/ioat: add statistics Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 09/12] dma/ioat: add support for vchan status function Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 10/12] dma/ioat: add burst capacity function Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 11/12] devbind: move ioat device IDs to dmadev category Conor Walsh 2021-09-24 14:33 ` [dpdk-dev] [PATCH v5 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh 2021-09-24 16:21 ` Kevin Laatz 2021-09-24 16:56 ` Bruce Richardson 2021-09-27 9:36 ` Walsh, Conor 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 00/12] dma: add dmadev driver for ioat devices Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 01/12] dma/ioat: add device probe and removal functions Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 02/12] dma/ioat: create dmadev instances on PCI probe Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 03/12] dma/ioat: add datapath structures Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 04/12] dma/ioat: add configuration functions Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 05/12] dma/ioat: add start and stop functions Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 06/12] dma/ioat: add data path job submission functions Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 07/12] dma/ioat: add data path completion functions Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 08/12] dma/ioat: add statistics Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 09/12] dma/ioat: add support for vchan status function Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 10/12] dma/ioat: add burst capacity function Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 11/12] devbind: move ioat device IDs to dmadev category Conor Walsh 2021-09-27 10:21 ` [dpdk-dev] [PATCH v6 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 00/12] dma: add dmadev driver for ioat devices Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 01/12] dma/ioat: add device probe and removal functions Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 02/12] dma/ioat: create dmadev instances on PCI probe Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 03/12] dma/ioat: add datapath structures Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 04/12] dma/ioat: add configuration functions Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 05/12] dma/ioat: add start and stop functions Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 06/12] dma/ioat: add data path job submission functions Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 07/12] dma/ioat: add data path completion functions Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 08/12] dma/ioat: add statistics Conor Walsh 2021-10-14 9:48 ` [dpdk-dev] [PATCH v7 09/12] dma/ioat: add support for vchan status function Conor Walsh 2021-10-14 9:49 ` [dpdk-dev] [PATCH v7 10/12] dma/ioat: add burst capacity function Conor Walsh 2021-10-14 9:49 ` [dpdk-dev] [PATCH v7 11/12] devbind: move ioat device IDs to dmadev category Conor Walsh 2021-10-14 9:49 ` [dpdk-dev] [PATCH v7 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh 2021-10-18 7:54 ` Thomas Monjalon 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 01/12] dma/ioat: add device probe and removal functions Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 02/12] dma/ioat: create dmadev instances on PCI probe Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 03/12] dma/ioat: add datapath structures Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 04/12] dma/ioat: add configuration functions Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 05/12] dma/ioat: add start and stop functions Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 06/12] dma/ioat: add data path job submission functions Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 07/12] dma/ioat: add data path completion functions Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 08/12] dma/ioat: add statistics Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 09/12] dma/ioat: add support for vchan status function Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 10/12] dma/ioat: add burst capacity function Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 11/12] devbind: move ioat device IDs to dmadev category Conor Walsh 2021-10-18 12:38 ` [dpdk-dev] [PATCH v8 12/12] raw/ioat: deprecate ioat rawdev driver Conor Walsh 2021-10-19 11:28 ` [dpdk-dev] [PATCH v8 00/12] dma: add dmadev driver for ioat devices Walsh, Conor 2021-10-19 14:23 ` Walsh, Conor 2021-10-20 16:35 ` Walsh, Conor 2021-10-22 19:25 ` Thomas Monjalon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).