[dpdk-dev] [PATCH 0/8] raw/ioat: driver for Intel QuickData Technology

[dpdk-dev] [PATCH 0/8] raw/ioat: driver for Intel QuickData Technology

[Bruce Richardson]

This patch series adds support for the Intel QuickData Technology
device, part of the Intel I/O Acceleration Technology (Intel I/OAT). It
is a raw device for allowing hardware DMA i.e. data copies in hardware.

Bruce Richardson (8):
  raw/ioat: add initial support for ioat rawdev driver
  usertools/dpdk-devbind.py: add support for IOAT devices
  raw/ioat: add register definition file
  raw/ioat: create device on probe and destroy on release
  raw/ioat: add device info function
  raw/ioat: add configure, start and stop functions
  raw/ioat: add statistics functions
  raw/ioat: add local API to perform copies

 MAINTAINERS                                 |   7 +-
 app/test/Makefile                           |   1 +
 app/test/meson.build                        |   4 +
 app/test/test_ioat_rawdev.c                 | 269 +++++++++++++++++
 config/common_armv8a_linux                  |   1 +
 config/common_base                          |   5 +
 config/defconfig_arm-armv7a-linuxapp-gcc    |   1 +
 config/defconfig_ppc_64-power8-linuxapp-gcc |   1 +
 doc/guides/rawdevs/index.rst                |   1 +
 doc/guides/rawdevs/ioat_rawdev.rst          | 227 ++++++++++++++
 doc/guides/rel_notes/release_19_08.rst      |  11 +
 drivers/raw/Makefile                        |   1 +
 drivers/raw/ioat/Makefile                   |  29 ++
 drivers/raw/ioat/ioat_rawdev.c              | 310 ++++++++++++++++++++
 drivers/raw/ioat/meson.build                |   9 +
 drivers/raw/ioat/rte_ioat_rawdev.h          | 228 ++++++++++++++
 drivers/raw/ioat/rte_ioat_spec.h            | 301 +++++++++++++++++++
 drivers/raw/ioat/rte_pmd_ioat_version.map   |   4 +
 drivers/raw/meson.build                     |   3 +-
 mk/rte.app.mk                               |   1 +
 usertools/dpdk-devbind.py                   |  10 +
 21 files changed, 1422 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_ioat_rawdev.c
 create mode 100644 doc/guides/rawdevs/ioat_rawdev.rst
 create mode 100644 drivers/raw/ioat/Makefile
 create mode 100644 drivers/raw/ioat/ioat_rawdev.c
 create mode 100644 drivers/raw/ioat/meson.build
 create mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h
 create mode 100644 drivers/raw/ioat/rte_ioat_spec.h
 create mode 100644 drivers/raw/ioat/rte_pmd_ioat_version.map

-- 
2.21.0

[dpdk-dev] [PATCH 1/8] raw/ioat: add initial support for ioat rawdev driver

[Bruce Richardson]

Add stubs for ioat rawdev driver support in DPDK, specifically:

  * makefile and meson build hooks
  * initial public header file
  * rawdev main C file, with probe and release functions
  * release note update announcing the driver
  * initial documentation for the new section in the rawdev doc
  * unit test stubs for device unit tests

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 MAINTAINERS                                 |  7 +-
 app/test/Makefile                           |  1 +
 app/test/meson.build                        |  1 +
 app/test/test_ioat_rawdev.c                 | 22 +++++
 config/common_armv8a_linux                  |  1 +
 config/common_base                          |  5 ++
 config/defconfig_arm-armv7a-linuxapp-gcc    |  1 +
 config/defconfig_ppc_64-power8-linuxapp-gcc |  1 +
 doc/guides/rawdevs/index.rst                |  1 +
 doc/guides/rawdevs/ioat_rawdev.rst          | 25 ++++++
 doc/guides/rel_notes/release_19_08.rst      | 11 +++
 drivers/raw/Makefile                        |  1 +
 drivers/raw/ioat/Makefile                   | 28 +++++++
 drivers/raw/ioat/ioat_rawdev.c              | 93 +++++++++++++++++++++
 drivers/raw/ioat/meson.build                |  8 ++
 drivers/raw/ioat/rte_ioat_rawdev.h          | 24 ++++++
 drivers/raw/ioat/rte_pmd_ioat_version.map   |  4 +
 drivers/raw/meson.build                     |  3 +-
 mk/rte.app.mk                               |  1 +
 19 files changed, 236 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_ioat_rawdev.c
 create mode 100644 doc/guides/rawdevs/ioat_rawdev.rst
 create mode 100644 drivers/raw/ioat/Makefile
 create mode 100644 drivers/raw/ioat/ioat_rawdev.c
 create mode 100644 drivers/raw/ioat/meson.build
 create mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h
 create mode 100644 drivers/raw/ioat/rte_pmd_ioat_version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 15d0829c5..b613a1e74 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1042,6 +1042,12 @@ M: Tianfei zhang <tianfei.zhang@intel.com>
 F: drivers/raw/ifpga_rawdev/
 F: doc/guides/rawdevs/ifpga_rawdev.rst
 
+IOAT Rawdev
+M: Bruce Richardson <bruce.richardson@intel.com>
+F: drivers/raw/ioat/
+F: doc/guides/rawdevs/ioat_rawdev.rst
+F: app/test/test_ioat_rawdev.c
+
 NXP DPAA2 QDMA
 M: Nipun Gupta <nipun.gupta@nxp.com>
 F: drivers/raw/dpaa2_qdma/
@@ -1052,7 +1058,6 @@ M: Nipun Gupta <nipun.gupta@nxp.com>
 F: drivers/raw/dpaa2_cmdif/
 F: doc/guides/rawdevs/dpaa2_cmdif.rst
 
-
 Packet processing
 -----------------
 
diff --git a/app/test/Makefile b/app/test/Makefile
index 68d6b4fbc..7fbdd0755 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -212,6 +212,7 @@ endif
 
 ifeq ($(CONFIG_RTE_LIBRTE_RAWDEV),y)
 SRCS-y += test_rawdev.c
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += test_ioat_rawdev.c
 endif
 
 SRCS-$(CONFIG_RTE_LIBRTE_KVARGS) += test_kvargs.c
diff --git a/app/test/meson.build b/app/test/meson.build
index 83391cef0..9867619d3 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -52,6 +52,7 @@ test_sources = files('commands.c',
 	'test_hash_perf.c',
 	'test_hash_readwrite_lf.c',
 	'test_interrupts.c',
+	'test_ioat_rawdev.c',
 	'test_ipsec.c',
 	'test_kni.c',
 	'test_kvargs.c',
diff --git a/app/test/test_ioat_rawdev.c b/app/test/test_ioat_rawdev.c
new file mode 100644
index 000000000..bd1bb2827
--- /dev/null
+++ b/app/test/test_ioat_rawdev.c
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include "test.h"
+
+#ifndef RTE_LIBRTE_PMD_IOAT_RAWDEV
+
+static int
+test_ioat_rawdev(void) { return TEST_SKIPPED; }
+
+#else
+
+static int
+test_ioat_rawdev(void)
+{
+	return 0;
+}
+
+#endif /* RTE_LIBRTE_PMD_IOAT_RAWDEV */
+
+REGISTER_TEST_COMMAND(ioat_rawdev_autotest, test_ioat_rawdev);
diff --git a/config/common_armv8a_linux b/config/common_armv8a_linux
index 72091de1c..481712ebc 100644
--- a/config/common_armv8a_linux
+++ b/config/common_armv8a_linux
@@ -34,5 +34,6 @@ CONFIG_RTE_ARCH_ARM64_MEMCPY=n
 CONFIG_RTE_LIBRTE_FM10K_PMD=n
 CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
 
 CONFIG_RTE_SCHED_VECTOR=n
diff --git a/config/common_base b/config/common_base
index 6f19ad5d2..2b8db4880 100644
--- a/config/common_base
+++ b/config/common_base
@@ -741,6 +741,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n
 #
 CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
 
+#
+# Compile PMD for Intel IOAT raw device
+#
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=y
+
 #
 # Compile librte_ring
 #
diff --git a/config/defconfig_arm-armv7a-linuxapp-gcc b/config/defconfig_arm-armv7a-linuxapp-gcc
index c9509b274..ee158ef9d 100644
--- a/config/defconfig_arm-armv7a-linuxapp-gcc
+++ b/config/defconfig_arm-armv7a-linuxapp-gcc
@@ -54,3 +54,4 @@ CONFIG_RTE_LIBRTE_QEDE_PMD=n
 CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
 CONFIG_RTE_LIBRTE_NFP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
diff --git a/config/defconfig_ppc_64-power8-linuxapp-gcc b/config/defconfig_ppc_64-power8-linuxapp-gcc
index 7e248b755..9f3670ec0 100644
--- a/config/defconfig_ppc_64-power8-linuxapp-gcc
+++ b/config/defconfig_ppc_64-power8-linuxapp-gcc
@@ -56,3 +56,4 @@ CONFIG_RTE_LIBRTE_ENIC_PMD=n
 CONFIG_RTE_LIBRTE_FM10K_PMD=n
 CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
diff --git a/doc/guides/rawdevs/index.rst b/doc/guides/rawdevs/index.rst
index 7c3bd9586..0a21989e4 100644
--- a/doc/guides/rawdevs/index.rst
+++ b/doc/guides/rawdevs/index.rst
@@ -14,3 +14,4 @@ application through rawdev API.
     dpaa2_cmdif
     dpaa2_qdma
     ifpga_rawdev
+    ioat_rawdev
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
new file mode 100644
index 000000000..40ab1b466
--- /dev/null
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -0,0 +1,25 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+.. include:: <isonum.txt>
+
+IOAT Rawdev Driver for Intel\ |reg| QuickData Technology
+======================================================================
+
+The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg|
+QuickData Technology, part of Intel\ |reg| I/O Acceleration Technology
+`(Intel I/OAT) <https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_.
+This PMD allows data copies, for example, cloning packet data, to be
+accelerated by hardware rather than having to be done by software, freeing
+up CPU cycles for other tasks.
+
+Compilation
+------------
+
+For builds done with ``make``, the driver compilation is enabled by the
+``CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV`` build configuration option. This is
+enabled by default in builds for x86 platforms, and disabled in other
+configurations.
+
+For builds using ``meson`` and ``ninja``, the driver will be built when the
+target platform is x86-based.
diff --git a/doc/guides/rel_notes/release_19_08.rst b/doc/guides/rel_notes/release_19_08.rst
index a17e7dea5..a65f3514a 100644
--- a/doc/guides/rel_notes/release_19_08.rst
+++ b/doc/guides/rel_notes/release_19_08.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright 2019 The DPDK contributors
 
+.. include:: <isonum.txt>
+
 DPDK Release 19.08
 ==================
 
@@ -54,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **Added Intel QuickData Technology PMD**
+
+  The PMD for Intel\ |reg|  QuickData Technology, part of
+  Intel\ |reg|  I/O Acceleration Technology `(Intel I/OAT)
+  <https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_,
+  allows data copies to be done by hardware instead
+  of via software, reducing cycles spent copying large blocks of data in
+  applications.
+
 
 Removed Items
 -------------
diff --git a/drivers/raw/Makefile b/drivers/raw/Makefile
index 8e29b4a56..c1b85c8c7 100644
--- a/drivers/raw/Makefile
+++ b/drivers/raw/Makefile
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev
+DIRS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat
 
 include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
new file mode 100644
index 000000000..7726e310a
--- /dev/null
+++ b/drivers/raw/ioat/Makefile
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_pmd_ioat_rawdev.a
+
+# build flags
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+
+LDLIBS += -lrte_eal -lrte_rawdev
+LDLIBS += -lrte_pci -lrte_bus_pci
+
+# library version
+LIBABIVER := 1
+
+# versioning export map
+EXPORT_MAP := rte_pmd_ioat_version.map
+
+# library source files
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
+
+# export include files
+SYMLINK-y-include += rte_ioat_rawdev.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
new file mode 100644
index 000000000..d9fc3091a
--- /dev/null
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -0,0 +1,93 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include <rte_bus_pci.h>
+#include <rte_rawdev_pmd.h>
+
+#include "rte_ioat_rawdev.h"
+
+/* Dynamic log type identifier */
+int ioat_pmd_logtype;
+
+static struct rte_pci_driver ioat_pmd_drv;
+
+#define IOAT_VENDOR_ID		0x8086
+#define IOAT_DEVICE_ID		0x2021
+
+#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \
+	ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args)
+
+#define IOAT_PMD_DEBUG(fmt, args...)  IOAT_PMD_LOG(DEBUG, fmt, ## args)
+#define IOAT_PMD_INFO(fmt, args...)   IOAT_PMD_LOG(INFO, fmt, ## args)
+#define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
+#define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
+
+static int
+ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
+{
+	RTE_SET_USED(name);
+	RTE_SET_USED(dev);
+	return 0;
+}
+
+static int
+ioat_rawdev_destroy(const char *name)
+{
+	RTE_SET_USED(name);
+	return 0;
+}
+
+static int
+ioat_rawdev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev)
+{
+	char name[32];
+	int ret = 0;
+
+
+	rte_pci_device_name(&dev->addr, name, sizeof(name));
+	IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node);
+
+	dev->device.driver = &drv->driver;
+	ret = ioat_rawdev_create(name, dev);
+	return ret;
+}
+
+static int
+ioat_rawdev_remove(struct rte_pci_device *dev)
+{
+	char name[32];
+	int ret;
+
+	rte_pci_device_name(&dev->addr, name, sizeof(name));
+
+	IOAT_PMD_INFO("Closing %s on NUMA node %d",
+			name, dev->device.numa_node);
+
+	ret = ioat_rawdev_destroy(name);
+	return ret;
+}
+
+static const struct rte_pci_id pci_id_ioat_map[] = {
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID) },
+	{ .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver ioat_pmd_drv = {
+	.id_table = pci_id_ioat_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+		     RTE_PCI_DRV_IOVA_AS_VA,
+	.probe = ioat_rawdev_probe,
+	.remove = ioat_rawdev_remove,
+};
+
+RTE_PMD_REGISTER_PCI(IOAT_PMD_RAWDEV_NAME, ioat_pmd_drv);
+RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_RAWDEV_NAME, pci_id_ioat_map);
+RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_RAWDEV_NAME, "* igb_uio | uio_pci_generic");
+
+RTE_INIT(ioat_pmd_init_log)
+{
+	ioat_pmd_logtype = rte_log_register(IOAT_PMD_LOG_NAME);
+	if (ioat_pmd_logtype >= 0)
+		rte_log_set_level(ioat_pmd_logtype, RTE_LOG_INFO);
+}
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
new file mode 100644
index 000000000..ba7620a68
--- /dev/null
+++ b/drivers/raw/ioat/meson.build
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2019 Intel Corporation
+
+build = dpdk_conf.has('RTE_ARCH_X86')
+sources = files('ioat_rawdev.c')
+deps += ['rawdev', 'bus_pci']
+
+install_headers('rte_ioat_rawdev.h')
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
new file mode 100644
index 000000000..e77406403
--- /dev/null
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_IOAT_RAWDEV_H_
+#define _RTE_IOAT_RAWDEV_H_
+
+/**
+ * @file rte_ioat_rawdev.h
+ *
+ * Definitions for using the ioat rawdev device driver
+ *
+ * @warning
+ * @b EXPERIMENTAL: these structures and APIs may change without prior notice
+ */
+
+/** Name of the device driver */
+#define IOAT_PMD_RAWDEV_NAME rawdev_ioat
+/** String reported as the device driver name by rte_rawdev_info_get() */
+#define IOAT_PMD_RAWDEV_NAME_STR "rawdev_ioat"
+/** Name used to adjust the log level for this driver */
+#define IOAT_PMD_LOG_NAME "rawdev.ioat"
+
+#endif
diff --git a/drivers/raw/ioat/rte_pmd_ioat_version.map b/drivers/raw/ioat/rte_pmd_ioat_version.map
new file mode 100644
index 000000000..9a61188cd
--- /dev/null
+++ b/drivers/raw/ioat/rte_pmd_ioat_version.map
@@ -0,0 +1,4 @@
+DPDK_19.08 {
+
+	local: *;
+};
diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build
index a61cdccef..39eb54a90 100644
--- a/drivers/raw/meson.build
+++ b/drivers/raw/meson.build
@@ -1,7 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2018 NXP
 
-drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
+drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma',
+	'ifpga_rawdev', 'ioat']
 std_deps = ['rawdev']
 config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
 driver_name_fmt = 'rte_pmd_@0@'
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 7c9b4b538..9232877b0 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -303,6 +303,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV)   += -lrte_pmd_ifpga_rawdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD)       += -lrte_pmd_ipn3ke
 endif # CONFIG_RTE_LIBRTE_IFPGA_BUS
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV)   += -lrte_pmd_ioat_rawdev
 endif # CONFIG_RTE_LIBRTE_RAWDEV
 
 endif # !CONFIG_RTE_BUILD_SHARED_LIBS
-- 
2.21.0

[dpdk-dev] [PATCH 2/8] usertools/dpdk-devbind.py: add support for IOAT devices

[Bruce Richardson]

In order to allow binding/unbinding of devices for use by the
ioat_rawdev, we need to update the devbind script to add a new class
of device, and add device ids for the specific HW instances.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/rawdevs/ioat_rawdev.rst | 11 +++++++++++
 usertools/dpdk-devbind.py          | 10 ++++++++++
 2 files changed, 21 insertions(+)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 40ab1b466..99e757498 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -23,3 +23,14 @@ configurations.
 
 For builds using ``meson`` and ``ninja``, the driver will be built when the
 target platform is x86-based.
+
+Device Setup
+-------------
+
+The Intel\ |reg| QuickData Technology HW devices will need to be bound to a
+user-space IO driver for use. The script ``dpdk-devbind.py`` script
+included with DPDK can be used to view the state of the devices and to bind
+them to a suitable DPDK-supported kernel driver. When querying the
+status of the devices, they will appear under the category of "dma
+devices", i.e. the command ``dpdk-devbind.py --status-dev dma`` can be used
+to see the state of those devices alone.
diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py
index 9e79f0d28..bd0d97df3 100755
--- a/usertools/dpdk-devbind.py
+++ b/usertools/dpdk-devbind.py
@@ -36,11 +36,17 @@
 octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc',
               'SVendor': None, 'SDevice': None}
 
+intel_ioat_bdw = {'Class': '08', 'Vendor': '8086', 'Device': '6f20,6f21,6f22,6f23,6f24,6f25,6f26,6f27,6f2e,6f2f',
+              'SVendor': None, 'SDevice': None}
+intel_ioat_skx = {'Class': '08', 'Vendor': '8086', 'Device': '2021',
+              'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 crypto_devices = [encryption_class, intel_processor_class]
 eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso]
 mempool_devices = [cavium_fpa, octeontx2_npa]
 compress_devices = [cavium_zip]
+dma_devices = [intel_ioat_bdw, intel_ioat_skx]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
@@ -595,6 +601,8 @@ def show_status():
     if status_dev == "compress" or status_dev == "all":
         show_device_status(compress_devices , "Compress")
 
+    if status_dev == "dma" or status_dev == "all":
+        show_device_status(dma_devices, "DMA")
 
 def parse_args():
     '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +678,7 @@ def do_arg_actions():
             get_device_details(eventdev_devices)
             get_device_details(mempool_devices)
             get_device_details(compress_devices)
+            get_device_details(dma_devices)
         show_status()
 
 
@@ -690,6 +699,7 @@ def main():
     get_device_details(eventdev_devices)
     get_device_details(mempool_devices)
     get_device_details(compress_devices)
+    get_device_details(dma_devices)
     do_arg_actions()
 
 if __name__ == "__main__":
-- 
2.21.0

[dpdk-dev] [PATCH 3/8] raw/ioat: add register definition file

[Bruce Richardson]

Add in the list of registers for the device. File is taken from the SPDK
project:

  https://github.com/spdk/spdk/blob/master/include/spdk/ioat_spec.h

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 drivers/raw/ioat/Makefile        |   1 +
 drivers/raw/ioat/meson.build     |   3 +-
 drivers/raw/ioat/rte_ioat_spec.h | 301 +++++++++++++++++++++++++++++++
 3 files changed, 304 insertions(+), 1 deletion(-)
 create mode 100644 drivers/raw/ioat/rte_ioat_spec.h

diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
index 7726e310a..1e10938f3 100644
--- a/drivers/raw/ioat/Makefile
+++ b/drivers/raw/ioat/Makefile
@@ -24,5 +24,6 @@ SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
 
 # export include files
 SYMLINK-y-include += rte_ioat_rawdev.h
+SYMLINK-y-include += rte_ioat_spec.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
index ba7620a68..ca23e23fc 100644
--- a/drivers/raw/ioat/meson.build
+++ b/drivers/raw/ioat/meson.build
@@ -5,4 +5,5 @@ build = dpdk_conf.has('RTE_ARCH_X86')
 sources = files('ioat_rawdev.c')
 deps += ['rawdev', 'bus_pci']
 
-install_headers('rte_ioat_rawdev.h')
+install_headers('rte_ioat_rawdev.h',
+		'rte_ioat_spec.h')
diff --git a/drivers/raw/ioat/rte_ioat_spec.h b/drivers/raw/ioat/rte_ioat_spec.h
new file mode 100644
index 000000000..305e36ded
--- /dev/null
+++ b/drivers/raw/ioat/rte_ioat_spec.h
@@ -0,0 +1,301 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) Intel Corporation
+ */
+
+/**
+ * \file
+ * I/OAT specification definitions
+ *
+ * Taken from ioat_spec.h from SPDK project, with prefix renames and
+ * other minor changes.
+ */
+
+#ifndef RTE_IOAT_SPEC_H
+#define RTE_IOAT_SPEC_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+#define RTE_IOAT_PCI_CHANERR_INT_OFFSET	0x180
+
+#define RTE_IOAT_INTRCTRL_MASTER_INT_EN	0x01
+
+#define RTE_IOAT_VER_3_0                0x30
+#define RTE_IOAT_VER_3_3                0x33
+
+/* DMA Channel Registers */
+#define RTE_IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK	0xF000
+#define RTE_IOAT_CHANCTRL_COMPL_DCA_EN		0x0200
+#define RTE_IOAT_CHANCTRL_CHANNEL_IN_USE		0x0100
+#define RTE_IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL	0x0020
+#define RTE_IOAT_CHANCTRL_ERR_INT_EN		0x0010
+#define RTE_IOAT_CHANCTRL_ANY_ERR_ABORT_EN		0x0008
+#define RTE_IOAT_CHANCTRL_ERR_COMPLETION_EN		0x0004
+#define RTE_IOAT_CHANCTRL_INT_REARM			0x0001
+
+/* DMA Channel Capabilities */
+#define	RTE_IOAT_DMACAP_PB		(1 << 0)
+#define	RTE_IOAT_DMACAP_DCA		(1 << 4)
+#define	RTE_IOAT_DMACAP_BFILL		(1 << 6)
+#define	RTE_IOAT_DMACAP_XOR		(1 << 8)
+#define	RTE_IOAT_DMACAP_PQ		(1 << 9)
+#define	RTE_IOAT_DMACAP_DMA_DIF	(1 << 10)
+
+struct rte_ioat_registers {
+	uint8_t		chancnt;
+	uint8_t		xfercap;
+	uint8_t		genctrl;
+	uint8_t		intrctrl;
+	uint32_t	attnstatus;
+	uint8_t		cbver;		/* 0x08 */
+	uint8_t		reserved4[0x3]; /* 0x09 */
+	uint16_t	intrdelay;	/* 0x0C */
+	uint16_t	cs_status;	/* 0x0E */
+	uint32_t	dmacapability;	/* 0x10 */
+	uint8_t		reserved5[0x6C]; /* 0x14 */
+	uint16_t	chanctrl;	/* 0x80 */
+	uint8_t		reserved6[0x2];	/* 0x82 */
+	uint8_t		chancmd;	/* 0x84 */
+	uint8_t		reserved3[1];	/* 0x85 */
+	uint16_t	dmacount;	/* 0x86 */
+	uint64_t	chansts;	/* 0x88 */
+	uint64_t	chainaddr;	/* 0x90 */
+	uint64_t	chancmp;	/* 0x98 */
+	uint8_t		reserved2[0x8];	/* 0xA0 */
+	uint32_t	chanerr;	/* 0xA8 */
+	uint32_t	chanerrmask;	/* 0xAC */
+} __attribute__((packed));
+
+#define RTE_IOAT_CHANCMD_RESET			0x20
+#define RTE_IOAT_CHANCMD_SUSPEND		0x04
+
+#define RTE_IOAT_CHANSTS_STATUS		0x7ULL
+#define RTE_IOAT_CHANSTS_ACTIVE		0x0
+#define RTE_IOAT_CHANSTS_IDLE			0x1
+#define RTE_IOAT_CHANSTS_SUSPENDED		0x2
+#define RTE_IOAT_CHANSTS_HALTED		0x3
+#define RTE_IOAT_CHANSTS_ARMED			0x4
+
+#define RTE_IOAT_CHANSTS_UNAFFILIATED_ERROR	0x8ULL
+#define RTE_IOAT_CHANSTS_SOFT_ERROR		0x10ULL
+
+#define RTE_IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK	(~0x3FULL)
+
+#define RTE_IOAT_CHANCMP_ALIGN			8	/* CHANCMP address must be 64-bit aligned */
+
+struct rte_ioat_generic_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t reserved2: 1;
+			uint32_t src_page_break: 1;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t op_specific[4];
+};
+
+struct rte_ioat_dma_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t null: 1;
+			uint32_t src_page_break: 1;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+#define RTE_IOAT_OP_COPY 0x00
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t reserved;
+	uint64_t reserved2;
+	uint64_t user1;
+	uint64_t user2;
+};
+
+struct rte_ioat_fill_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t reserved: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t reserved2: 2;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t reserved3: 15;
+#define RTE_IOAT_OP_FILL 0x01
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_data;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t reserved;
+	uint64_t next_dest_addr;
+	uint64_t user1;
+	uint64_t user2;
+};
+
+struct rte_ioat_xor_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_count: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+#define RTE_IOAT_OP_XOR 0x87
+#define RTE_IOAT_OP_XOR_VAL 0x88
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t src_addr3;
+	uint64_t src_addr4;
+	uint64_t src_addr5;
+};
+
+struct rte_ioat_xor_ext_hw_desc {
+	uint64_t src_addr6;
+	uint64_t src_addr7;
+	uint64_t src_addr8;
+	uint64_t next;
+	uint64_t reserved[4];
+};
+
+struct rte_ioat_pq_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_count: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t p_disable: 1;
+			uint32_t q_disable: 1;
+			uint32_t reserved: 11;
+#define RTE_IOAT_OP_PQ 0x89
+#define RTE_IOAT_OP_PQ_VAL 0x8a
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t p_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t src_addr3;
+	uint8_t  coef[8];
+	uint64_t q_addr;
+};
+
+struct rte_ioat_pq_ext_hw_desc {
+	uint64_t src_addr4;
+	uint64_t src_addr5;
+	uint64_t src_addr6;
+	uint64_t next;
+	uint64_t src_addr7;
+	uint64_t src_addr8;
+	uint64_t reserved[2];
+};
+
+struct rte_ioat_pq_update_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_cnt: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t p_disable: 1;
+			uint32_t q_disable: 1;
+			uint32_t reserved: 3;
+			uint32_t coef: 8;
+#define RTE_IOAT_OP_PQ_UP 0x8b
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t p_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t p_src;
+	uint64_t q_src;
+	uint64_t q_addr;
+};
+
+struct rte_ioat_raw_hw_desc {
+	uint64_t field[8];
+};
+
+union rte_ioat_hw_desc {
+	struct rte_ioat_raw_hw_desc raw;
+	struct rte_ioat_generic_hw_desc generic;
+	struct rte_ioat_dma_hw_desc dma;
+	struct rte_ioat_fill_hw_desc fill;
+	struct rte_ioat_xor_hw_desc xor_desc;
+	struct rte_ioat_xor_ext_hw_desc xor_ext;
+	struct rte_ioat_pq_hw_desc pq;
+	struct rte_ioat_pq_ext_hw_desc pq_ext;
+	struct rte_ioat_pq_update_hw_desc pq_update;
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_IOAT_SPEC_H */
-- 
2.21.0

Re: [dpdk-dev] [PATCH 3/8] raw/ioat: add register definition file

[Stephen Hemminger]

On Thu, 30 May 2019 22:25:20 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +

This a driver private file, why the C++ guard here?

Re: [dpdk-dev] [PATCH 3/8] raw/ioat: add register definition file

[Bruce Richardson]

On Thu, May 30, 2019 at 04:53:05PM -0700, Stephen Hemminger wrote:
> On Thu, 30 May 2019 22:25:20 +0100
> Bruce Richardson <bruce.richardson@intel.com> wrote:
> 
> > +
> > +#ifdef __cplusplus
> > +extern "C" {
> > +#endif
> > +
> 
> This a driver private file, why the C++ guard here?

It's not just a private file, it's included in the regular driver .h file,
as the inline functions there need access to the register definitions.

As for whether the functions need to be inline, most of them are inline for
performance reasons, and the one that I would consider suitable for moving
to the driver is inline also so that the user does not need to link their
application against the PMD in order to use the device. Having the header
at compile-time, and then the driver loaded at runtime is sufficient.

/Bruce

[dpdk-dev] [PATCH 4/8] raw/ioat: create device on probe and destroy on release

[Bruce Richardson]

Add the create/destroy driver functions so that we can actually allocate
a rawdev and destroy it when done. No rawdev API functions are actually
implemented at this point.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/rawdevs/ioat_rawdev.rst | 11 ++++
 drivers/raw/ioat/ioat_rawdev.c     | 93 +++++++++++++++++++++++++++++-
 drivers/raw/ioat/rte_ioat_rawdev.h | 20 +++++++
 3 files changed, 121 insertions(+), 3 deletions(-)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 99e757498..476b0503f 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -34,3 +34,14 @@ them to a suitable DPDK-supported kernel driver. When querying the
 status of the devices, they will appear under the category of "dma
 devices", i.e. the command ``dpdk-devbind.py --status-dev dma`` can be used
 to see the state of those devices alone.
+
+Device Probing and Initialization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once bound to a suitable kernel device driver, the HW devices will be found
+as part of the PCI scan done at application initialization time. No vdev
+parameters need to be passed to create or initialize the device.
+
+Once probed successfully, the device will appear as a ``rawdev``, that is a
+"raw device type" inside DPDK, and can be accessed using APIs from the
+``rte_rawdev`` library.
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index d9fc3091a..b6964bccd 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -2,6 +2,7 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <rte_cycles.h>
 #include <rte_bus_pci.h>
 #include <rte_rawdev_pmd.h>
 
@@ -26,15 +27,101 @@ static struct rte_pci_driver ioat_pmd_drv;
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
-	RTE_SET_USED(name);
-	RTE_SET_USED(dev);
+	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+	};
+
+	struct rte_rawdev *rawdev = NULL;
+	struct rte_ioat_rawdev *ioat = NULL;
+	int ret = 0;
+	int retry = 0;
+
+	if (!name) {
+		IOAT_PMD_ERR("Invalid name of the device!");
+		ret = -EINVAL;
+		goto cleanup;
+	}
+
+	/* Allocate device structure */
+	rawdev = rte_rawdev_pmd_allocate(name, sizeof(struct rte_ioat_rawdev),
+					 dev->device.numa_node);
+	if (rawdev == NULL) {
+		IOAT_PMD_ERR("Unable to allocate raw device");
+		ret = -EINVAL;
+		goto cleanup;
+	}
+
+	rawdev->dev_ops = &ioat_rawdev_ops;
+	rawdev->device = &dev->device;
+	rawdev->driver_name = dev->device.driver->name;
+
+	ioat = rawdev->dev_private;
+	ioat->rawdev = rawdev;
+	ioat->regs = dev->mem_resource[0].addr;
+	ioat->ring_size = 0;
+	ioat->desc_ring = NULL;
+	ioat->status_addr = rte_malloc_virt2iova(ioat) +
+			offsetof(struct rte_ioat_rawdev, status);
+
+	/* do device initialization - reset and set error behaviour */
+	if (ioat->regs->chancnt != 1)
+		IOAT_PMD_ERR("%s: Channel count == %d\n", __func__,
+				ioat->regs->chancnt);
+
+	if (ioat->regs->chanctrl & 0x100) { /* locked by someone else */
+		IOAT_PMD_WARN("%s: Channel appears locked\n", __func__);
+		ioat->regs->chanctrl = 0;
+	}
+
+	ioat->regs->chancmd = RTE_IOAT_CHANCMD_SUSPEND;
+	rte_delay_ms(1);
+	ioat->regs->chancmd = RTE_IOAT_CHANCMD_RESET;
+	rte_delay_ms(1);
+	while (ioat->regs->chancmd & RTE_IOAT_CHANCMD_RESET) {
+		ioat->regs->chainaddr = 0;
+		rte_delay_ms(1);
+		if (++retry >= 200) {
+			IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=0x%llx, CHANSTS=0x%llx, CHANERR=0x%llx\n",
+					__func__,
+					(unsigned long long)ioat->regs->chancmd,
+					(unsigned long long)ioat->regs->chansts,
+					(unsigned long long)ioat->regs->chanerr);
+			ret = -EIO;
+		}
+	}
+	ioat->regs->chanctrl = RTE_IOAT_CHANCTRL_ANY_ERR_ABORT_EN |
+			RTE_IOAT_CHANCTRL_ERR_COMPLETION_EN;
+
 	return 0;
+
+cleanup:
+	if (rawdev)
+		rte_rawdev_pmd_release(rawdev);
+
+	return ret;
 }
 
 static int
 ioat_rawdev_destroy(const char *name)
 {
-	RTE_SET_USED(name);
+	int ret;
+	struct rte_rawdev *rdev;
+
+	if (!name) {
+		IOAT_PMD_ERR("Invalid device name");
+		return -EINVAL;
+	}
+
+	rdev = rte_rawdev_pmd_get_named_dev(name);
+	if (!rdev) {
+		IOAT_PMD_ERR("Invalid device name (%s)", name);
+		return -EINVAL;
+	}
+
+	/* rte_rawdev_close is called by pmd_release */
+	ret = rte_rawdev_pmd_release(rdev);
+	if (ret)
+		IOAT_PMD_DEBUG("Device cleanup failed");
+
 	return 0;
 }
 
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index e77406403..c3216a174 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -14,6 +14,9 @@
  * @b EXPERIMENTAL: these structures and APIs may change without prior notice
  */
 
+#include <rte_memory.h>
+#include <rte_ioat_spec.h>
+
 /** Name of the device driver */
 #define IOAT_PMD_RAWDEV_NAME rawdev_ioat
 /** String reported as the device driver name by rte_rawdev_info_get() */
@@ -21,4 +24,21 @@
 /** Name used to adjust the log level for this driver */
 #define IOAT_PMD_LOG_NAME "rawdev.ioat"
 
+/**
+ * @internal
+ * Structure representing a device instance
+ */
+struct rte_ioat_rawdev {
+	struct rte_rawdev *rawdev;
+	volatile struct rte_ioat_registers *regs;
+	phys_addr_t status_addr;
+	phys_addr_t ring_addr;
+
+	unsigned short ring_size;
+	struct rte_ioat_desc *desc_ring;
+
+	/* to report completions, the device will write status back here */
+	volatile uint64_t status __rte_cache_aligned;
+};
+
 #endif
-- 
2.21.0

[dpdk-dev] [PATCH 5/8] raw/ioat: add device info function

[Bruce Richardson]

Add in the "info_get" function to the driver, to allow us to query the
device. This allows us to have the unit test pick up the presence of
supported hardware or not.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/meson.build               |  3 +++
 app/test/test_ioat_rawdev.c        | 23 ++++++++++++++++++++
 doc/guides/rawdevs/ioat_rawdev.rst | 34 ++++++++++++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev.c     | 11 ++++++++++
 drivers/raw/ioat/rte_ioat_rawdev.h | 11 ++++++++++
 5 files changed, 82 insertions(+)

diff --git a/app/test/meson.build b/app/test/meson.build
index 9867619d3..9fe3ddc89 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -305,6 +305,9 @@ endif
 if dpdk_conf.has('RTE_LIBRTE_KNI')
 	test_deps += 'kni'
 endif
+if dpdk_conf.has('RTE_LIBRTE_PMD_IOAT_RAWDEV')
+	test_deps += 'pmd_ioat'
+endif
 
 cflags = machine_args
 if cc.has_argument('-Wno-format-truncation')
diff --git a/app/test/test_ioat_rawdev.c b/app/test/test_ioat_rawdev.c
index bd1bb2827..ac1389f6e 100644
--- a/app/test/test_ioat_rawdev.c
+++ b/app/test/test_ioat_rawdev.c
@@ -11,9 +11,32 @@ test_ioat_rawdev(void) { return TEST_SKIPPED; }
 
 #else
 
+#include <string.h>
+#include <unistd.h>
+
+#include <rte_mbuf.h>
+#include <rte_rawdev.h>
+#include <rte_ioat_rawdev.h>
+
 static int
 test_ioat_rawdev(void)
 {
+	const int count = rte_rawdev_count();
+	int i, found = 0;
+
+	printf("Checking %d rawdevs\n", count);
+	for (i = 0; i < count && !found; i++) {
+		struct rte_rawdev_info info = { .dev_private = NULL };
+		found = (rte_rawdev_info_get(i, &info) == 0 &&
+				strcmp(info.driver_name,
+						IOAT_PMD_RAWDEV_NAME_STR) == 0);
+	}
+
+	if (!found) {
+		printf("No IOAT rawdev found, skipping tests\n");
+		return TEST_SKIPPED;
+	}
+
 	return 0;
 }
 
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 476b0503f..b68cdffc3 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -45,3 +45,37 @@ parameters need to be passed to create or initialize the device.
 Once probed successfully, the device will appear as a ``rawdev``, that is a
 "raw device type" inside DPDK, and can be accessed using APIs from the
 ``rte_rawdev`` library.
+
+Using IOAT Rawdev Devices
+--------------------------
+
+To use the devices from an application, the rawdev API can be used, along
+with definitions taken from the device-specific header file
+``rte_ioat_rawdev.h``. This header is needed to get the definition of
+structure parameters used by some of the rawdev APIs for IOAT rawdev
+devices, as well as providing key functions for using the device for memory
+copies.
+
+Getting Device Information
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Basic information about each rawdev device can be got using the
+``rte_rawdev_info_get()`` API. For most applications, this API will be
+needed to verify that the rawdev in question is of the expected type. For
+example, the following code in ``test_ioat_rawdev.c`` is used to identify
+the IOAT rawdev device for use for the tests:
+
+.. code-block:: C
+
+        for (i = 0; i < count && !found; i++) {
+                struct rte_rawdev_info info = { .dev_private = NULL };
+                found = (rte_rawdev_info_get(i, &info) == 0 &&
+                                strcmp(info.driver_name,
+                                                IOAT_PMD_RAWDEV_NAME_STR) == 0);
+        }
+
+When calling the ``rte_rawdev_info_get()`` API for an IOAT rawdev device,
+the ``dev_private`` field in the ``rte_rawdev_info`` struct should either
+be NULL, or else be set to point to a structure of type
+``rte_ioat_rawdev_config``, in which case the size of the configured device
+input ring will be returned in that structure.
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index b6964bccd..90bed2810 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -24,10 +24,21 @@ static struct rte_pci_driver ioat_pmd_drv;
 #define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
 #define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
 
+static void
+ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
+{
+	struct rte_ioat_rawdev_config *cfg = dev_info;
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+
+	if (cfg != NULL)
+		cfg->ring_size = ioat->ring_size;
+}
+
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
 	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+			.dev_info_get = ioat_dev_info_get,
 	};
 
 	struct rte_rawdev *rawdev = NULL;
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index c3216a174..7e0d72ca3 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -24,6 +24,17 @@
 /** Name used to adjust the log level for this driver */
 #define IOAT_PMD_LOG_NAME "rawdev.ioat"
 
+/**
+ * Configuration structure for an ioat rawdev instance
+ *
+ * This structure is to be passed as the ".dev_private" parameter when
+ * calling the rte_rawdev_get_info() and rte_rawdev_configure() APIs on
+ * an ioat rawdev instance.
+ */
+struct rte_ioat_rawdev_config {
+	unsigned short ring_size;
+};
+
 /**
  * @internal
  * Structure representing a device instance
-- 
2.21.0

[dpdk-dev] [PATCH 6/8] raw/ioat: add configure, start and stop functions

[Bruce Richardson]

Allow initializing a driver instance.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/test_ioat_rawdev.c        | 35 +++++++++++++-
 doc/guides/rawdevs/ioat_rawdev.rst | 32 +++++++++++++
 drivers/raw/ioat/ioat_rawdev.c     | 75 ++++++++++++++++++++++++++++++
 drivers/raw/ioat/rte_ioat_rawdev.h | 14 ++++++
 4 files changed, 155 insertions(+), 1 deletion(-)

diff --git a/app/test/test_ioat_rawdev.c b/app/test/test_ioat_rawdev.c
index ac1389f6e..36e97347c 100644
--- a/app/test/test_ioat_rawdev.c
+++ b/app/test/test_ioat_rawdev.c
@@ -18,6 +18,39 @@ test_ioat_rawdev(void) { return TEST_SKIPPED; }
 #include <rte_rawdev.h>
 #include <rte_ioat_rawdev.h>
 
+static int
+run_ioat_tests(int dev_id)
+{
+#define IOAT_TEST_RINGSIZE 512
+	struct rte_ioat_rawdev_config p = { .ring_size = -1 };
+	struct rte_rawdev_info info = { .dev_private = &p };
+
+	rte_rawdev_info_get(dev_id, &info);
+	if (p.ring_size != 0) {
+		printf("Error, initial ring size is non-zero (%d)\n",
+				(int)p.ring_size);
+		return -1;
+	}
+
+	p.ring_size = IOAT_TEST_RINGSIZE;
+	if (rte_rawdev_configure(dev_id, &info) != 0) {
+		printf("Error with rte_rawdev_configure()\n");
+		return -1;
+	}
+	rte_rawdev_info_get(dev_id, &info);
+	if (p.ring_size != IOAT_TEST_RINGSIZE) {
+		printf("Error, ring size is not %d (%d)\n",
+				IOAT_TEST_RINGSIZE, (int)p.ring_size);
+		return -1;
+	}
+
+	if (rte_rawdev_start(dev_id) != 0) {
+		printf("Error with rte_rawdev_start()\n");
+		return -1;
+	}
+	return 0;
+}
+
 static int
 test_ioat_rawdev(void)
 {
@@ -37,7 +70,7 @@ test_ioat_rawdev(void)
 		return TEST_SKIPPED;
 	}
 
-	return 0;
+	return run_ioat_tests(i);
 }
 
 #endif /* RTE_LIBRTE_PMD_IOAT_RAWDEV */
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index b68cdffc3..b3fe79033 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -79,3 +79,35 @@ the ``dev_private`` field in the ``rte_rawdev_info`` struct should either
 be NULL, or else be set to point to a structure of type
 ``rte_ioat_rawdev_config``, in which case the size of the configured device
 input ring will be returned in that structure.
+
+Device Configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+Configuring an IOAT rawdev device is done using the
+``rte_rawdev_configure()`` API, which takes the same structure parameters
+as the, previously referenced, ``rte_rawdev_info_get()`` API. The main
+difference is that, because the parameter is used as input rather than
+output, the ``dev_private`` structure element cannot be NULL, and must
+point to a valid ``rte_ioat_rawdev_config`` structure, containing the ring
+size to be used by the device. The ring size must be a power of two,
+between 64 and 4096.
+
+The following code shows how the device is configured in
+``test_ioat_rawdev.c``:
+
+.. code-block:: C
+
+   #define IOAT_TEST_RINGSIZE 512
+        struct rte_ioat_rawdev_config p = { .ring_size = -1 };
+        struct rte_rawdev_info info = { .dev_private = &p };
+
+        /* ... */
+
+        p.ring_size = IOAT_TEST_RINGSIZE;
+        if (rte_rawdev_configure(dev_id, &info) != 0) {
+                printf("Error with rte_rawdev_configure()\n");
+                return -1;
+        }
+
+Once configured, the device can then be made ready for use by calling the
+``rte_rawdev_start()`` API.
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index 90bed2810..b4b70a1e6 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -24,6 +24,78 @@ static struct rte_pci_driver ioat_pmd_drv;
 #define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
 #define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
 
+#define DESC_SZ sizeof(struct rte_ioat_desc)
+#define COMPLETION_SZ sizeof(__m128i)
+
+static int
+ioat_dev_configure(const struct rte_rawdev *dev, rte_rawdev_obj_t config)
+{
+	struct rte_ioat_rawdev_config *params = config;
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+	unsigned short i;
+
+	if (dev->started)
+		return -EBUSY;
+
+	if (params == NULL)
+		return -EINVAL;
+
+	if (params->ring_size > 4096 || params->ring_size < 64 ||
+			!rte_is_power_of_2(params->ring_size))
+		return -EINVAL;
+
+	ioat->ring_size = params->ring_size;
+	if (ioat->desc_ring != NULL) {
+		rte_free(ioat->desc_ring);
+		ioat->desc_ring = NULL;
+	}
+
+	/* allocate one block of memory for both descriptors
+	 * and completion handles.
+	 */
+	ioat->desc_ring = rte_zmalloc_socket(NULL,
+			(DESC_SZ + COMPLETION_SZ) * ioat->ring_size,
+			0, /* alignment, default to 64Byte */
+			dev->device->numa_node);
+	if (ioat->desc_ring == NULL)
+		return -ENOMEM;
+	ioat->hdls = (void *)&ioat->desc_ring[ioat->ring_size];
+
+	ioat->ring_addr = rte_malloc_virt2iova(ioat->desc_ring);
+
+	/* configure descriptor ring - each one points to next */
+	for (i = 0; i < ioat->ring_size; i++) {
+		ioat->desc_ring[i].next_desc_addr = ioat->ring_addr +
+				(((i + 1) % ioat->ring_size) * DESC_SZ);
+	}
+
+	return 0;
+}
+
+static int
+ioat_dev_start(struct rte_rawdev *dev)
+{
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+
+	if (ioat->ring_size == 0 || ioat->desc_ring == NULL)
+		return -EBUSY;
+
+	/* inform hardware of where the descriptor ring is */
+	ioat->regs->chainaddr = ioat->ring_addr;
+	/* inform hardware of where to write the status/completions */
+	ioat->regs->chancmp = ioat->status_addr;
+
+	/* prime the status register to be set to the last element */
+	ioat->status =  ioat->ring_addr + ((ioat->ring_size - 1) * DESC_SZ);
+	return 0;
+}
+
+static void
+ioat_dev_stop(struct rte_rawdev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
 static void
 ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
 {
@@ -38,6 +110,9 @@ static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
 	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+			.dev_configure = ioat_dev_configure,
+			.dev_start = ioat_dev_start,
+			.dev_stop = ioat_dev_stop,
 			.dev_info_get = ioat_dev_info_get,
 	};
 
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index 7e0d72ca3..7eab216c8 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -47,9 +47,23 @@ struct rte_ioat_rawdev {
 
 	unsigned short ring_size;
 	struct rte_ioat_desc *desc_ring;
+	__m128i *hdls; /* completion handles for returning to user */
 
 	/* to report completions, the device will write status back here */
 	volatile uint64_t status __rte_cache_aligned;
 };
 
+/**
+ * @internal
+ * Structure representing a descriptor for a copy operation
+ */
+struct rte_ioat_desc {
+	uint32_t xfer_size __rte_cache_aligned;
+	uint32_t desc_control;
+	phys_addr_t src_addr; /* 64 bits */
+	phys_addr_t dest_addr; /* 64 bits */
+	phys_addr_t next_desc_addr; /* 64 bits */
+	uint64_t op_type_specific[4];
+};
+
 #endif
-- 
2.21.0

[dpdk-dev] [PATCH 7/8] raw/ioat: add statistics functions

[Bruce Richardson]

Add stats functions to track what is happening in the driver, and put
unit tests to check those.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/test_ioat_rawdev.c        | 38 ++++++++++++++++++++++++++
 doc/guides/rawdevs/ioat_rawdev.rst | 14 ++++++++++
 drivers/raw/ioat/ioat_rawdev.c     | 44 ++++++++++++++++++++++++++++++
 drivers/raw/ioat/rte_ioat_rawdev.h |  6 ++++
 4 files changed, 102 insertions(+)

diff --git a/app/test/test_ioat_rawdev.c b/app/test/test_ioat_rawdev.c
index 36e97347c..7081f3365 100644
--- a/app/test/test_ioat_rawdev.c
+++ b/app/test/test_ioat_rawdev.c
@@ -24,6 +24,11 @@ run_ioat_tests(int dev_id)
 #define IOAT_TEST_RINGSIZE 512
 	struct rte_ioat_rawdev_config p = { .ring_size = -1 };
 	struct rte_rawdev_info info = { .dev_private = &p };
+	struct rte_rawdev_xstats_name *snames = NULL;
+	uint64_t *stats = NULL;
+	unsigned int *ids = NULL;
+	unsigned int nb_xstats;
+	unsigned int i;
 
 	rte_rawdev_info_get(dev_id, &info);
 	if (p.ring_size != 0) {
@@ -48,6 +53,39 @@ run_ioat_tests(int dev_id)
 		printf("Error with rte_rawdev_start()\n");
 		return -1;
 	}
+
+	/* allocate memory for xstats names and values */
+	nb_xstats = rte_rawdev_xstats_names_get(dev_id, NULL, 0);
+
+	snames = malloc(sizeof(*snames) * nb_xstats);
+	if (snames == NULL) {
+		printf("Error allocating xstat names memory\n");
+		return -1;
+	}
+	rte_rawdev_xstats_names_get(dev_id, snames, nb_xstats);
+
+	ids = malloc(sizeof(*ids) * nb_xstats);
+	if (ids == NULL) {
+		printf("Error allocating xstat ids memory\n");
+		return -1;
+	}
+	for (i = 0; i < nb_xstats; i++)
+		ids[i] = i;
+
+	stats = malloc(sizeof(*stats) * nb_xstats);
+	if (stats == NULL) {
+		printf("Error allocating xstat memory\n");
+		return -1;
+	}
+
+	rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
+	for (i = 0; i < nb_xstats; i++)
+		printf("%s: %"PRIu64"   ", snames[i].name, stats[i]);
+	printf("\n");
+
+	free(snames);
+	free(stats);
+	free(ids);
 	return 0;
 }
 
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index b3fe79033..47f12e95c 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -111,3 +111,17 @@ The following code shows how the device is configured in
 
 Once configured, the device can then be made ready for use by calling the
 ``rte_rawdev_start()`` API.
+
+Querying Device Statistics
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The statistics from the IOAT rawdev device can be got via the xstats
+functions in the ``rte_rawdev`` library, i.e.
+``rte_rawdev_xstats_names_get()``, ``rte_rawdev_xstats_get()`` and
+``rte_rawdev_xstats_by_name_get``. The statistics returned for each device
+instance are:
+
+* ``failed_enqueues``
+* ``successful_enqueues``
+* ``copies_started``
+* ``copies_completed``
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index b4b70a1e6..09fbdbf9c 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -4,6 +4,7 @@
 
 #include <rte_cycles.h>
 #include <rte_bus_pci.h>
+#include <rte_string_fns.h>
 #include <rte_rawdev_pmd.h>
 
 #include "rte_ioat_rawdev.h"
@@ -106,6 +107,47 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
 		cfg->ring_size = ioat->ring_size;
 }
 
+static const char *xstat_names[] = {
+		"failed_enqueues", "successful_enqueues",
+		"copies_started", "copies_completed"
+};
+
+static int
+ioat_xstats_get(const struct rte_rawdev *dev, const unsigned int ids[],
+		uint64_t values[], unsigned int n)
+{
+	const struct rte_ioat_rawdev *ioat = dev->dev_private;
+	unsigned int i;
+
+	for (i = 0; i < n; i++) {
+		switch (ids[i]){
+		case 0: values[i] = ioat->enqueue_failed; break;
+		case 1: values[i] = ioat->enqueued; break;
+		case 2: values[i] = ioat->started; break;
+		case 3: values[i] = ioat->completed; break;
+		default: values[i] = 0; break;
+		}
+	}
+	return n;
+}
+
+static int
+ioat_xstats_get_names(const struct rte_rawdev *dev,
+		struct rte_rawdev_xstats_name *names,
+		unsigned int size)
+{
+	unsigned int i;
+
+	RTE_SET_USED(dev);
+	if (size < RTE_DIM(xstat_names))
+		return RTE_DIM(xstat_names);
+
+	for (i = 0; i < RTE_DIM(xstat_names); i++)
+		strlcpy(names[i].name, xstat_names[i], sizeof(names[i]));
+
+	return RTE_DIM(xstat_names);
+}
+
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
@@ -114,6 +156,8 @@ ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 			.dev_start = ioat_dev_start,
 			.dev_stop = ioat_dev_stop,
 			.dev_info_get = ioat_dev_info_get,
+			.xstats_get = ioat_xstats_get,
+			.xstats_get_names = ioat_xstats_get_names,
 	};
 
 	struct rte_rawdev *rawdev = NULL;
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index 7eab216c8..abe5ba298 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -49,6 +49,12 @@ struct rte_ioat_rawdev {
 	struct rte_ioat_desc *desc_ring;
 	__m128i *hdls; /* completion handles for returning to user */
 
+	/* some statistics for tracking, if added/changed update xstats fns*/
+	uint64_t enqueue_failed __rte_cache_aligned;
+	uint64_t enqueued;
+	uint64_t started;
+	uint64_t completed;
+
 	/* to report completions, the device will write status back here */
 	volatile uint64_t status __rte_cache_aligned;
 };
-- 
2.21.0

[dpdk-dev] [PATCH 8/8] raw/ioat: add local API to perform copies

[Bruce Richardson]

Add local APIs to trigger data copies, and retrieve handle values once
those copies are completed. Included are unit tests to validate the data
is copies correctly.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/test_ioat_rawdev.c        | 159 ++++++++++++++++++++++++++++-
 doc/guides/rawdevs/ioat_rawdev.rst | 100 ++++++++++++++++++
 drivers/raw/ioat/rte_ioat_rawdev.h | 155 +++++++++++++++++++++++++++-
 3 files changed, 410 insertions(+), 4 deletions(-)

diff --git a/app/test/test_ioat_rawdev.c b/app/test/test_ioat_rawdev.c
index 7081f3365..f2240adec 100644
--- a/app/test/test_ioat_rawdev.c
+++ b/app/test/test_ioat_rawdev.c
@@ -18,6 +18,131 @@ test_ioat_rawdev(void) { return TEST_SKIPPED; }
 #include <rte_rawdev.h>
 #include <rte_ioat_rawdev.h>
 
+static struct rte_mempool *pool;
+
+static int
+test_enqueue_copies(int dev_id)
+{
+	const unsigned int length = 1024;
+	unsigned int i;
+
+	do {
+		struct rte_mbuf *src, *dst;
+		char *src_data, *dst_data;
+		struct rte_mbuf *completed[2] = {0};
+
+		/* test doing a single copy */
+		src = rte_pktmbuf_alloc(pool);
+		dst = rte_pktmbuf_alloc(pool);
+		src->data_len = src->pkt_len = length;
+		dst->data_len = dst->pkt_len = length;
+		src_data = rte_pktmbuf_mtod(src, char *);
+		dst_data = rte_pktmbuf_mtod(dst, char *);
+
+		for (i = 0; i < length; i++)
+			src_data[i] = rand() & 0xFF;
+
+		if (rte_ioat_enqueue_copy(dev_id,
+				src->buf_iova + src->data_off,
+				dst->buf_iova + dst->data_off,
+				length,
+				(uintptr_t)src,
+				(uintptr_t)dst,
+				0 /* no fence */) != 1) {
+			printf("Error with rte_ioat_enqueue_copy\n");
+			return -1;
+		}
+		rte_ioat_do_copies(dev_id);
+		usleep(10);
+
+		if (rte_ioat_completed_copies(dev_id, 1, (void *)&completed[0],
+				(void *)&completed[1]) != 1) {
+			printf("Error with rte_ioat_completed_copies\n");
+			return -1;
+		}
+		if (completed[0] != src || completed[1] != dst) {
+			printf("Error with completions: got (%p, %p), not (%p,%p)\n",
+					completed[0], completed[1], src, dst);
+			return -1;
+		}
+
+		for (i = 0; i < length; i++)
+			if (dst_data[i] != src_data[i]) {
+				printf("Data mismatch at char %u\n", i);
+				return -1;
+			}
+		rte_pktmbuf_free(src);
+		rte_pktmbuf_free(dst);
+	} while(0);
+
+	/* test doing multiple copies */
+	do {
+		struct rte_mbuf *srcs[32], *dsts[32];
+		struct rte_mbuf *completed_src[64];
+		struct rte_mbuf *completed_dst[64];
+		unsigned int j;
+
+		for (i = 0; i < RTE_DIM(srcs); i++) {
+			char *src_data;
+
+			srcs[i] = rte_pktmbuf_alloc(pool);
+			dsts[i] = rte_pktmbuf_alloc(pool);
+			srcs[i]->data_len = srcs[i]->pkt_len = length;
+			dsts[i]->data_len = dsts[i]->pkt_len = length;
+			src_data = rte_pktmbuf_mtod(srcs[i], char *);
+
+			for (j = 0; j < length; j++)
+				src_data[j] = rand() & 0xFF;
+
+			if (rte_ioat_enqueue_copy(dev_id,
+					srcs[i]->buf_iova + srcs[i]->data_off,
+					dsts[i]->buf_iova + dsts[i]->data_off,
+					length,
+					(uintptr_t)srcs[i],
+					(uintptr_t)dsts[i],
+					0 /* nofence */) != 1) {
+				printf("Error with rte_ioat_enqueue_copy for buffer %u\n",
+						i);
+				return -1;
+			}
+		}
+		rte_ioat_do_copies(dev_id);
+		usleep(100);
+
+		if (rte_ioat_completed_copies(dev_id, 64, (void *)completed_src,
+				(void *)completed_dst) != RTE_DIM(srcs)) {
+			printf("Error with rte_ioat_completed_copies\n");
+			return -1;
+		}
+		for (i = 0; i < RTE_DIM(srcs); i++) {
+			char *src_data, *dst_data;
+
+			if (completed_src[i] != srcs[i]) {
+				printf("Error with source pointer %u\n", i);
+				return -1;
+			}
+			if (completed_dst[i] != dsts[i]) {
+				printf("Error with dest pointer %u\n", i);
+				return -1;
+			}
+
+			src_data = rte_pktmbuf_mtod(srcs[i], char *);
+			dst_data = rte_pktmbuf_mtod(dsts[i], char *);
+			for (j = 0; j < length; j++)
+				if (src_data[j] != dst_data[j]) {
+					printf("Error with copy of packet %u, byte %u\n",
+							i, j);
+					return -1;
+				}
+			rte_pktmbuf_free(srcs[i]);
+			rte_pktmbuf_free(dsts[i]);
+		}
+
+	} while(0);
+
+	return 0;
+}
+
 static int
 run_ioat_tests(int dev_id)
 {
@@ -54,6 +179,17 @@ run_ioat_tests(int dev_id)
 		return -1;
 	}
 
+	pool = rte_pktmbuf_pool_create("TEST_IOAT_POOL",
+			256, /* n == num elements */
+			32,  /* cache size */
+			0,   /* priv size */
+			2048, /* data room size */
+			info.socket_id);
+	if (pool == NULL) {
+		printf("Error with mempool creation\n");
+		return -1;
+	}
+
 	/* allocate memory for xstats names and values */
 	nb_xstats = rte_rawdev_xstats_names_get(dev_id, NULL, 0);
 
@@ -78,15 +214,32 @@ run_ioat_tests(int dev_id)
 		return -1;
 	}
 
-	rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
-	for (i = 0; i < nb_xstats; i++)
-		printf("%s: %"PRIu64"   ", snames[i].name, stats[i]);
+	/* run the test cases */
+	for (i = 0; i < 100; i++) {
+		unsigned int j;
+
+		if (test_enqueue_copies(dev_id) != 0)
+			goto err;
+
+		rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
+		for (j = 0; j < nb_xstats; j++)
+			printf("%s: %"PRIu64"   ", snames[j].name, stats[j]);
+		printf("\r");
+	}
 	printf("\n");
 
+	rte_mempool_free(pool);
 	free(snames);
 	free(stats);
 	free(ids);
 	return 0;
+
+err:
+	rte_mempool_free(pool);
+	free(snames);
+	free(stats);
+	free(ids);
+	return -1;
 }
 
 static int
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 47f12e95c..3ed12c964 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -112,6 +112,106 @@ The following code shows how the device is configured in
 Once configured, the device can then be made ready for use by calling the
 ``rte_rawdev_start()`` API.
 
+Performing Data Copies
+~~~~~~~~~~~~~~~~~~~~~~~
+
+To perform data copies using IOAT rawdev devices, the functions
+``rte_ioat_enqueue_copy()`` and ``rte_ioat_do_copies()`` should be used.
+Once copies have been completed, the completion will be reported back when
+the application calls ``rte_ioat_completed_copies()``.
+
+The ``rte_ioat_enqueue_copy()`` function enqueues a single copy to the
+device ring for copying at a later point. The parameters to that function
+include the physical addresses of both the source and destination buffers,
+as well as two "handles" to be returned to the user when the copy is
+completed. These handles can be arbitrary values, but two are provided so
+that the library can track handles for both source and destination on
+behalf of the user, e.g. virtual addresses for the buffers, or mbuf
+pointers if packet data is being copied.
+
+While the ``rte_ioat_enqueue_copy()`` function enqueues a copy operation on
+the device ring, the copy will not actually be performed until after the
+application calls the ``rte_ioat_do_copies()`` function. This function
+informs the device hardware of the elements enqueued on the ring, and the
+device will begin to process them. It is expected that, for efficiency
+reasons, a burst of operations will be enqueued to the device via multiple
+enqueue calls between calls to the ``rte_ioat_do_copies()`` function.
+
+The following code from ``test_ioat_rawdev.c`` demonstrates how to enqueue
+a burst of copies to the device and start the hardware processing of them:
+
+.. code-block:: C
+
+        struct rte_mbuf *srcs[32], *dsts[32];
+        unsigned int j;
+
+        for (i = 0; i < RTE_DIM(srcs); i++) {
+                char *src_data;
+
+                srcs[i] = rte_pktmbuf_alloc(pool);
+                dsts[i] = rte_pktmbuf_alloc(pool);
+                srcs[i]->data_len = srcs[i]->pkt_len = length;
+                dsts[i]->data_len = dsts[i]->pkt_len = length;
+                src_data = rte_pktmbuf_mtod(srcs[i], char *);
+
+                for (j = 0; j < length; j++)
+                        src_data[j] = rand() & 0xFF;
+
+                if (rte_ioat_enqueue_copy(dev_id,
+                                srcs[i]->buf_iova + srcs[i]->data_off,
+                                dsts[i]->buf_iova + dsts[i]->data_off,
+                                length,
+                                (uintptr_t)srcs[i],
+                                (uintptr_t)dsts[i],
+                                0 /* nofence */) != 1) {
+                        printf("Error with rte_ioat_enqueue_copy for buffer %u\n",
+                                        i);
+                        return -1;
+                }
+        }
+        rte_ioat_do_copies(dev_id);
+
+To retrieve information about completed copies, the API
+``rte_ioat_completed_copies()`` should be used. This API will return to the
+application a set of completion handles passed in when the relevant copies
+were enqueued.
+
+The following code from ``test_ioat_rawdev.c`` shows the test code
+retrieving information about the completed copies and validating the data
+is correct before freeing the data buffers using the returned handles:
+
+.. code-block:: C
+
+        if (rte_ioat_completed_copies(dev_id, 64, (void *)completed_src,
+                        (void *)completed_dst) != RTE_DIM(srcs)) {
+                printf("Error with rte_ioat_completed_copies\n");
+                return -1;
+        }
+        for (i = 0; i < RTE_DIM(srcs); i++) {
+                char *src_data, *dst_data;
+
+                if (completed_src[i] != srcs[i]) {
+                        printf("Error with source pointer %u\n", i);
+                        return -1;
+                }
+                if (completed_dst[i] != dsts[i]) {
+                        printf("Error with dest pointer %u\n", i);
+                        return -1;
+                }
+
+                src_data = rte_pktmbuf_mtod(srcs[i], char *);
+                dst_data = rte_pktmbuf_mtod(dsts[i], char *);
+                for (j = 0; j < length; j++)
+                        if (src_data[j] != dst_data[j]) {
+                                printf("Error with copy of packet %u, byte %u\n",
+                                                i, j);
+                                return -1;
+                        }
+                rte_pktmbuf_free(srcs[i]);
+                rte_pktmbuf_free(dsts[i]);
+        }
+
+
 Querying Device Statistics
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index abe5ba298..7750edba7 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -14,7 +14,10 @@
  * @b EXPERIMENTAL: these structures and APIs may change without prior notice
  */
 
+#include <x86intrin.h>
+
 #include <rte_memory.h>
+#include <rte_prefetch.h>
 #include <rte_ioat_spec.h>
 
 /** Name of the device driver */
@@ -49,6 +52,10 @@ struct rte_ioat_rawdev {
 	struct rte_ioat_desc *desc_ring;
 	__m128i *hdls; /* completion handles for returning to user */
 
+
+	unsigned short next_read;
+	unsigned short next_write;
+
 	/* some statistics for tracking, if added/changed update xstats fns*/
 	uint64_t enqueue_failed __rte_cache_aligned;
 	uint64_t enqueued;
@@ -72,4 +79,150 @@ struct rte_ioat_desc {
 	uint64_t op_type_specific[4];
 };
 
-#endif
+/**
+ * Enqueue a copy operation onto the ioat device
+ *
+ * This queues up a copy operation to be performed by hardware, but does not
+ * trigger hardware to begin that operation.
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ * @param src
+ *   The physical address of the source buffer
+ * @param dst
+ *   The physical address of the destination buffer
+ * @param length
+ *   The length of the data to be copied
+ * @param src_hdl
+ *   An opaque handle for the source data, to be returned when this operation
+ *   has been completed and the user polls for the completion details
+ * @param dst_hdl
+ *   An opaque handle for the destination data, to be returned when this
+ *   operation has been completed and the user polls for the completion details
+ * @param fence
+ *   A flag parameter indicating that hardware should not begin to perform any
+ *   subsequently enqueued copy operations until after this operation has
+ *   completed
+ * @return
+ *   Number of operations enqueued, either 0 or 1
+ */
+static inline int
+rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst,
+		unsigned int length, uintptr_t src_hdl, uintptr_t dst_hdl,
+		int fence)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	unsigned short read = ioat->next_read;
+	unsigned short write = ioat->next_write;
+	unsigned short mask = ioat->ring_size - 1;
+	unsigned short space = mask + read - write;
+
+	if (space == 0) {
+		ioat->enqueue_failed++;
+		return 0;
+	}
+
+	ioat->next_write = write + 1;
+	write &= mask;
+	/* write transfer size and the descriptor control bits */
+	*((uint64_t *)&ioat->desc_ring[write]) =
+			length | (uint64_t)!!fence << 36 |
+			(uint64_t)(!(write & 0xF)) << 35;
+	ioat->desc_ring[write].src_addr = src;
+	ioat->desc_ring[write].dest_addr = dst;
+	ioat->hdls[write] = _mm_set_epi64((__m64)dst_hdl, (__m64)src_hdl);
+	rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]);
+
+	ioat->enqueued++;
+	return 1;
+}
+
+/**
+ * Trigger hardware to begin performing enqueued copy operations
+ *
+ * This API is used to write the "doorbell" to the hardware to trigger it
+ * to begin the copy operations previously enqueued by rte_ioat_enqueue_copy()
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ */
+static inline void
+rte_ioat_do_copies(int dev_id)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	ioat->desc_ring[(ioat->next_write - 1) & (ioat->ring_size - 1)].desc_control = 8;
+	rte_compiler_barrier();
+	ioat->regs->dmacount = ioat->next_write;
+	ioat->started = ioat->enqueued;
+}
+
+/**
+ * @internal
+ * Returns the index of the last completed operation.
+ */
+static inline int
+rte_ioat_get_last_completed(struct rte_ioat_rawdev *ioat)
+{
+	uint64_t status = ioat->status;
+	/* lower 3 bits indicate "transfer status" : active, idle, halted.. */
+	if (status & 0x6)
+		rte_panic("Error with cbdma: %u\n", (unsigned)status & 0x7);
+
+	return (status - ioat->ring_addr) >> 6;
+}
+
+/**
+ * Returns details of copy operations that have been completed
+ *
+ * Returns to the caller the user-provided "handles" for the copy operations
+ * which have been completed by the hardware, and not already returned by
+ * a previous call to this API.
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ * @param max_copies
+ *   The number of entries which can fit in the src_hdls and dst_hdls
+ *   arrays, i.e. max number of completed operations to report
+ * @param src_hdls
+ *   Array to hold the source handle parameters of the completed copies
+ * @param dst_hdls
+ *   Array to hold the destination handle parameters of the completed copies
+ * @return
+ *   Number of completed operations i.e. number of entries written to the
+ *   src_hdls and dst_hdls array parameters.
+ */
+static inline int
+rte_ioat_completed_copies(int dev_id, uint8_t max_copies,
+		uintptr_t *src_hdls, uintptr_t *dst_hdls)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	unsigned short mask = (ioat->ring_size - 1);
+	unsigned short read = ioat->next_read;
+	unsigned short end_read = (rte_ioat_get_last_completed(ioat) + 1) & mask;
+	unsigned short count = (end_read - (read & mask)) & mask;
+	int i = 0;
+
+	if (count > max_copies)
+		count = max_copies;
+
+	for (; i < count - 1; i += 2, read += 2) {
+		__m128i hdls0 = _mm_load_si128(&ioat->hdls[read & mask]);
+		__m128i hdls1 = _mm_load_si128(&ioat->hdls[(read + 1) & mask]);
+
+		_mm_storeu_si128((void *)&src_hdls[i],
+				_mm_unpacklo_epi64(hdls0, hdls1));
+		_mm_storeu_si128((void *)&dst_hdls[i],
+				_mm_unpackhi_epi64(hdls0, hdls1));
+	}
+	for (; i < count; i++, read++) {
+		uintptr_t *hdls = (void *)&ioat->hdls[read & mask];
+		src_hdls[i] = hdls[0];
+		dst_hdls[i] = hdls[1];
+	}
+
+	ioat->next_read = read;
+	ioat->completed += count;
+	return count;
+}
+
+#endif /* _RTE_IOAT_RAWDEV_H_ */
-- 
2.21.0

Re: [dpdk-dev] [PATCH 8/8] raw/ioat: add local API to perform copies

[Thomas Monjalon]

30/05/2019 23:25, Bruce Richardson:
> Add local APIs to trigger data copies, and retrieve handle values once
> those copies are completed. Included are unit tests to validate the data
> is copies correctly.

Would be nice to have a small design overview in the guide.

We may need also to answer these questions:
What are the performance gains? In which use-case?
What are the hardware requirements?

Re: [dpdk-dev] [PATCH 8/8] raw/ioat: add local API to perform copies

[Bruce Richardson]

On Mon, Jun 03, 2019 at 04:44:17PM +0200, Thomas Monjalon wrote:
> 30/05/2019 23:25, Bruce Richardson:
> > Add local APIs to trigger data copies, and retrieve handle values once
> > those copies are completed. Included are unit tests to validate the data
> > is copies correctly.
> 
> Would be nice to have a small design overview in the guide.
> 
> We may need also to answer these questions:
> What are the performance gains? In which use-case?
> What are the hardware requirements?
> 
Thanks for the comments. Will try and add more details in subsequent
revisions.

/Bruce

Re: [dpdk-dev] [PATCH 8/8] raw/ioat: add local API to perform copies

[Jerin Jacob Kollanukkaran]

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Bruce Richardson
> Sent: Friday, May 31, 2019 2:55 AM
> To: dev@dpdk.org
> Cc: Bruce Richardson <bruce.richardson@intel.com>
> Subject: [dpdk-dev] [PATCH 8/8] raw/ioat: add local API to perform copies
> 
> Add local APIs to trigger data copies, and retrieve handle values once those
> copies are completed. Included are unit tests to validate the data is copies
> correctly.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
>  app/test/test_ioat_rawdev.c        | 159 ++++++++++++++++++++++++++++-

How about moving this test to drivers/raw/ioat/ directory by implementing
dev_selftest ops? IMO, it is better to not pollute app/test with PMD specific
test cases.

Re: [dpdk-dev] [PATCH 8/8] raw/ioat: add local API to perform copies

[Bruce Richardson]

On Wed, Jun 05, 2019 at 01:52:01PM +0000, Jerin Jacob Kollanukkaran wrote:
> 
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Bruce Richardson
> > Sent: Friday, May 31, 2019 2:55 AM
> > To: dev@dpdk.org
> > Cc: Bruce Richardson <bruce.richardson@intel.com>
> > Subject: [dpdk-dev] [PATCH 8/8] raw/ioat: add local API to perform copies
> > 
> > Add local APIs to trigger data copies, and retrieve handle values once those
> > copies are completed. Included are unit tests to validate the data is copies
> > correctly.
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
> >  app/test/test_ioat_rawdev.c        | 159 ++++++++++++++++++++++++++++-
> 
> How about moving this test to drivers/raw/ioat/ directory by implementing
> dev_selftest ops? IMO, it is better to not pollute app/test with PMD specific
> test cases.
> 
Let me look into it.

/Bruce

[dpdk-dev] [PATCH v2 0/8] raw/ioat: driver for Intel QuickData Technology

[Bruce Richardson]

This patch series adds support for the Intel QuickData Technology
device, part of the Intel I/O Acceleration Technology (Intel I/OAT). It
is a raw device for allowing hardware DMA i.e. data copies in hardware.

Performing the copies in hardware can provide performance improvements
for applications where the average copy size is reasonably large, e.g.
1k packets. For smaller packets, e.g. 64-256 bytes, offloading the copy
may reduce performance due to the overhead of using hardware.

V2:
* moved tests to rawdev selftest function
* some checkpatch and other small cleanups
* added extra documentation details on supported hardware
* aligned the changes to dpdk-devbind with the changes in the NTB set
  for consistency

Bruce Richardson (8):
  raw/ioat: add initial support for ioat rawdev driver
  usertools/dpdk-devbind.py: add support for IOAT devices
  raw/ioat: add register definition file
  raw/ioat: create device on probe and destroy on release
  raw/ioat: add device info function
  raw/ioat: add configure, start and stop functions
  raw/ioat: add statistics functions
  raw/ioat: add local API to perform copies

 MAINTAINERS                                 |   6 +-
 app/test/test_rawdev.c                      |  19 ++
 config/common_armv8a_linux                  |   1 +
 config/common_base                          |   5 +
 config/defconfig_arm-armv7a-linuxapp-gcc    |   1 +
 config/defconfig_ppc_64-power8-linuxapp-gcc |   1 +
 doc/guides/rawdevs/index.rst                |   1 +
 doc/guides/rawdevs/ioat_rawdev.rst          | 265 +++++++++++++++++
 doc/guides/rel_notes/release_19_08.rst      |  11 +
 drivers/raw/Makefile                        |   1 +
 drivers/raw/ioat/Makefile                   |  31 ++
 drivers/raw/ioat/ioat_rawdev.c              | 313 ++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev_test.c         | 235 +++++++++++++++
 drivers/raw/ioat/meson.build                |  10 +
 drivers/raw/ioat/rte_ioat_rawdev.h          | 242 +++++++++++++++
 drivers/raw/ioat/rte_ioat_spec.h            | 301 +++++++++++++++++++
 drivers/raw/ioat/rte_pmd_ioat_version.map   |   4 +
 drivers/raw/meson.build                     |   4 +-
 mk/rte.app.mk                               |   1 +
 usertools/dpdk-devbind.py                   |  10 +
 20 files changed, 1460 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/rawdevs/ioat_rawdev.rst
 create mode 100644 drivers/raw/ioat/Makefile
 create mode 100644 drivers/raw/ioat/ioat_rawdev.c
 create mode 100644 drivers/raw/ioat/ioat_rawdev_test.c
 create mode 100644 drivers/raw/ioat/meson.build
 create mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h
 create mode 100644 drivers/raw/ioat/rte_ioat_spec.h
 create mode 100644 drivers/raw/ioat/rte_pmd_ioat_version.map

-- 
2.21.0

[dpdk-dev] [PATCH v2 1/8] raw/ioat: add initial support for ioat rawdev driver

[Bruce Richardson]

Add stubs for ioat rawdev driver support in DPDK, specifically:

  * makefile and meson build hooks
  * initial public header file
  * rawdev main C file, with probe and release functions
  * release note update announcing the driver
  * initial documentation for the new section in the rawdev doc
  * unit test stubs for device unit tests

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---
V2: don't create a new file for unit testing, add to existing rawdev test
    file, and place test cases in the driver selftest routine (added later
    in set)
    Add new section in document about identifying hardware using lspci
---
 MAINTAINERS                                 |  6 +-
 app/test/test_rawdev.c                      |  8 ++
 config/common_armv8a_linux                  |  1 +
 config/common_base                          |  5 ++
 config/defconfig_arm-armv7a-linuxapp-gcc    |  1 +
 config/defconfig_ppc_64-power8-linuxapp-gcc |  1 +
 doc/guides/rawdevs/index.rst                |  1 +
 doc/guides/rawdevs/ioat_rawdev.rst          | 63 ++++++++++++++
 doc/guides/rel_notes/release_19_08.rst      | 11 +++
 drivers/raw/Makefile                        |  1 +
 drivers/raw/ioat/Makefile                   | 28 +++++++
 drivers/raw/ioat/ioat_rawdev.c              | 93 +++++++++++++++++++++
 drivers/raw/ioat/meson.build                |  8 ++
 drivers/raw/ioat/rte_ioat_rawdev.h          | 24 ++++++
 drivers/raw/ioat/rte_pmd_ioat_version.map   |  4 +
 drivers/raw/meson.build                     |  4 +-
 mk/rte.app.mk                               |  1 +
 17 files changed, 258 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/rawdevs/ioat_rawdev.rst
 create mode 100644 drivers/raw/ioat/Makefile
 create mode 100644 drivers/raw/ioat/ioat_rawdev.c
 create mode 100644 drivers/raw/ioat/meson.build
 create mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h
 create mode 100644 drivers/raw/ioat/rte_pmd_ioat_version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index fdc083db9..65735c01e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1047,6 +1047,11 @@ M: Tianfei zhang <tianfei.zhang@intel.com>
 F: drivers/raw/ifpga_rawdev/
 F: doc/guides/rawdevs/ifpga_rawdev.rst
 
+IOAT Rawdev
+M: Bruce Richardson <bruce.richardson@intel.com>
+F: drivers/raw/ioat/
+F: doc/guides/rawdevs/ioat_rawdev.rst
+
 NXP DPAA2 QDMA
 M: Nipun Gupta <nipun.gupta@nxp.com>
 F: drivers/raw/dpaa2_qdma/
@@ -1057,7 +1062,6 @@ M: Nipun Gupta <nipun.gupta@nxp.com>
 F: drivers/raw/dpaa2_cmdif/
 F: doc/guides/rawdevs/dpaa2_cmdif.rst
 
-
 Packet processing
 -----------------
 
diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
index 043a38a13..88549fb61 100644
--- a/app/test/test_rawdev.c
+++ b/app/test/test_rawdev.c
@@ -25,3 +25,11 @@ test_rawdev_selftest_skeleton(void)
 }
 
 REGISTER_TEST_COMMAND(rawdev_autotest, test_rawdev_selftest_skeleton);
+
+static int
+test_rawdev_selftest_ioat(void)
+{
+	return TEST_SKIPPED;
+}
+
+REGISTER_TEST_COMMAND(ioat_rawdev_autotest, test_rawdev_selftest_ioat);
diff --git a/config/common_armv8a_linux b/config/common_armv8a_linux
index 72091de1c..481712ebc 100644
--- a/config/common_armv8a_linux
+++ b/config/common_armv8a_linux
@@ -34,5 +34,6 @@ CONFIG_RTE_ARCH_ARM64_MEMCPY=n
 CONFIG_RTE_LIBRTE_FM10K_PMD=n
 CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
 
 CONFIG_RTE_SCHED_VECTOR=n
diff --git a/config/common_base b/config/common_base
index e406e7836..7c342d8e7 100644
--- a/config/common_base
+++ b/config/common_base
@@ -746,6 +746,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n
 #
 CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
 
+#
+# Compile PMD for Intel IOAT raw device
+#
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=y
+
 #
 # Compile librte_ring
 #
diff --git a/config/defconfig_arm-armv7a-linuxapp-gcc b/config/defconfig_arm-armv7a-linuxapp-gcc
index c9509b274..ee158ef9d 100644
--- a/config/defconfig_arm-armv7a-linuxapp-gcc
+++ b/config/defconfig_arm-armv7a-linuxapp-gcc
@@ -54,3 +54,4 @@ CONFIG_RTE_LIBRTE_QEDE_PMD=n
 CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
 CONFIG_RTE_LIBRTE_NFP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
diff --git a/config/defconfig_ppc_64-power8-linuxapp-gcc b/config/defconfig_ppc_64-power8-linuxapp-gcc
index 7e248b755..9f3670ec0 100644
--- a/config/defconfig_ppc_64-power8-linuxapp-gcc
+++ b/config/defconfig_ppc_64-power8-linuxapp-gcc
@@ -56,3 +56,4 @@ CONFIG_RTE_LIBRTE_ENIC_PMD=n
 CONFIG_RTE_LIBRTE_FM10K_PMD=n
 CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
diff --git a/doc/guides/rawdevs/index.rst b/doc/guides/rawdevs/index.rst
index 7c3bd9586..0a21989e4 100644
--- a/doc/guides/rawdevs/index.rst
+++ b/doc/guides/rawdevs/index.rst
@@ -14,3 +14,4 @@ application through rawdev API.
     dpaa2_cmdif
     dpaa2_qdma
     ifpga_rawdev
+    ioat_rawdev
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
new file mode 100644
index 000000000..0c612e73a
--- /dev/null
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -0,0 +1,63 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+.. include:: <isonum.txt>
+
+IOAT Rawdev Driver for Intel\ |reg| QuickData Technology
+======================================================================
+
+The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg|
+QuickData Technology, part of Intel\ |reg| I/O Acceleration Technology
+`(Intel I/OAT)
+<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_.
+This PMD, when used on supported hardware, allows data copies, for example,
+cloning packet data, to be accelerated by that hardware rather than having to
+be done by software, freeing up CPU cycles for other tasks.
+
+Hardware Requirements
+----------------------
+
+On Linux, the presence of an Intel\ |reg| QuickData Technology hardware can
+be detected by checking the output of the ``lspci`` command, where the
+hardware will be often listed as "Crystal Beach DMA" or "CBDMA". For
+example, on a system with Intel\ |reg| Xeon\ |reg| CPU E5-2699 v4 @2.20GHz,
+lspci shows:
+
+.. code-block:: console
+
+  # lspci | grep DMA
+  00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01)
+  00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01)
+  00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01)
+  00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01)
+  00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01)
+  00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01)
+  00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01)
+  00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01)
+
+On a system with Intel\ |reg| Xeon\ |reg| Gold 6154 CPU @ 3.00GHz, lspci
+shows:
+
+.. code-block:: console
+
+  # lspci | grep DMA
+  00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+
+
+Compilation
+------------
+
+For builds done with ``make``, the driver compilation is enabled by the
+``CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV`` build configuration option. This is
+enabled by default in builds for x86 platforms, and disabled in other
+configurations.
+
+For builds using ``meson`` and ``ninja``, the driver will be built when the
+target platform is x86-based.
diff --git a/doc/guides/rel_notes/release_19_08.rst b/doc/guides/rel_notes/release_19_08.rst
index 8c3932d06..9d26dc437 100644
--- a/doc/guides/rel_notes/release_19_08.rst
+++ b/doc/guides/rel_notes/release_19_08.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright 2019 The DPDK contributors
 
+.. include:: <isonum.txt>
+
 DPDK Release 19.08
 ==================
 
@@ -88,6 +90,15 @@ New Features
   * Added multi-queue support to allow one af_xdp vdev with multiple netdev
     queues
 
+* **Added Intel QuickData Technology PMD**
+
+  The PMD for Intel\ |reg|  QuickData Technology, part of
+  Intel\ |reg|  I/O Acceleration Technology `(Intel I/OAT)
+  <https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_,
+  allows data copies to be done by hardware instead
+  of via software, reducing cycles spent copying large blocks of data in
+  applications.
+
 
 Removed Items
 -------------
diff --git a/drivers/raw/Makefile b/drivers/raw/Makefile
index 8e29b4a56..c1b85c8c7 100644
--- a/drivers/raw/Makefile
+++ b/drivers/raw/Makefile
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev
+DIRS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat
 
 include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
new file mode 100644
index 000000000..7726e310a
--- /dev/null
+++ b/drivers/raw/ioat/Makefile
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_pmd_ioat_rawdev.a
+
+# build flags
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+
+LDLIBS += -lrte_eal -lrte_rawdev
+LDLIBS += -lrte_pci -lrte_bus_pci
+
+# library version
+LIBABIVER := 1
+
+# versioning export map
+EXPORT_MAP := rte_pmd_ioat_version.map
+
+# library source files
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
+
+# export include files
+SYMLINK-y-include += rte_ioat_rawdev.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
new file mode 100644
index 000000000..d9fc3091a
--- /dev/null
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -0,0 +1,93 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include <rte_bus_pci.h>
+#include <rte_rawdev_pmd.h>
+
+#include "rte_ioat_rawdev.h"
+
+/* Dynamic log type identifier */
+int ioat_pmd_logtype;
+
+static struct rte_pci_driver ioat_pmd_drv;
+
+#define IOAT_VENDOR_ID		0x8086
+#define IOAT_DEVICE_ID		0x2021
+
+#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \
+	ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args)
+
+#define IOAT_PMD_DEBUG(fmt, args...)  IOAT_PMD_LOG(DEBUG, fmt, ## args)
+#define IOAT_PMD_INFO(fmt, args...)   IOAT_PMD_LOG(INFO, fmt, ## args)
+#define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
+#define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
+
+static int
+ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
+{
+	RTE_SET_USED(name);
+	RTE_SET_USED(dev);
+	return 0;
+}
+
+static int
+ioat_rawdev_destroy(const char *name)
+{
+	RTE_SET_USED(name);
+	return 0;
+}
+
+static int
+ioat_rawdev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev)
+{
+	char name[32];
+	int ret = 0;
+
+
+	rte_pci_device_name(&dev->addr, name, sizeof(name));
+	IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node);
+
+	dev->device.driver = &drv->driver;
+	ret = ioat_rawdev_create(name, dev);
+	return ret;
+}
+
+static int
+ioat_rawdev_remove(struct rte_pci_device *dev)
+{
+	char name[32];
+	int ret;
+
+	rte_pci_device_name(&dev->addr, name, sizeof(name));
+
+	IOAT_PMD_INFO("Closing %s on NUMA node %d",
+			name, dev->device.numa_node);
+
+	ret = ioat_rawdev_destroy(name);
+	return ret;
+}
+
+static const struct rte_pci_id pci_id_ioat_map[] = {
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID) },
+	{ .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver ioat_pmd_drv = {
+	.id_table = pci_id_ioat_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+		     RTE_PCI_DRV_IOVA_AS_VA,
+	.probe = ioat_rawdev_probe,
+	.remove = ioat_rawdev_remove,
+};
+
+RTE_PMD_REGISTER_PCI(IOAT_PMD_RAWDEV_NAME, ioat_pmd_drv);
+RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_RAWDEV_NAME, pci_id_ioat_map);
+RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_RAWDEV_NAME, "* igb_uio | uio_pci_generic");
+
+RTE_INIT(ioat_pmd_init_log)
+{
+	ioat_pmd_logtype = rte_log_register(IOAT_PMD_LOG_NAME);
+	if (ioat_pmd_logtype >= 0)
+		rte_log_set_level(ioat_pmd_logtype, RTE_LOG_INFO);
+}
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
new file mode 100644
index 000000000..ba7620a68
--- /dev/null
+++ b/drivers/raw/ioat/meson.build
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2019 Intel Corporation
+
+build = dpdk_conf.has('RTE_ARCH_X86')
+sources = files('ioat_rawdev.c')
+deps += ['rawdev', 'bus_pci']
+
+install_headers('rte_ioat_rawdev.h')
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
new file mode 100644
index 000000000..e77406403
--- /dev/null
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_IOAT_RAWDEV_H_
+#define _RTE_IOAT_RAWDEV_H_
+
+/**
+ * @file rte_ioat_rawdev.h
+ *
+ * Definitions for using the ioat rawdev device driver
+ *
+ * @warning
+ * @b EXPERIMENTAL: these structures and APIs may change without prior notice
+ */
+
+/** Name of the device driver */
+#define IOAT_PMD_RAWDEV_NAME rawdev_ioat
+/** String reported as the device driver name by rte_rawdev_info_get() */
+#define IOAT_PMD_RAWDEV_NAME_STR "rawdev_ioat"
+/** Name used to adjust the log level for this driver */
+#define IOAT_PMD_LOG_NAME "rawdev.ioat"
+
+#endif
diff --git a/drivers/raw/ioat/rte_pmd_ioat_version.map b/drivers/raw/ioat/rte_pmd_ioat_version.map
new file mode 100644
index 000000000..9a61188cd
--- /dev/null
+++ b/drivers/raw/ioat/rte_pmd_ioat_version.map
@@ -0,0 +1,4 @@
+DPDK_19.08 {
+
+	local: *;
+};
diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build
index a61cdccef..2af8a70d4 100644
--- a/drivers/raw/meson.build
+++ b/drivers/raw/meson.build
@@ -1,7 +1,9 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2018 NXP
 
-drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
+drivers = ['dpaa2_cmdif', 'dpaa2_qdma',
+	'ifpga_rawdev', 'ioat',
+	'skeleton_rawdev']
 std_deps = ['rawdev']
 config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
 driver_name_fmt = 'rte_pmd_@0@'
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index d0df0b023..52a50bc4e 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -301,6 +301,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV)   += -lrte_pmd_ifpga_rawdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD)       += -lrte_pmd_ipn3ke
 endif # CONFIG_RTE_LIBRTE_IFPGA_BUS
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV)   += -lrte_pmd_ioat_rawdev
 endif # CONFIG_RTE_LIBRTE_RAWDEV
 
 endif # !CONFIG_RTE_BUILD_SHARED_LIBS
-- 
2.21.0

[dpdk-dev] [PATCH v2 2/8] usertools/dpdk-devbind.py: add support for IOAT devices

[Bruce Richardson]

In order to allow binding/unbinding of devices for use by the
ioat_rawdev, we need to update the devbind script to add a new class
of device, and add device ids for the specific HW instances.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---

V2: rather than adding a DMA section, add to "misc (rawdev)" section where
    other device types, e.g. ntb can also do.

NOTE: this set largely overlaps with the equivalent changes made by the
patchset adding NTB support [1]. Since it's unclear which set will be added
first, this set is based off the latest head.

[1] http://patches.dpdk.org/patch/55127/
---
 doc/guides/rawdevs/ioat_rawdev.rst | 11 +++++++++++
 usertools/dpdk-devbind.py          | 10 ++++++++++
 2 files changed, 21 insertions(+)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 0c612e73a..1a4b0e03e 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -61,3 +61,14 @@ configurations.
 
 For builds using ``meson`` and ``ninja``, the driver will be built when the
 target platform is x86-based.
+
+Device Setup
+-------------
+
+The Intel\ |reg| QuickData Technology HW devices will need to be bound to a
+user-space IO driver for use. The script ``dpdk-devbind.py`` script
+included with DPDK can be used to view the state of the devices and to bind
+them to a suitable DPDK-supported kernel driver. When querying the status
+of the devices, they will appear under the category of "Misc (rawdev)
+devices", i.e. the command ``dpdk-devbind.py --status-dev misc`` can be
+used to see the state of those devices alone.
diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py
index 9e79f0d28..5c1cd3548 100755
--- a/usertools/dpdk-devbind.py
+++ b/usertools/dpdk-devbind.py
@@ -36,11 +36,17 @@
 octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc',
               'SVendor': None, 'SDevice': None}
 
+intel_ioat_bdw = {'Class': '08', 'Vendor': '8086', 'Device': '6f20,6f21,6f22,6f23,6f24,6f25,6f26,6f27,6f2e,6f2f',
+              'SVendor': None, 'SDevice': None}
+intel_ioat_skx = {'Class': '08', 'Vendor': '8086', 'Device': '2021',
+              'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 crypto_devices = [encryption_class, intel_processor_class]
 eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso]
 mempool_devices = [cavium_fpa, octeontx2_npa]
 compress_devices = [cavium_zip]
+misc_devices = [intel_ioat_bdw, intel_ioat_skx]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
@@ -595,6 +601,8 @@ def show_status():
     if status_dev == "compress" or status_dev == "all":
         show_device_status(compress_devices , "Compress")
 
+    if status_dev == "misc" or status_dev == "all":
+        show_device_status(misc_devices, "Misc (rawdev)")
 
 def parse_args():
     '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +678,7 @@ def do_arg_actions():
             get_device_details(eventdev_devices)
             get_device_details(mempool_devices)
             get_device_details(compress_devices)
+            get_device_details(misc_devices)
         show_status()
 
 
@@ -690,6 +699,7 @@ def main():
     get_device_details(eventdev_devices)
     get_device_details(mempool_devices)
     get_device_details(compress_devices)
+    get_device_details(misc_devices)
     do_arg_actions()
 
 if __name__ == "__main__":
-- 
2.21.0

[dpdk-dev] [PATCH v2 3/8] raw/ioat: add register definition file

[Bruce Richardson]

Add in the list of registers for the device. File is taken from the SPDK
project:

  https://github.com/spdk/spdk/blob/master/include/spdk/ioat_spec.h

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 drivers/raw/ioat/Makefile        |   1 +
 drivers/raw/ioat/meson.build     |   3 +-
 drivers/raw/ioat/rte_ioat_spec.h | 301 +++++++++++++++++++++++++++++++
 3 files changed, 304 insertions(+), 1 deletion(-)
 create mode 100644 drivers/raw/ioat/rte_ioat_spec.h

diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
index 7726e310a..1e10938f3 100644
--- a/drivers/raw/ioat/Makefile
+++ b/drivers/raw/ioat/Makefile
@@ -24,5 +24,6 @@ SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
 
 # export include files
 SYMLINK-y-include += rte_ioat_rawdev.h
+SYMLINK-y-include += rte_ioat_spec.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
index ba7620a68..ca23e23fc 100644
--- a/drivers/raw/ioat/meson.build
+++ b/drivers/raw/ioat/meson.build
@@ -5,4 +5,5 @@ build = dpdk_conf.has('RTE_ARCH_X86')
 sources = files('ioat_rawdev.c')
 deps += ['rawdev', 'bus_pci']
 
-install_headers('rte_ioat_rawdev.h')
+install_headers('rte_ioat_rawdev.h',
+		'rte_ioat_spec.h')
diff --git a/drivers/raw/ioat/rte_ioat_spec.h b/drivers/raw/ioat/rte_ioat_spec.h
new file mode 100644
index 000000000..305e36ded
--- /dev/null
+++ b/drivers/raw/ioat/rte_ioat_spec.h
@@ -0,0 +1,301 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) Intel Corporation
+ */
+
+/**
+ * \file
+ * I/OAT specification definitions
+ *
+ * Taken from ioat_spec.h from SPDK project, with prefix renames and
+ * other minor changes.
+ */
+
+#ifndef RTE_IOAT_SPEC_H
+#define RTE_IOAT_SPEC_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+#define RTE_IOAT_PCI_CHANERR_INT_OFFSET	0x180
+
+#define RTE_IOAT_INTRCTRL_MASTER_INT_EN	0x01
+
+#define RTE_IOAT_VER_3_0                0x30
+#define RTE_IOAT_VER_3_3                0x33
+
+/* DMA Channel Registers */
+#define RTE_IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK	0xF000
+#define RTE_IOAT_CHANCTRL_COMPL_DCA_EN		0x0200
+#define RTE_IOAT_CHANCTRL_CHANNEL_IN_USE		0x0100
+#define RTE_IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL	0x0020
+#define RTE_IOAT_CHANCTRL_ERR_INT_EN		0x0010
+#define RTE_IOAT_CHANCTRL_ANY_ERR_ABORT_EN		0x0008
+#define RTE_IOAT_CHANCTRL_ERR_COMPLETION_EN		0x0004
+#define RTE_IOAT_CHANCTRL_INT_REARM			0x0001
+
+/* DMA Channel Capabilities */
+#define	RTE_IOAT_DMACAP_PB		(1 << 0)
+#define	RTE_IOAT_DMACAP_DCA		(1 << 4)
+#define	RTE_IOAT_DMACAP_BFILL		(1 << 6)
+#define	RTE_IOAT_DMACAP_XOR		(1 << 8)
+#define	RTE_IOAT_DMACAP_PQ		(1 << 9)
+#define	RTE_IOAT_DMACAP_DMA_DIF	(1 << 10)
+
+struct rte_ioat_registers {
+	uint8_t		chancnt;
+	uint8_t		xfercap;
+	uint8_t		genctrl;
+	uint8_t		intrctrl;
+	uint32_t	attnstatus;
+	uint8_t		cbver;		/* 0x08 */
+	uint8_t		reserved4[0x3]; /* 0x09 */
+	uint16_t	intrdelay;	/* 0x0C */
+	uint16_t	cs_status;	/* 0x0E */
+	uint32_t	dmacapability;	/* 0x10 */
+	uint8_t		reserved5[0x6C]; /* 0x14 */
+	uint16_t	chanctrl;	/* 0x80 */
+	uint8_t		reserved6[0x2];	/* 0x82 */
+	uint8_t		chancmd;	/* 0x84 */
+	uint8_t		reserved3[1];	/* 0x85 */
+	uint16_t	dmacount;	/* 0x86 */
+	uint64_t	chansts;	/* 0x88 */
+	uint64_t	chainaddr;	/* 0x90 */
+	uint64_t	chancmp;	/* 0x98 */
+	uint8_t		reserved2[0x8];	/* 0xA0 */
+	uint32_t	chanerr;	/* 0xA8 */
+	uint32_t	chanerrmask;	/* 0xAC */
+} __attribute__((packed));
+
+#define RTE_IOAT_CHANCMD_RESET			0x20
+#define RTE_IOAT_CHANCMD_SUSPEND		0x04
+
+#define RTE_IOAT_CHANSTS_STATUS		0x7ULL
+#define RTE_IOAT_CHANSTS_ACTIVE		0x0
+#define RTE_IOAT_CHANSTS_IDLE			0x1
+#define RTE_IOAT_CHANSTS_SUSPENDED		0x2
+#define RTE_IOAT_CHANSTS_HALTED		0x3
+#define RTE_IOAT_CHANSTS_ARMED			0x4
+
+#define RTE_IOAT_CHANSTS_UNAFFILIATED_ERROR	0x8ULL
+#define RTE_IOAT_CHANSTS_SOFT_ERROR		0x10ULL
+
+#define RTE_IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK	(~0x3FULL)
+
+#define RTE_IOAT_CHANCMP_ALIGN			8	/* CHANCMP address must be 64-bit aligned */
+
+struct rte_ioat_generic_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t reserved2: 1;
+			uint32_t src_page_break: 1;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t op_specific[4];
+};
+
+struct rte_ioat_dma_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t null: 1;
+			uint32_t src_page_break: 1;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+#define RTE_IOAT_OP_COPY 0x00
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t reserved;
+	uint64_t reserved2;
+	uint64_t user1;
+	uint64_t user2;
+};
+
+struct rte_ioat_fill_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t reserved: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t reserved2: 2;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t reserved3: 15;
+#define RTE_IOAT_OP_FILL 0x01
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_data;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t reserved;
+	uint64_t next_dest_addr;
+	uint64_t user1;
+	uint64_t user2;
+};
+
+struct rte_ioat_xor_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_count: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+#define RTE_IOAT_OP_XOR 0x87
+#define RTE_IOAT_OP_XOR_VAL 0x88
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t src_addr3;
+	uint64_t src_addr4;
+	uint64_t src_addr5;
+};
+
+struct rte_ioat_xor_ext_hw_desc {
+	uint64_t src_addr6;
+	uint64_t src_addr7;
+	uint64_t src_addr8;
+	uint64_t next;
+	uint64_t reserved[4];
+};
+
+struct rte_ioat_pq_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_count: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t p_disable: 1;
+			uint32_t q_disable: 1;
+			uint32_t reserved: 11;
+#define RTE_IOAT_OP_PQ 0x89
+#define RTE_IOAT_OP_PQ_VAL 0x8a
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t p_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t src_addr3;
+	uint8_t  coef[8];
+	uint64_t q_addr;
+};
+
+struct rte_ioat_pq_ext_hw_desc {
+	uint64_t src_addr4;
+	uint64_t src_addr5;
+	uint64_t src_addr6;
+	uint64_t next;
+	uint64_t src_addr7;
+	uint64_t src_addr8;
+	uint64_t reserved[2];
+};
+
+struct rte_ioat_pq_update_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_cnt: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t p_disable: 1;
+			uint32_t q_disable: 1;
+			uint32_t reserved: 3;
+			uint32_t coef: 8;
+#define RTE_IOAT_OP_PQ_UP 0x8b
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t p_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t p_src;
+	uint64_t q_src;
+	uint64_t q_addr;
+};
+
+struct rte_ioat_raw_hw_desc {
+	uint64_t field[8];
+};
+
+union rte_ioat_hw_desc {
+	struct rte_ioat_raw_hw_desc raw;
+	struct rte_ioat_generic_hw_desc generic;
+	struct rte_ioat_dma_hw_desc dma;
+	struct rte_ioat_fill_hw_desc fill;
+	struct rte_ioat_xor_hw_desc xor_desc;
+	struct rte_ioat_xor_ext_hw_desc xor_ext;
+	struct rte_ioat_pq_hw_desc pq;
+	struct rte_ioat_pq_ext_hw_desc pq_ext;
+	struct rte_ioat_pq_update_hw_desc pq_update;
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_IOAT_SPEC_H */
-- 
2.21.0

[dpdk-dev] [PATCH v2 4/8] raw/ioat: create device on probe and destroy on release

[Bruce Richardson]

Add the create/destroy driver functions so that we can actually allocate
a rawdev and destroy it when done. No rawdev API functions are actually
implemented at this point.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/rawdevs/ioat_rawdev.rst | 11 ++++
 drivers/raw/ioat/ioat_rawdev.c     | 93 +++++++++++++++++++++++++++++-
 drivers/raw/ioat/rte_ioat_rawdev.h | 20 +++++++
 3 files changed, 121 insertions(+), 3 deletions(-)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 1a4b0e03e..4b7fe8a8f 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -72,3 +72,14 @@ them to a suitable DPDK-supported kernel driver. When querying the status
 of the devices, they will appear under the category of "Misc (rawdev)
 devices", i.e. the command ``dpdk-devbind.py --status-dev misc`` can be
 used to see the state of those devices alone.
+
+Device Probing and Initialization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once bound to a suitable kernel device driver, the HW devices will be found
+as part of the PCI scan done at application initialization time. No vdev
+parameters need to be passed to create or initialize the device.
+
+Once probed successfully, the device will appear as a ``rawdev``, that is a
+"raw device type" inside DPDK, and can be accessed using APIs from the
+``rte_rawdev`` library.
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index d9fc3091a..d13391dd5 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -2,6 +2,7 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <rte_cycles.h>
 #include <rte_bus_pci.h>
 #include <rte_rawdev_pmd.h>
 
@@ -26,15 +27,101 @@ static struct rte_pci_driver ioat_pmd_drv;
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
-	RTE_SET_USED(name);
-	RTE_SET_USED(dev);
+	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+	};
+
+	struct rte_rawdev *rawdev = NULL;
+	struct rte_ioat_rawdev *ioat = NULL;
+	int ret = 0;
+	int retry = 0;
+
+	if (!name) {
+		IOAT_PMD_ERR("Invalid name of the device!");
+		ret = -EINVAL;
+		goto cleanup;
+	}
+
+	/* Allocate device structure */
+	rawdev = rte_rawdev_pmd_allocate(name, sizeof(struct rte_ioat_rawdev),
+					 dev->device.numa_node);
+	if (rawdev == NULL) {
+		IOAT_PMD_ERR("Unable to allocate raw device");
+		ret = -EINVAL;
+		goto cleanup;
+	}
+
+	rawdev->dev_ops = &ioat_rawdev_ops;
+	rawdev->device = &dev->device;
+	rawdev->driver_name = dev->device.driver->name;
+
+	ioat = rawdev->dev_private;
+	ioat->rawdev = rawdev;
+	ioat->regs = dev->mem_resource[0].addr;
+	ioat->ring_size = 0;
+	ioat->desc_ring = NULL;
+	ioat->status_addr = rte_malloc_virt2iova(ioat) +
+			offsetof(struct rte_ioat_rawdev, status);
+
+	/* do device initialization - reset and set error behaviour */
+	if (ioat->regs->chancnt != 1)
+		IOAT_PMD_ERR("%s: Channel count == %d\n", __func__,
+				ioat->regs->chancnt);
+
+	if (ioat->regs->chanctrl & 0x100) { /* locked by someone else */
+		IOAT_PMD_WARN("%s: Channel appears locked\n", __func__);
+		ioat->regs->chanctrl = 0;
+	}
+
+	ioat->regs->chancmd = RTE_IOAT_CHANCMD_SUSPEND;
+	rte_delay_ms(1);
+	ioat->regs->chancmd = RTE_IOAT_CHANCMD_RESET;
+	rte_delay_ms(1);
+	while (ioat->regs->chancmd & RTE_IOAT_CHANCMD_RESET) {
+		ioat->regs->chainaddr = 0;
+		rte_delay_ms(1);
+		if (++retry >= 200) {
+			IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=0x%"PRIx8", CHANSTS=0x%"PRIx64", CHANERR=0x%"PRIx32"\n",
+					__func__,
+					ioat->regs->chancmd,
+					ioat->regs->chansts,
+					ioat->regs->chanerr);
+			ret = -EIO;
+		}
+	}
+	ioat->regs->chanctrl = RTE_IOAT_CHANCTRL_ANY_ERR_ABORT_EN |
+			RTE_IOAT_CHANCTRL_ERR_COMPLETION_EN;
+
 	return 0;
+
+cleanup:
+	if (rawdev)
+		rte_rawdev_pmd_release(rawdev);
+
+	return ret;
 }
 
 static int
 ioat_rawdev_destroy(const char *name)
 {
-	RTE_SET_USED(name);
+	int ret;
+	struct rte_rawdev *rdev;
+
+	if (!name) {
+		IOAT_PMD_ERR("Invalid device name");
+		return -EINVAL;
+	}
+
+	rdev = rte_rawdev_pmd_get_named_dev(name);
+	if (!rdev) {
+		IOAT_PMD_ERR("Invalid device name (%s)", name);
+		return -EINVAL;
+	}
+
+	/* rte_rawdev_close is called by pmd_release */
+	ret = rte_rawdev_pmd_release(rdev);
+	if (ret)
+		IOAT_PMD_DEBUG("Device cleanup failed");
+
 	return 0;
 }
 
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index e77406403..c3216a174 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -14,6 +14,9 @@
  * @b EXPERIMENTAL: these structures and APIs may change without prior notice
  */
 
+#include <rte_memory.h>
+#include <rte_ioat_spec.h>
+
 /** Name of the device driver */
 #define IOAT_PMD_RAWDEV_NAME rawdev_ioat
 /** String reported as the device driver name by rte_rawdev_info_get() */
@@ -21,4 +24,21 @@
 /** Name used to adjust the log level for this driver */
 #define IOAT_PMD_LOG_NAME "rawdev.ioat"
 
+/**
+ * @internal
+ * Structure representing a device instance
+ */
+struct rte_ioat_rawdev {
+	struct rte_rawdev *rawdev;
+	volatile struct rte_ioat_registers *regs;
+	phys_addr_t status_addr;
+	phys_addr_t ring_addr;
+
+	unsigned short ring_size;
+	struct rte_ioat_desc *desc_ring;
+
+	/* to report completions, the device will write status back here */
+	volatile uint64_t status __rte_cache_aligned;
+};
+
 #endif
-- 
2.21.0

[dpdk-dev] [PATCH v2 5/8] raw/ioat: add device info function

[Bruce Richardson]

Add in the "info_get" function to the driver, to allow us to query the
device. This allows us to have the unit test pick up the presence of
supported hardware or not.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---

V2: Test case is placed in driver self-test routine
---
 app/test/test_rawdev.c             | 11 ++++++++++
 doc/guides/rawdevs/ioat_rawdev.rst | 34 ++++++++++++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev.c     | 11 ++++++++++
 drivers/raw/ioat/rte_ioat_rawdev.h | 11 ++++++++++
 4 files changed, 67 insertions(+)

diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
index 88549fb61..4db762b4c 100644
--- a/app/test/test_rawdev.c
+++ b/app/test/test_rawdev.c
@@ -29,6 +29,17 @@ REGISTER_TEST_COMMAND(rawdev_autotest, test_rawdev_selftest_skeleton);
 static int
 test_rawdev_selftest_ioat(void)
 {
+	const int count = rte_rawdev_count();
+	int i;
+
+	for (i = 0; i < count; i++) {
+		struct rte_rawdev_info info = { .dev_private = NULL };
+		if (rte_rawdev_info_get(i, &info) == 0 &&
+				strstr(info.driver_name, "ioat") != NULL)
+			return 0;
+	}
+
+	printf("No IOAT rawdev found, skipping tests\n");
 	return TEST_SKIPPED;
 }
 
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 4b7fe8a8f..0ce984490 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -83,3 +83,37 @@ parameters need to be passed to create or initialize the device.
 Once probed successfully, the device will appear as a ``rawdev``, that is a
 "raw device type" inside DPDK, and can be accessed using APIs from the
 ``rte_rawdev`` library.
+
+Using IOAT Rawdev Devices
+--------------------------
+
+To use the devices from an application, the rawdev API can be used, along
+with definitions taken from the device-specific header file
+``rte_ioat_rawdev.h``. This header is needed to get the definition of
+structure parameters used by some of the rawdev APIs for IOAT rawdev
+devices, as well as providing key functions for using the device for memory
+copies.
+
+Getting Device Information
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Basic information about each rawdev device can be got using the
+``rte_rawdev_info_get()`` API. For most applications, this API will be
+needed to verify that the rawdev in question is of the expected type. For
+example, the following code in ``test_ioat_rawdev.c`` is used to identify
+the IOAT rawdev device for use for the tests:
+
+.. code-block:: C
+
+        for (i = 0; i < count && !found; i++) {
+                struct rte_rawdev_info info = { .dev_private = NULL };
+                found = (rte_rawdev_info_get(i, &info) == 0 &&
+                                strcmp(info.driver_name,
+                                                IOAT_PMD_RAWDEV_NAME_STR) == 0);
+        }
+
+When calling the ``rte_rawdev_info_get()`` API for an IOAT rawdev device,
+the ``dev_private`` field in the ``rte_rawdev_info`` struct should either
+be NULL, or else be set to point to a structure of type
+``rte_ioat_rawdev_config``, in which case the size of the configured device
+input ring will be returned in that structure.
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index d13391dd5..08e7586c6 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -24,10 +24,21 @@ static struct rte_pci_driver ioat_pmd_drv;
 #define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
 #define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
 
+static void
+ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
+{
+	struct rte_ioat_rawdev_config *cfg = dev_info;
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+
+	if (cfg != NULL)
+		cfg->ring_size = ioat->ring_size;
+}
+
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
 	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+			.dev_info_get = ioat_dev_info_get,
 	};
 
 	struct rte_rawdev *rawdev = NULL;
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index c3216a174..7e0d72ca3 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -24,6 +24,17 @@
 /** Name used to adjust the log level for this driver */
 #define IOAT_PMD_LOG_NAME "rawdev.ioat"
 
+/**
+ * Configuration structure for an ioat rawdev instance
+ *
+ * This structure is to be passed as the ".dev_private" parameter when
+ * calling the rte_rawdev_get_info() and rte_rawdev_configure() APIs on
+ * an ioat rawdev instance.
+ */
+struct rte_ioat_rawdev_config {
+	unsigned short ring_size;
+};
+
 /**
  * @internal
  * Structure representing a device instance
-- 
2.21.0

[dpdk-dev] [PATCH v2 6/8] raw/ioat: add configure, start and stop functions

[Bruce Richardson]

Allow initializing a driver instance. Include selftest to validate these
functions.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---

V2: test cases placed in self-test routine
---
 app/test/test_rawdev.c              |  2 +-
 doc/guides/rawdevs/ioat_rawdev.rst  | 32 ++++++++++++
 drivers/raw/ioat/Makefile           |  1 +
 drivers/raw/ioat/ioat_rawdev.c      | 78 +++++++++++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev_test.c | 41 +++++++++++++++
 drivers/raw/ioat/meson.build        |  3 +-
 drivers/raw/ioat/rte_ioat_rawdev.h  | 15 ++++++
 7 files changed, 170 insertions(+), 2 deletions(-)
 create mode 100644 drivers/raw/ioat/ioat_rawdev_test.c

diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
index 4db762b4c..731e51717 100644
--- a/app/test/test_rawdev.c
+++ b/app/test/test_rawdev.c
@@ -36,7 +36,7 @@ test_rawdev_selftest_ioat(void)
 		struct rte_rawdev_info info = { .dev_private = NULL };
 		if (rte_rawdev_info_get(i, &info) == 0 &&
 				strstr(info.driver_name, "ioat") != NULL)
-			return 0;
+			return rte_rawdev_selftest(i);
 	}
 
 	printf("No IOAT rawdev found, skipping tests\n");
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 0ce984490..9ab97e2aa 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -117,3 +117,35 @@ the ``dev_private`` field in the ``rte_rawdev_info`` struct should either
 be NULL, or else be set to point to a structure of type
 ``rte_ioat_rawdev_config``, in which case the size of the configured device
 input ring will be returned in that structure.
+
+Device Configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+Configuring an IOAT rawdev device is done using the
+``rte_rawdev_configure()`` API, which takes the same structure parameters
+as the, previously referenced, ``rte_rawdev_info_get()`` API. The main
+difference is that, because the parameter is used as input rather than
+output, the ``dev_private`` structure element cannot be NULL, and must
+point to a valid ``rte_ioat_rawdev_config`` structure, containing the ring
+size to be used by the device. The ring size must be a power of two,
+between 64 and 4096.
+
+The following code shows how the device is configured in
+``test_ioat_rawdev.c``:
+
+.. code-block:: C
+
+   #define IOAT_TEST_RINGSIZE 512
+        struct rte_ioat_rawdev_config p = { .ring_size = -1 };
+        struct rte_rawdev_info info = { .dev_private = &p };
+
+        /* ... */
+
+        p.ring_size = IOAT_TEST_RINGSIZE;
+        if (rte_rawdev_configure(dev_id, &info) != 0) {
+                printf("Error with rte_rawdev_configure()\n");
+                return -1;
+        }
+
+Once configured, the device can then be made ready for use by calling the
+``rte_rawdev_start()`` API.
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
index 1e10938f3..b1af9c666 100644
--- a/drivers/raw/ioat/Makefile
+++ b/drivers/raw/ioat/Makefile
@@ -21,6 +21,7 @@ EXPORT_MAP := rte_pmd_ioat_version.map
 
 # library source files
 SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev_test.c
 
 # export include files
 SYMLINK-y-include += rte_ioat_rawdev.h
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index 08e7586c6..46f41c661 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -24,6 +24,78 @@ static struct rte_pci_driver ioat_pmd_drv;
 #define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
 #define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
 
+#define DESC_SZ sizeof(struct rte_ioat_desc)
+#define COMPLETION_SZ sizeof(__m128i)
+
+static int
+ioat_dev_configure(const struct rte_rawdev *dev, rte_rawdev_obj_t config)
+{
+	struct rte_ioat_rawdev_config *params = config;
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+	unsigned short i;
+
+	if (dev->started)
+		return -EBUSY;
+
+	if (params == NULL)
+		return -EINVAL;
+
+	if (params->ring_size > 4096 || params->ring_size < 64 ||
+			!rte_is_power_of_2(params->ring_size))
+		return -EINVAL;
+
+	ioat->ring_size = params->ring_size;
+	if (ioat->desc_ring != NULL) {
+		rte_free(ioat->desc_ring);
+		ioat->desc_ring = NULL;
+	}
+
+	/* allocate one block of memory for both descriptors
+	 * and completion handles.
+	 */
+	ioat->desc_ring = rte_zmalloc_socket(NULL,
+			(DESC_SZ + COMPLETION_SZ) * ioat->ring_size,
+			0, /* alignment, default to 64Byte */
+			dev->device->numa_node);
+	if (ioat->desc_ring == NULL)
+		return -ENOMEM;
+	ioat->hdls = (void *)&ioat->desc_ring[ioat->ring_size];
+
+	ioat->ring_addr = rte_malloc_virt2iova(ioat->desc_ring);
+
+	/* configure descriptor ring - each one points to next */
+	for (i = 0; i < ioat->ring_size; i++) {
+		ioat->desc_ring[i].next_desc_addr = ioat->ring_addr +
+				(((i + 1) % ioat->ring_size) * DESC_SZ);
+	}
+
+	return 0;
+}
+
+static int
+ioat_dev_start(struct rte_rawdev *dev)
+{
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+
+	if (ioat->ring_size == 0 || ioat->desc_ring == NULL)
+		return -EBUSY;
+
+	/* inform hardware of where the descriptor ring is */
+	ioat->regs->chainaddr = ioat->ring_addr;
+	/* inform hardware of where to write the status/completions */
+	ioat->regs->chancmp = ioat->status_addr;
+
+	/* prime the status register to be set to the last element */
+	ioat->status =  ioat->ring_addr + ((ioat->ring_size - 1) * DESC_SZ);
+	return 0;
+}
+
+static void
+ioat_dev_stop(struct rte_rawdev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
 static void
 ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
 {
@@ -34,11 +106,17 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
 		cfg->ring_size = ioat->ring_size;
 }
 
+int ioat_rawdev_test(uint16_t dev_id);
+
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
 	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+			.dev_configure = ioat_dev_configure,
+			.dev_start = ioat_dev_start,
+			.dev_stop = ioat_dev_stop,
 			.dev_info_get = ioat_dev_info_get,
+			.dev_selftest = ioat_rawdev_test,
 	};
 
 	struct rte_rawdev *rawdev = NULL;
diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c
new file mode 100644
index 000000000..5375da26c
--- /dev/null
+++ b/drivers/raw/ioat/ioat_rawdev_test.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include "rte_rawdev.h"
+#include "rte_ioat_rawdev.h"
+
+int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */
+
+int
+ioat_rawdev_test(uint16_t dev_id)
+{
+#define IOAT_TEST_RINGSIZE 512
+	struct rte_ioat_rawdev_config p = { .ring_size = -1 };
+	struct rte_rawdev_info info = { .dev_private = &p };
+
+	rte_rawdev_info_get(dev_id, &info);
+	if (p.ring_size != 0) {
+		printf("Error, initial ring size is non-zero (%d)\n",
+				(int)p.ring_size);
+		return -1;
+	}
+
+	p.ring_size = IOAT_TEST_RINGSIZE;
+	if (rte_rawdev_configure(dev_id, &info) != 0) {
+		printf("Error with rte_rawdev_configure()\n");
+		return -1;
+	}
+	rte_rawdev_info_get(dev_id, &info);
+	if (p.ring_size != IOAT_TEST_RINGSIZE) {
+		printf("Error, ring size is not %d (%d)\n",
+				IOAT_TEST_RINGSIZE, (int)p.ring_size);
+		return -1;
+	}
+
+	if (rte_rawdev_start(dev_id) != 0) {
+		printf("Error with rte_rawdev_start()\n");
+		return -1;
+	}
+	return 0;
+}
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
index ca23e23fc..40fff6654 100644
--- a/drivers/raw/ioat/meson.build
+++ b/drivers/raw/ioat/meson.build
@@ -2,7 +2,8 @@
 # Copyright 2019 Intel Corporation
 
 build = dpdk_conf.has('RTE_ARCH_X86')
-sources = files('ioat_rawdev.c')
+sources = files('ioat_rawdev.c',
+		'ioat_rawdev_test.c')
 deps += ['rawdev', 'bus_pci']
 
 install_headers('rte_ioat_rawdev.h',
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index 7e0d72ca3..0c41a9dcf 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -14,6 +14,7 @@
  * @b EXPERIMENTAL: these structures and APIs may change without prior notice
  */
 
+#include <x86intrin.h>
 #include <rte_memory.h>
 #include <rte_ioat_spec.h>
 
@@ -47,9 +48,23 @@ struct rte_ioat_rawdev {
 
 	unsigned short ring_size;
 	struct rte_ioat_desc *desc_ring;
+	__m128i *hdls; /* completion handles for returning to user */
 
 	/* to report completions, the device will write status back here */
 	volatile uint64_t status __rte_cache_aligned;
 };
 
+/**
+ * @internal
+ * Structure representing a descriptor for a copy operation
+ */
+struct rte_ioat_desc {
+	uint32_t xfer_size __rte_cache_aligned;
+	uint32_t desc_control;
+	phys_addr_t src_addr; /* 64 bits */
+	phys_addr_t dest_addr; /* 64 bits */
+	phys_addr_t next_desc_addr; /* 64 bits */
+	uint64_t op_type_specific[4];
+};
+
 #endif
-- 
2.21.0

[dpdk-dev] [PATCH v2 7/8] raw/ioat: add statistics functions

[Bruce Richardson]

Add stats functions to track what is happening in the driver, and put
unit tests to check those.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---

V2: test cases moved to self-test routine
---
 doc/guides/rawdevs/ioat_rawdev.rst  | 14 +++++++++
 drivers/raw/ioat/ioat_rawdev.c      | 44 +++++++++++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev_test.c | 39 +++++++++++++++++++++++++
 drivers/raw/ioat/rte_ioat_rawdev.h  |  6 ++++
 4 files changed, 103 insertions(+)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 9ab97e2aa..b908f31e0 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -149,3 +149,17 @@ The following code shows how the device is configured in
 
 Once configured, the device can then be made ready for use by calling the
 ``rte_rawdev_start()`` API.
+
+Querying Device Statistics
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The statistics from the IOAT rawdev device can be got via the xstats
+functions in the ``rte_rawdev`` library, i.e.
+``rte_rawdev_xstats_names_get()``, ``rte_rawdev_xstats_get()`` and
+``rte_rawdev_xstats_by_name_get``. The statistics returned for each device
+instance are:
+
+* ``failed_enqueues``
+* ``successful_enqueues``
+* ``copies_started``
+* ``copies_completed``
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index 46f41c661..bdecbc727 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -4,6 +4,7 @@
 
 #include <rte_cycles.h>
 #include <rte_bus_pci.h>
+#include <rte_string_fns.h>
 #include <rte_rawdev_pmd.h>
 
 #include "rte_ioat_rawdev.h"
@@ -106,6 +107,47 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
 		cfg->ring_size = ioat->ring_size;
 }
 
+static const char * const xstat_names[] = {
+		"failed_enqueues", "successful_enqueues",
+		"copies_started", "copies_completed"
+};
+
+static int
+ioat_xstats_get(const struct rte_rawdev *dev, const unsigned int ids[],
+		uint64_t values[], unsigned int n)
+{
+	const struct rte_ioat_rawdev *ioat = dev->dev_private;
+	unsigned int i;
+
+	for (i = 0; i < n; i++) {
+		switch (ids[i]) {
+		case 0: values[i] = ioat->enqueue_failed; break;
+		case 1: values[i] = ioat->enqueued; break;
+		case 2: values[i] = ioat->started; break;
+		case 3: values[i] = ioat->completed; break;
+		default: values[i] = 0; break;
+		}
+	}
+	return n;
+}
+
+static int
+ioat_xstats_get_names(const struct rte_rawdev *dev,
+		struct rte_rawdev_xstats_name *names,
+		unsigned int size)
+{
+	unsigned int i;
+
+	RTE_SET_USED(dev);
+	if (size < RTE_DIM(xstat_names))
+		return RTE_DIM(xstat_names);
+
+	for (i = 0; i < RTE_DIM(xstat_names); i++)
+		strlcpy(names[i].name, xstat_names[i], sizeof(names[i]));
+
+	return RTE_DIM(xstat_names);
+}
+
 int ioat_rawdev_test(uint16_t dev_id);
 
 static int
@@ -116,6 +158,8 @@ ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 			.dev_start = ioat_dev_start,
 			.dev_stop = ioat_dev_stop,
 			.dev_info_get = ioat_dev_info_get,
+			.xstats_get = ioat_xstats_get,
+			.xstats_get_names = ioat_xstats_get_names,
 			.dev_selftest = ioat_rawdev_test,
 	};
 
diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c
index 5375da26c..5098d71ae 100644
--- a/drivers/raw/ioat/ioat_rawdev_test.c
+++ b/drivers/raw/ioat/ioat_rawdev_test.c
@@ -2,6 +2,7 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <inttypes.h>
 #include "rte_rawdev.h"
 #include "rte_ioat_rawdev.h"
 
@@ -13,6 +14,11 @@ ioat_rawdev_test(uint16_t dev_id)
 #define IOAT_TEST_RINGSIZE 512
 	struct rte_ioat_rawdev_config p = { .ring_size = -1 };
 	struct rte_rawdev_info info = { .dev_private = &p };
+	struct rte_rawdev_xstats_name *snames = NULL;
+	uint64_t *stats = NULL;
+	unsigned int *ids = NULL;
+	unsigned int nb_xstats;
+	unsigned int i;
 
 	rte_rawdev_info_get(dev_id, &info);
 	if (p.ring_size != 0) {
@@ -37,5 +43,38 @@ ioat_rawdev_test(uint16_t dev_id)
 		printf("Error with rte_rawdev_start()\n");
 		return -1;
 	}
+
+	/* allocate memory for xstats names and values */
+	nb_xstats = rte_rawdev_xstats_names_get(dev_id, NULL, 0);
+
+	snames = malloc(sizeof(*snames) * nb_xstats);
+	if (snames == NULL) {
+		printf("Error allocating xstat names memory\n");
+		return -1;
+	}
+	rte_rawdev_xstats_names_get(dev_id, snames, nb_xstats);
+
+	ids = malloc(sizeof(*ids) * nb_xstats);
+	if (ids == NULL) {
+		printf("Error allocating xstat ids memory\n");
+		return -1;
+	}
+	for (i = 0; i < nb_xstats; i++)
+		ids[i] = i;
+
+	stats = malloc(sizeof(*stats) * nb_xstats);
+	if (stats == NULL) {
+		printf("Error allocating xstat memory\n");
+		return -1;
+	}
+
+	rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
+	for (i = 0; i < nb_xstats; i++)
+		printf("%s: %"PRIu64"   ", snames[i].name, stats[i]);
+	printf("\n");
+
+	free(snames);
+	free(stats);
+	free(ids);
 	return 0;
 }
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index 0c41a9dcf..90119bc70 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -50,6 +50,12 @@ struct rte_ioat_rawdev {
 	struct rte_ioat_desc *desc_ring;
 	__m128i *hdls; /* completion handles for returning to user */
 
+	/* some statistics for tracking, if added/changed update xstats fns*/
+	uint64_t enqueue_failed __rte_cache_aligned;
+	uint64_t enqueued;
+	uint64_t started;
+	uint64_t completed;
+
 	/* to report completions, the device will write status back here */
 	volatile uint64_t status __rte_cache_aligned;
 };
-- 
2.21.0

[dpdk-dev] [PATCH v2 8/8] raw/ioat: add local API to perform copies

[Bruce Richardson]

Add local APIs to trigger data copies, and retrieve handle values once
those copies are completed. Included are unit tests to validate the data
is copies correctly.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---

V2: test cases moved to self-test routine
---
 doc/guides/rawdevs/ioat_rawdev.rst  | 100 +++++++++++++++++
 drivers/raw/ioat/Makefile           |   1 +
 drivers/raw/ioat/ioat_rawdev_test.c | 161 +++++++++++++++++++++++++-
 drivers/raw/ioat/meson.build        |   2 +-
 drivers/raw/ioat/rte_ioat_rawdev.h  | 168 +++++++++++++++++++++++++++-
 5 files changed, 427 insertions(+), 5 deletions(-)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index b908f31e0..22bb8a22c 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -150,6 +150,106 @@ The following code shows how the device is configured in
 Once configured, the device can then be made ready for use by calling the
 ``rte_rawdev_start()`` API.
 
+Performing Data Copies
+~~~~~~~~~~~~~~~~~~~~~~~
+
+To perform data copies using IOAT rawdev devices, the functions
+``rte_ioat_enqueue_copy()`` and ``rte_ioat_do_copies()`` should be used.
+Once copies have been completed, the completion will be reported back when
+the application calls ``rte_ioat_completed_copies()``.
+
+The ``rte_ioat_enqueue_copy()`` function enqueues a single copy to the
+device ring for copying at a later point. The parameters to that function
+include the physical addresses of both the source and destination buffers,
+as well as two "handles" to be returned to the user when the copy is
+completed. These handles can be arbitrary values, but two are provided so
+that the library can track handles for both source and destination on
+behalf of the user, e.g. virtual addresses for the buffers, or mbuf
+pointers if packet data is being copied.
+
+While the ``rte_ioat_enqueue_copy()`` function enqueues a copy operation on
+the device ring, the copy will not actually be performed until after the
+application calls the ``rte_ioat_do_copies()`` function. This function
+informs the device hardware of the elements enqueued on the ring, and the
+device will begin to process them. It is expected that, for efficiency
+reasons, a burst of operations will be enqueued to the device via multiple
+enqueue calls between calls to the ``rte_ioat_do_copies()`` function.
+
+The following code from ``test_ioat_rawdev.c`` demonstrates how to enqueue
+a burst of copies to the device and start the hardware processing of them:
+
+.. code-block:: C
+
+        struct rte_mbuf *srcs[32], *dsts[32];
+        unsigned int j;
+
+        for (i = 0; i < RTE_DIM(srcs); i++) {
+                char *src_data;
+
+                srcs[i] = rte_pktmbuf_alloc(pool);
+                dsts[i] = rte_pktmbuf_alloc(pool);
+                srcs[i]->data_len = srcs[i]->pkt_len = length;
+                dsts[i]->data_len = dsts[i]->pkt_len = length;
+                src_data = rte_pktmbuf_mtod(srcs[i], char *);
+
+                for (j = 0; j < length; j++)
+                        src_data[j] = rand() & 0xFF;
+
+                if (rte_ioat_enqueue_copy(dev_id,
+                                srcs[i]->buf_iova + srcs[i]->data_off,
+                                dsts[i]->buf_iova + dsts[i]->data_off,
+                                length,
+                                (uintptr_t)srcs[i],
+                                (uintptr_t)dsts[i],
+                                0 /* nofence */) != 1) {
+                        printf("Error with rte_ioat_enqueue_copy for buffer %u\n",
+                                        i);
+                        return -1;
+                }
+        }
+        rte_ioat_do_copies(dev_id);
+
+To retrieve information about completed copies, the API
+``rte_ioat_completed_copies()`` should be used. This API will return to the
+application a set of completion handles passed in when the relevant copies
+were enqueued.
+
+The following code from ``test_ioat_rawdev.c`` shows the test code
+retrieving information about the completed copies and validating the data
+is correct before freeing the data buffers using the returned handles:
+
+.. code-block:: C
+
+        if (rte_ioat_completed_copies(dev_id, 64, (void *)completed_src,
+                        (void *)completed_dst) != RTE_DIM(srcs)) {
+                printf("Error with rte_ioat_completed_copies\n");
+                return -1;
+        }
+        for (i = 0; i < RTE_DIM(srcs); i++) {
+                char *src_data, *dst_data;
+
+                if (completed_src[i] != srcs[i]) {
+                        printf("Error with source pointer %u\n", i);
+                        return -1;
+                }
+                if (completed_dst[i] != dsts[i]) {
+                        printf("Error with dest pointer %u\n", i);
+                        return -1;
+                }
+
+                src_data = rte_pktmbuf_mtod(srcs[i], char *);
+                dst_data = rte_pktmbuf_mtod(dsts[i], char *);
+                for (j = 0; j < length; j++)
+                        if (src_data[j] != dst_data[j]) {
+                                printf("Error with copy of packet %u, byte %u\n",
+                                                i, j);
+                                return -1;
+                        }
+                rte_pktmbuf_free(srcs[i]);
+                rte_pktmbuf_free(dsts[i]);
+        }
+
+
 Querying Device Statistics
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
index b1af9c666..32f079845 100644
--- a/drivers/raw/ioat/Makefile
+++ b/drivers/raw/ioat/Makefile
@@ -12,6 +12,7 @@ CFLAGS += $(WERROR_FLAGS)
 
 LDLIBS += -lrte_eal -lrte_rawdev
 LDLIBS += -lrte_pci -lrte_bus_pci
+LDLIBS += -lrte_mbuf -lrte_mempool
 
 # library version
 LIBABIVER := 1
diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c
index 5098d71ae..8e57cfd7c 100644
--- a/drivers/raw/ioat/ioat_rawdev_test.c
+++ b/drivers/raw/ioat/ioat_rawdev_test.c
@@ -2,12 +2,139 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <unistd.h>
 #include <inttypes.h>
+#include <rte_mbuf.h>
 #include "rte_rawdev.h"
 #include "rte_ioat_rawdev.h"
 
 int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */
 
+static struct rte_mempool *pool;
+
+static int
+test_enqueue_copies(int dev_id)
+{
+	const unsigned int length = 1024;
+	unsigned int i;
+
+	do {
+		struct rte_mbuf *src, *dst;
+		char *src_data, *dst_data;
+		struct rte_mbuf *completed[2] = {0};
+
+		/* test doing a single copy */
+		src = rte_pktmbuf_alloc(pool);
+		dst = rte_pktmbuf_alloc(pool);
+		src->data_len = src->pkt_len = length;
+		dst->data_len = dst->pkt_len = length;
+		src_data = rte_pktmbuf_mtod(src, char *);
+		dst_data = rte_pktmbuf_mtod(dst, char *);
+
+		for (i = 0; i < length; i++)
+			src_data[i] = rand() & 0xFF;
+
+		if (rte_ioat_enqueue_copy(dev_id,
+				src->buf_iova + src->data_off,
+				dst->buf_iova + dst->data_off,
+				length,
+				(uintptr_t)src,
+				(uintptr_t)dst,
+				0 /* no fence */) != 1) {
+			printf("Error with rte_ioat_enqueue_copy\n");
+			return -1;
+		}
+		rte_ioat_do_copies(dev_id);
+		usleep(10);
+
+		if (rte_ioat_completed_copies(dev_id, 1, (void *)&completed[0],
+				(void *)&completed[1]) != 1) {
+			printf("Error with rte_ioat_completed_copies\n");
+			return -1;
+		}
+		if (completed[0] != src || completed[1] != dst) {
+			printf("Error with completions: got (%p, %p), not (%p,%p)\n",
+					completed[0], completed[1], src, dst);
+			return -1;
+		}
+
+		for (i = 0; i < length; i++)
+			if (dst_data[i] != src_data[i]) {
+				printf("Data mismatch at char %u\n", i);
+				return -1;
+			}
+		rte_pktmbuf_free(src);
+		rte_pktmbuf_free(dst);
+	} while (0);
+
+	/* test doing multiple copies */
+	do {
+		struct rte_mbuf *srcs[32], *dsts[32];
+		struct rte_mbuf *completed_src[64];
+		struct rte_mbuf *completed_dst[64];
+		unsigned int j;
+
+		for (i = 0; i < RTE_DIM(srcs); i++) {
+			char *src_data;
+
+			srcs[i] = rte_pktmbuf_alloc(pool);
+			dsts[i] = rte_pktmbuf_alloc(pool);
+			srcs[i]->data_len = srcs[i]->pkt_len = length;
+			dsts[i]->data_len = dsts[i]->pkt_len = length;
+			src_data = rte_pktmbuf_mtod(srcs[i], char *);
+
+			for (j = 0; j < length; j++)
+				src_data[j] = rand() & 0xFF;
+
+			if (rte_ioat_enqueue_copy(dev_id,
+					srcs[i]->buf_iova + srcs[i]->data_off,
+					dsts[i]->buf_iova + dsts[i]->data_off,
+					length,
+					(uintptr_t)srcs[i],
+					(uintptr_t)dsts[i],
+					0 /* nofence */) != 1) {
+				printf("Error with rte_ioat_enqueue_copy for buffer %u\n",
+						i);
+				return -1;
+			}
+		}
+		rte_ioat_do_copies(dev_id);
+		usleep(100);
+
+		if (rte_ioat_completed_copies(dev_id, 64, (void *)completed_src,
+				(void *)completed_dst) != RTE_DIM(srcs)) {
+			printf("Error with rte_ioat_completed_copies\n");
+			return -1;
+		}
+		for (i = 0; i < RTE_DIM(srcs); i++) {
+			char *src_data, *dst_data;
+
+			if (completed_src[i] != srcs[i]) {
+				printf("Error with source pointer %u\n", i);
+				return -1;
+			}
+			if (completed_dst[i] != dsts[i]) {
+				printf("Error with dest pointer %u\n", i);
+				return -1;
+			}
+
+			src_data = rte_pktmbuf_mtod(srcs[i], char *);
+			dst_data = rte_pktmbuf_mtod(dsts[i], char *);
+			for (j = 0; j < length; j++)
+				if (src_data[j] != dst_data[j]) {
+					printf("Error with copy of packet %u, byte %u\n",
+							i, j);
+					return -1;
+				}
+			rte_pktmbuf_free(srcs[i]);
+			rte_pktmbuf_free(dsts[i]);
+		}
+
+	} while (0);
+
+	return 0;
+}
+
 int
 ioat_rawdev_test(uint16_t dev_id)
 {
@@ -44,6 +171,17 @@ ioat_rawdev_test(uint16_t dev_id)
 		return -1;
 	}
 
+	pool = rte_pktmbuf_pool_create("TEST_IOAT_POOL",
+			256, /* n == num elements */
+			32,  /* cache size */
+			0,   /* priv size */
+			2048, /* data room size */
+			info.socket_id);
+	if (pool == NULL) {
+		printf("Error with mempool creation\n");
+		return -1;
+	}
+
 	/* allocate memory for xstats names and values */
 	nb_xstats = rte_rawdev_xstats_names_get(dev_id, NULL, 0);
 
@@ -68,13 +206,30 @@ ioat_rawdev_test(uint16_t dev_id)
 		return -1;
 	}
 
-	rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
-	for (i = 0; i < nb_xstats; i++)
-		printf("%s: %"PRIu64"   ", snames[i].name, stats[i]);
+	/* run the test cases */
+	for (i = 0; i < 100; i++) {
+		unsigned int j;
+
+		if (test_enqueue_copies(dev_id) != 0)
+			goto err;
+
+		rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
+		for (j = 0; j < nb_xstats; j++)
+			printf("%s: %"PRIu64"   ", snames[j].name, stats[j]);
+		printf("\r");
+	}
 	printf("\n");
 
+	rte_mempool_free(pool);
 	free(snames);
 	free(stats);
 	free(ids);
 	return 0;
+
+err:
+	rte_mempool_free(pool);
+	free(snames);
+	free(stats);
+	free(ids);
+	return -1;
 }
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
index 40fff6654..247ff88bf 100644
--- a/drivers/raw/ioat/meson.build
+++ b/drivers/raw/ioat/meson.build
@@ -4,7 +4,7 @@
 build = dpdk_conf.has('RTE_ARCH_X86')
 sources = files('ioat_rawdev.c',
 		'ioat_rawdev_test.c')
-deps += ['rawdev', 'bus_pci']
+deps += ['rawdev', 'bus_pci', 'mbuf']
 
 install_headers('rte_ioat_rawdev.h',
 		'rte_ioat_spec.h')
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index 90119bc70..126d52e6d 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -15,7 +15,9 @@
  */
 
 #include <x86intrin.h>
+#include <rte_atomic.h>
 #include <rte_memory.h>
+#include <rte_prefetch.h>
 #include <rte_ioat_spec.h>
 
 /** Name of the device driver */
@@ -50,6 +52,10 @@ struct rte_ioat_rawdev {
 	struct rte_ioat_desc *desc_ring;
 	__m128i *hdls; /* completion handles for returning to user */
 
+
+	unsigned short next_read;
+	unsigned short next_write;
+
 	/* some statistics for tracking, if added/changed update xstats fns*/
 	uint64_t enqueue_failed __rte_cache_aligned;
 	uint64_t enqueued;
@@ -73,4 +79,164 @@ struct rte_ioat_desc {
 	uint64_t op_type_specific[4];
 };
 
-#endif
+#define RTE_IOAT_ENABLE_COMPLETION (1 << 3) /* set in control field */
+
+/**
+ * Enqueue a copy operation onto the ioat device
+ *
+ * This queues up a copy operation to be performed by hardware, but does not
+ * trigger hardware to begin that operation.
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ * @param src
+ *   The physical address of the source buffer
+ * @param dst
+ *   The physical address of the destination buffer
+ * @param length
+ *   The length of the data to be copied
+ * @param src_hdl
+ *   An opaque handle for the source data, to be returned when this operation
+ *   has been completed and the user polls for the completion details
+ * @param dst_hdl
+ *   An opaque handle for the destination data, to be returned when this
+ *   operation has been completed and the user polls for the completion details
+ * @param fence
+ *   A flag parameter indicating that hardware should not begin to perform any
+ *   subsequently enqueued copy operations until after this operation has
+ *   completed
+ * @return
+ *   Number of operations enqueued, either 0 or 1
+ */
+static inline int
+rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst,
+		unsigned int length, uintptr_t src_hdl, uintptr_t dst_hdl,
+		int fence)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	unsigned short read = ioat->next_read;
+	unsigned short write = ioat->next_write;
+	unsigned short mask = ioat->ring_size - 1;
+	unsigned short space = mask + read - write;
+
+	if (space == 0) {
+		ioat->enqueue_failed++;
+		return 0;
+	}
+
+	ioat->next_write = write + 1;
+	write &= mask;
+	/* write transfer size and the descriptor control bits */
+	*((uint64_t *)&ioat->desc_ring[write]) =
+			length | (uint64_t)!!fence << 36 |
+			(uint64_t)(!(write & 0xF)) << 35;
+	ioat->desc_ring[write].src_addr = src;
+	ioat->desc_ring[write].dest_addr = dst;
+	ioat->hdls[write] = _mm_set_epi64((__m64)((uint64_t)dst_hdl),
+			(__m64)((uint64_t)src_hdl));
+	rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]);
+
+	ioat->enqueued++;
+	return 1;
+}
+
+/**
+ * Trigger hardware to begin performing enqueued copy operations
+ *
+ * This API is used to write the "doorbell" to the hardware to trigger it
+ * to begin the copy operations previously enqueued by rte_ioat_enqueue_copy()
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ */
+static inline void
+rte_ioat_do_copies(int dev_id)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	ioat->desc_ring[(ioat->next_write - 1) & (ioat->ring_size - 1)]
+			.desc_control = RTE_IOAT_ENABLE_COMPLETION;
+	rte_compiler_barrier();
+	ioat->regs->dmacount = ioat->next_write;
+	ioat->started = ioat->enqueued;
+}
+
+/**
+ * @internal
+ * Returns the index of the last completed operation.
+ */
+static inline int
+rte_ioat_get_last_completed(struct rte_ioat_rawdev *ioat, int *error)
+{
+	uint64_t status = ioat->status;
+
+	/* lower 3 bits indicate "transfer status" : active, idle, halted.
+	 * We can ignore bit 0.
+	 */
+	*error = status & (RTE_IOAT_CHANSTS_SUSPENDED | RTE_IOAT_CHANSTS_ARMED);
+	return (status - ioat->ring_addr) >> 6;
+}
+
+/**
+ * Returns details of copy operations that have been completed
+ *
+ * Returns to the caller the user-provided "handles" for the copy operations
+ * which have been completed by the hardware, and not already returned by
+ * a previous call to this API.
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ * @param max_copies
+ *   The number of entries which can fit in the src_hdls and dst_hdls
+ *   arrays, i.e. max number of completed operations to report
+ * @param src_hdls
+ *   Array to hold the source handle parameters of the completed copies
+ * @param dst_hdls
+ *   Array to hold the destination handle parameters of the completed copies
+ * @return
+ *   -1 on error, with rte_errno set appropriately.
+ *   Otherwise number of completed operations i.e. number of entries written
+ *   to the src_hdls and dst_hdls array parameters.
+ */
+static inline int
+rte_ioat_completed_copies(int dev_id, uint8_t max_copies,
+		uintptr_t *src_hdls, uintptr_t *dst_hdls)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	unsigned short mask = (ioat->ring_size - 1);
+	unsigned short read = ioat->next_read;
+	unsigned short end_read, count;
+	int error;
+	int i = 0;
+
+	end_read = (rte_ioat_get_last_completed(ioat, &error) + 1) & mask;
+	count = (end_read - (read & mask)) & mask;
+
+	if (error) {
+		rte_errno = EIO;
+		return -1;
+	}
+
+	if (count > max_copies)
+		count = max_copies;
+
+	for (; i < count - 1; i += 2, read += 2) {
+		__m128i hdls0 = _mm_load_si128(&ioat->hdls[read & mask]);
+		__m128i hdls1 = _mm_load_si128(&ioat->hdls[(read + 1) & mask]);
+
+		_mm_storeu_si128((void *)&src_hdls[i],
+				_mm_unpacklo_epi64(hdls0, hdls1));
+		_mm_storeu_si128((void *)&dst_hdls[i],
+				_mm_unpackhi_epi64(hdls0, hdls1));
+	}
+	for (; i < count; i++, read++) {
+		uintptr_t *hdls = (void *)&ioat->hdls[read & mask];
+		src_hdls[i] = hdls[0];
+		dst_hdls[i] = hdls[1];
+	}
+
+	ioat->next_read = read;
+	ioat->completed += count;
+	return count;
+}
+
+#endif /* _RTE_IOAT_RAWDEV_H_ */
-- 
2.21.0

[dpdk-dev] [PATCH v3 0/8] raw/ioat: driver for Intel QuickData Technology

[Bruce Richardson]

This patch series adds support for the Intel QuickData Technology
device, part of the Intel I/O Acceleration Technology (Intel I/OAT). It
is a raw device for allowing hardware DMA i.e. data copies in hardware.

Performing the copies in hardware can provide performance improvements
for applications where the average copy size is reasonably large, e.g.
1k packets. For smaller packets, e.g. 64-256 bytes, offloading the copy
may reduce performance due to the overhead of using hardware.

V3:
 * removed DPDK-specific structure for the descriptor format and reused
   the structure in the imported file rte_ioat_spec.h

V2:
* moved tests to rawdev selftest function
* some checkpatch and other small cleanups
* added extra documentation details on supported hardware
* aligned the changes to dpdk-devbind with the changes in the NTB set
  for consistency

Bruce Richardson (8):
  raw/ioat: add initial support for ioat rawdev driver
  usertools/dpdk-devbind.py: add support for IOAT devices
  raw/ioat: add register definition file
  raw/ioat: create device on probe and destroy on release
  raw/ioat: add device info function
  raw/ioat: add configure, start and stop functions
  raw/ioat: add statistics functions
  raw/ioat: add local API to perform copies

 MAINTAINERS                                 |   6 +-
 app/test/test_rawdev.c                      |  19 ++
 config/common_armv8a_linux                  |   1 +
 config/common_base                          |   5 +
 config/defconfig_arm-armv7a-linuxapp-gcc    |   1 +
 config/defconfig_ppc_64-power8-linuxapp-gcc |   1 +
 doc/guides/rawdevs/index.rst                |   1 +
 doc/guides/rawdevs/ioat_rawdev.rst          | 265 +++++++++++++++++
 doc/guides/rel_notes/release_19_08.rst      |  11 +
 drivers/raw/Makefile                        |   1 +
 drivers/raw/ioat/Makefile                   |  31 ++
 drivers/raw/ioat/ioat_rawdev.c              | 313 ++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev_test.c         | 235 +++++++++++++++
 drivers/raw/ioat/meson.build                |  10 +
 drivers/raw/ioat/rte_ioat_rawdev.h          | 230 ++++++++++++++
 drivers/raw/ioat/rte_ioat_spec.h            | 301 +++++++++++++++++++
 drivers/raw/ioat/rte_pmd_ioat_version.map   |   4 +
 drivers/raw/meson.build                     |   4 +-
 mk/rte.app.mk                               |   1 +
 usertools/dpdk-devbind.py                   |  10 +
 20 files changed, 1448 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/rawdevs/ioat_rawdev.rst
 create mode 100644 drivers/raw/ioat/Makefile
 create mode 100644 drivers/raw/ioat/ioat_rawdev.c
 create mode 100644 drivers/raw/ioat/ioat_rawdev_test.c
 create mode 100644 drivers/raw/ioat/meson.build
 create mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h
 create mode 100644 drivers/raw/ioat/rte_ioat_spec.h
 create mode 100644 drivers/raw/ioat/rte_pmd_ioat_version.map

-- 
2.21.0

[dpdk-dev] [PATCH v3 1/8] raw/ioat: add initial support for ioat rawdev driver

[Bruce Richardson]

Add stubs for ioat rawdev driver support in DPDK, specifically:

  * makefile and meson build hooks
  * initial public header file
  * rawdev main C file, with probe and release functions
  * release note update announcing the driver
  * initial documentation for the new section in the rawdev doc
  * unit test stubs for device unit tests

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---
V2: don't create a new file for unit testing, add to existing rawdev test
    file, and place test cases in the driver selftest routine (added later
    in set)
    Add new section in document about identifying hardware using lspci
---
 MAINTAINERS                                 |  6 +-
 app/test/test_rawdev.c                      |  8 ++
 config/common_armv8a_linux                  |  1 +
 config/common_base                          |  5 ++
 config/defconfig_arm-armv7a-linuxapp-gcc    |  1 +
 config/defconfig_ppc_64-power8-linuxapp-gcc |  1 +
 doc/guides/rawdevs/index.rst                |  1 +
 doc/guides/rawdevs/ioat_rawdev.rst          | 63 ++++++++++++++
 doc/guides/rel_notes/release_19_08.rst      | 11 +++
 drivers/raw/Makefile                        |  1 +
 drivers/raw/ioat/Makefile                   | 28 +++++++
 drivers/raw/ioat/ioat_rawdev.c              | 93 +++++++++++++++++++++
 drivers/raw/ioat/meson.build                |  8 ++
 drivers/raw/ioat/rte_ioat_rawdev.h          | 24 ++++++
 drivers/raw/ioat/rte_pmd_ioat_version.map   |  4 +
 drivers/raw/meson.build                     |  4 +-
 mk/rte.app.mk                               |  1 +
 17 files changed, 258 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/rawdevs/ioat_rawdev.rst
 create mode 100644 drivers/raw/ioat/Makefile
 create mode 100644 drivers/raw/ioat/ioat_rawdev.c
 create mode 100644 drivers/raw/ioat/meson.build
 create mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h
 create mode 100644 drivers/raw/ioat/rte_pmd_ioat_version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 0c3b48920..f28c526bc 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1068,6 +1068,11 @@ M: Tianfei zhang <tianfei.zhang@intel.com>
 F: drivers/raw/ifpga_rawdev/
 F: doc/guides/rawdevs/ifpga_rawdev.rst
 
+IOAT Rawdev
+M: Bruce Richardson <bruce.richardson@intel.com>
+F: drivers/raw/ioat/
+F: doc/guides/rawdevs/ioat_rawdev.rst
+
 NXP DPAA2 QDMA
 M: Nipun Gupta <nipun.gupta@nxp.com>
 F: drivers/raw/dpaa2_qdma/
@@ -1078,7 +1083,6 @@ M: Nipun Gupta <nipun.gupta@nxp.com>
 F: drivers/raw/dpaa2_cmdif/
 F: doc/guides/rawdevs/dpaa2_cmdif.rst
 
-
 Packet processing
 -----------------
 
diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
index 043a38a13..88549fb61 100644
--- a/app/test/test_rawdev.c
+++ b/app/test/test_rawdev.c
@@ -25,3 +25,11 @@ test_rawdev_selftest_skeleton(void)
 }
 
 REGISTER_TEST_COMMAND(rawdev_autotest, test_rawdev_selftest_skeleton);
+
+static int
+test_rawdev_selftest_ioat(void)
+{
+	return TEST_SKIPPED;
+}
+
+REGISTER_TEST_COMMAND(ioat_rawdev_autotest, test_rawdev_selftest_ioat);
diff --git a/config/common_armv8a_linux b/config/common_armv8a_linux
index 72091de1c..481712ebc 100644
--- a/config/common_armv8a_linux
+++ b/config/common_armv8a_linux
@@ -34,5 +34,6 @@ CONFIG_RTE_ARCH_ARM64_MEMCPY=n
 CONFIG_RTE_LIBRTE_FM10K_PMD=n
 CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
 
 CONFIG_RTE_SCHED_VECTOR=n
diff --git a/config/common_base b/config/common_base
index fa1ae249a..e6b830923 100644
--- a/config/common_base
+++ b/config/common_base
@@ -747,6 +747,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n
 #
 CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
 
+#
+# Compile PMD for Intel IOAT raw device
+#
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=y
+
 #
 # Compile librte_ring
 #
diff --git a/config/defconfig_arm-armv7a-linuxapp-gcc b/config/defconfig_arm-armv7a-linuxapp-gcc
index c9509b274..ee158ef9d 100644
--- a/config/defconfig_arm-armv7a-linuxapp-gcc
+++ b/config/defconfig_arm-armv7a-linuxapp-gcc
@@ -54,3 +54,4 @@ CONFIG_RTE_LIBRTE_QEDE_PMD=n
 CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
 CONFIG_RTE_LIBRTE_NFP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
diff --git a/config/defconfig_ppc_64-power8-linuxapp-gcc b/config/defconfig_ppc_64-power8-linuxapp-gcc
index 7e248b755..9f3670ec0 100644
--- a/config/defconfig_ppc_64-power8-linuxapp-gcc
+++ b/config/defconfig_ppc_64-power8-linuxapp-gcc
@@ -56,3 +56,4 @@ CONFIG_RTE_LIBRTE_ENIC_PMD=n
 CONFIG_RTE_LIBRTE_FM10K_PMD=n
 CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
diff --git a/doc/guides/rawdevs/index.rst b/doc/guides/rawdevs/index.rst
index 7c3bd9586..0a21989e4 100644
--- a/doc/guides/rawdevs/index.rst
+++ b/doc/guides/rawdevs/index.rst
@@ -14,3 +14,4 @@ application through rawdev API.
     dpaa2_cmdif
     dpaa2_qdma
     ifpga_rawdev
+    ioat_rawdev
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
new file mode 100644
index 000000000..0c612e73a
--- /dev/null
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -0,0 +1,63 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+.. include:: <isonum.txt>
+
+IOAT Rawdev Driver for Intel\ |reg| QuickData Technology
+======================================================================
+
+The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg|
+QuickData Technology, part of Intel\ |reg| I/O Acceleration Technology
+`(Intel I/OAT)
+<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_.
+This PMD, when used on supported hardware, allows data copies, for example,
+cloning packet data, to be accelerated by that hardware rather than having to
+be done by software, freeing up CPU cycles for other tasks.
+
+Hardware Requirements
+----------------------
+
+On Linux, the presence of an Intel\ |reg| QuickData Technology hardware can
+be detected by checking the output of the ``lspci`` command, where the
+hardware will be often listed as "Crystal Beach DMA" or "CBDMA". For
+example, on a system with Intel\ |reg| Xeon\ |reg| CPU E5-2699 v4 @2.20GHz,
+lspci shows:
+
+.. code-block:: console
+
+  # lspci | grep DMA
+  00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01)
+  00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01)
+  00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01)
+  00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01)
+  00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01)
+  00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01)
+  00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01)
+  00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01)
+
+On a system with Intel\ |reg| Xeon\ |reg| Gold 6154 CPU @ 3.00GHz, lspci
+shows:
+
+.. code-block:: console
+
+  # lspci | grep DMA
+  00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+
+
+Compilation
+------------
+
+For builds done with ``make``, the driver compilation is enabled by the
+``CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV`` build configuration option. This is
+enabled by default in builds for x86 platforms, and disabled in other
+configurations.
+
+For builds using ``meson`` and ``ninja``, the driver will be built when the
+target platform is x86-based.
diff --git a/doc/guides/rel_notes/release_19_08.rst b/doc/guides/rel_notes/release_19_08.rst
index 3da266705..720c13f8b 100644
--- a/doc/guides/rel_notes/release_19_08.rst
+++ b/doc/guides/rel_notes/release_19_08.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright 2019 The DPDK contributors
 
+.. include:: <isonum.txt>
+
 DPDK Release 19.08
 ==================
 
@@ -99,6 +101,15 @@ New Features
   Updated ``librte_telemetry`` to fetch the global metrics from the
   ``librte_metrics`` library.
 
+* **Added Intel QuickData Technology PMD**
+
+  The PMD for Intel\ |reg|  QuickData Technology, part of
+  Intel\ |reg|  I/O Acceleration Technology `(Intel I/OAT)
+  <https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_,
+  allows data copies to be done by hardware instead
+  of via software, reducing cycles spent copying large blocks of data in
+  applications.
+
 
 Removed Items
 -------------
diff --git a/drivers/raw/Makefile b/drivers/raw/Makefile
index 8e29b4a56..c1b85c8c7 100644
--- a/drivers/raw/Makefile
+++ b/drivers/raw/Makefile
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev
+DIRS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat
 
 include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
new file mode 100644
index 000000000..7726e310a
--- /dev/null
+++ b/drivers/raw/ioat/Makefile
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_pmd_ioat_rawdev.a
+
+# build flags
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+
+LDLIBS += -lrte_eal -lrte_rawdev
+LDLIBS += -lrte_pci -lrte_bus_pci
+
+# library version
+LIBABIVER := 1
+
+# versioning export map
+EXPORT_MAP := rte_pmd_ioat_version.map
+
+# library source files
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
+
+# export include files
+SYMLINK-y-include += rte_ioat_rawdev.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
new file mode 100644
index 000000000..d9fc3091a
--- /dev/null
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -0,0 +1,93 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include <rte_bus_pci.h>
+#include <rte_rawdev_pmd.h>
+
+#include "rte_ioat_rawdev.h"
+
+/* Dynamic log type identifier */
+int ioat_pmd_logtype;
+
+static struct rte_pci_driver ioat_pmd_drv;
+
+#define IOAT_VENDOR_ID		0x8086
+#define IOAT_DEVICE_ID		0x2021
+
+#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \
+	ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args)
+
+#define IOAT_PMD_DEBUG(fmt, args...)  IOAT_PMD_LOG(DEBUG, fmt, ## args)
+#define IOAT_PMD_INFO(fmt, args...)   IOAT_PMD_LOG(INFO, fmt, ## args)
+#define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
+#define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
+
+static int
+ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
+{
+	RTE_SET_USED(name);
+	RTE_SET_USED(dev);
+	return 0;
+}
+
+static int
+ioat_rawdev_destroy(const char *name)
+{
+	RTE_SET_USED(name);
+	return 0;
+}
+
+static int
+ioat_rawdev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev)
+{
+	char name[32];
+	int ret = 0;
+
+
+	rte_pci_device_name(&dev->addr, name, sizeof(name));
+	IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node);
+
+	dev->device.driver = &drv->driver;
+	ret = ioat_rawdev_create(name, dev);
+	return ret;
+}
+
+static int
+ioat_rawdev_remove(struct rte_pci_device *dev)
+{
+	char name[32];
+	int ret;
+
+	rte_pci_device_name(&dev->addr, name, sizeof(name));
+
+	IOAT_PMD_INFO("Closing %s on NUMA node %d",
+			name, dev->device.numa_node);
+
+	ret = ioat_rawdev_destroy(name);
+	return ret;
+}
+
+static const struct rte_pci_id pci_id_ioat_map[] = {
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID) },
+	{ .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver ioat_pmd_drv = {
+	.id_table = pci_id_ioat_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+		     RTE_PCI_DRV_IOVA_AS_VA,
+	.probe = ioat_rawdev_probe,
+	.remove = ioat_rawdev_remove,
+};
+
+RTE_PMD_REGISTER_PCI(IOAT_PMD_RAWDEV_NAME, ioat_pmd_drv);
+RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_RAWDEV_NAME, pci_id_ioat_map);
+RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_RAWDEV_NAME, "* igb_uio | uio_pci_generic");
+
+RTE_INIT(ioat_pmd_init_log)
+{
+	ioat_pmd_logtype = rte_log_register(IOAT_PMD_LOG_NAME);
+	if (ioat_pmd_logtype >= 0)
+		rte_log_set_level(ioat_pmd_logtype, RTE_LOG_INFO);
+}
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
new file mode 100644
index 000000000..ba7620a68
--- /dev/null
+++ b/drivers/raw/ioat/meson.build
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2019 Intel Corporation
+
+build = dpdk_conf.has('RTE_ARCH_X86')
+sources = files('ioat_rawdev.c')
+deps += ['rawdev', 'bus_pci']
+
+install_headers('rte_ioat_rawdev.h')
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
new file mode 100644
index 000000000..e77406403
--- /dev/null
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_IOAT_RAWDEV_H_
+#define _RTE_IOAT_RAWDEV_H_
+
+/**
+ * @file rte_ioat_rawdev.h
+ *
+ * Definitions for using the ioat rawdev device driver
+ *
+ * @warning
+ * @b EXPERIMENTAL: these structures and APIs may change without prior notice
+ */
+
+/** Name of the device driver */
+#define IOAT_PMD_RAWDEV_NAME rawdev_ioat
+/** String reported as the device driver name by rte_rawdev_info_get() */
+#define IOAT_PMD_RAWDEV_NAME_STR "rawdev_ioat"
+/** Name used to adjust the log level for this driver */
+#define IOAT_PMD_LOG_NAME "rawdev.ioat"
+
+#endif
diff --git a/drivers/raw/ioat/rte_pmd_ioat_version.map b/drivers/raw/ioat/rte_pmd_ioat_version.map
new file mode 100644
index 000000000..9a61188cd
--- /dev/null
+++ b/drivers/raw/ioat/rte_pmd_ioat_version.map
@@ -0,0 +1,4 @@
+DPDK_19.08 {
+
+	local: *;
+};
diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build
index a61cdccef..2af8a70d4 100644
--- a/drivers/raw/meson.build
+++ b/drivers/raw/meson.build
@@ -1,7 +1,9 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2018 NXP
 
-drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
+drivers = ['dpaa2_cmdif', 'dpaa2_qdma',
+	'ifpga_rawdev', 'ioat',
+	'skeleton_rawdev']
 std_deps = ['rawdev']
 config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
 driver_name_fmt = 'rte_pmd_@0@'
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 81be289a8..2a534796f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -306,6 +306,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV)   += -lrte_pmd_ifpga_rawdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD)       += -lrte_pmd_ipn3ke
 endif # CONFIG_RTE_LIBRTE_IFPGA_BUS
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV)   += -lrte_pmd_ioat_rawdev
 endif # CONFIG_RTE_LIBRTE_RAWDEV
 
 endif # !CONFIG_RTE_BUILD_SHARED_LIBS
-- 
2.21.0

Re: [dpdk-dev] [PATCH v3 1/8] raw/ioat: add initial support for ioat rawdev driver

[Burakov, Anatoly]

On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> Add stubs for ioat rawdev driver support in DPDK, specifically:
> 
>    * makefile and meson build hooks
>    * initial public header file
>    * rawdev main C file, with probe and release functions
>    * release note update announcing the driver
>    * initial documentation for the new section in the rawdev doc
>    * unit test stubs for device unit tests
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> 
> ---

<snip>

>   NXP DPAA2 QDMA
>   M: Nipun Gupta <nipun.gupta@nxp.com>
>   F: drivers/raw/dpaa2_qdma/
> @@ -1078,7 +1083,6 @@ M: Nipun Gupta <nipun.gupta@nxp.com>
>   F: drivers/raw/dpaa2_cmdif/
>   F: doc/guides/rawdevs/dpaa2_cmdif.rst
>   
> -
>   Packet processing

Unnecessary whitespace change :)

>   -----------------
>   
> diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
> index 043a38a13..88549fb61 100644
> --- a/app/test/test_rawdev.c
> +++ b/app/test/test_rawdev.c
> @@ -25,3 +25,11 @@ test_rawdev_selftest_skeleton(void)
>   }

<snip>

> diff --git a/doc/guides/rel_notes/release_19_08.rst b/doc/guides/rel_notes/release_19_08.rst
> index 3da266705..720c13f8b 100644
> --- a/doc/guides/rel_notes/release_19_08.rst
> +++ b/doc/guides/rel_notes/release_19_08.rst
> @@ -1,6 +1,8 @@
>   ..  SPDX-License-Identifier: BSD-3-Clause
>       Copyright 2019 The DPDK contributors
>   
> +.. include:: <isonum.txt>
> +

Is this accidental? This doesn't look related to this patch.


Other than a couple of nitpicks, LGTM

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3 1/8] raw/ioat: add initial support for ioat rawdev driver

[Bruce Richardson]

On Thu, Jun 27, 2019 at 12:55:12PM +0100, Burakov, Anatoly wrote:
> On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> > Add stubs for ioat rawdev driver support in DPDK, specifically:
> > 
> >    * makefile and meson build hooks
> >    * initial public header file
> >    * rawdev main C file, with probe and release functions
> >    * release note update announcing the driver
> >    * initial documentation for the new section in the rawdev doc
> >    * unit test stubs for device unit tests
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > 
> > ---
> 
> <snip>
> 
> >   NXP DPAA2 QDMA
> >   M: Nipun Gupta <nipun.gupta@nxp.com>
> >   F: drivers/raw/dpaa2_qdma/
> > @@ -1078,7 +1083,6 @@ M: Nipun Gupta <nipun.gupta@nxp.com>
> >   F: drivers/raw/dpaa2_cmdif/
> >   F: doc/guides/rawdevs/dpaa2_cmdif.rst
> > -
> >   Packet processing
> 
> Unnecessary whitespace change :)
> 

Thanks, good catch. 

> >   -----------------
> > diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
> > index 043a38a13..88549fb61 100644
> > --- a/app/test/test_rawdev.c
> > +++ b/app/test/test_rawdev.c
> > @@ -25,3 +25,11 @@ test_rawdev_selftest_skeleton(void)
> >   }
> 
> <snip>
> 
> > diff --git a/doc/guides/rel_notes/release_19_08.rst b/doc/guides/rel_notes/release_19_08.rst
> > index 3da266705..720c13f8b 100644
> > --- a/doc/guides/rel_notes/release_19_08.rst
> > +++ b/doc/guides/rel_notes/release_19_08.rst
> > @@ -1,6 +1,8 @@
> >   ..  SPDX-License-Identifier: BSD-3-Clause
> >       Copyright 2019 The DPDK contributors
> > +.. include:: <isonum.txt>
> > +
> 
> Is this accidental? This doesn't look related to this patch.
> 
No, it's not accidental. It's the sphinx / rst include which allows the use
of the actual symbols for (c) (R) etc. I make use of those symbols in the
official product names in the following documentation. [It's a useful
include that I think should always be put in our docs.]

Re: [dpdk-dev] [PATCH v3 1/8] raw/ioat: add initial support for ioat rawdev driver

[Hu, Jiayu]

Hi Bruce,

-----Original Message-----
From: dev <dev-bounces@dpdk.org> On Behalf Of Bruce Richardson
Sent: Thursday, June 27, 2019 6:41 PM
To: dev@dpdk.org
Cc: thomas@monjalon.net; jerinj@marvell.com; Richardson, Bruce <bruce.richardson@intel.com>
Subject: [dpdk-dev] [PATCH v3 1/8] raw/ioat: add initial support for ioat rawdev driver

Add stubs for ioat rawdev driver support in DPDK, specifically:

  * makefile and meson build hooks
  * initial public header file
  * rawdev main C file, with probe and release functions
  * release note update announcing the driver
  * initial documentation for the new section in the rawdev doc
  * unit test stubs for device unit tests

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---
V2: don't create a new file for unit testing, add to existing rawdev test
    file, and place test cases in the driver selftest routine (added later
    in set)
    Add new section in document about identifying hardware using lspci
---
 MAINTAINERS                                 |  6 +-
 app/test/test_rawdev.c                      |  8 ++
 config/common_armv8a_linux                  |  1 +
 config/common_base                          |  5 ++
 config/defconfig_arm-armv7a-linuxapp-gcc    |  1 +
 config/defconfig_ppc_64-power8-linuxapp-gcc |  1 +
 doc/guides/rawdevs/index.rst                |  1 +
 doc/guides/rawdevs/ioat_rawdev.rst          | 63 ++++++++++++++
 doc/guides/rel_notes/release_19_08.rst      | 11 +++
 drivers/raw/Makefile                        |  1 +
 drivers/raw/ioat/Makefile                   | 28 +++++++
 drivers/raw/ioat/ioat_rawdev.c              | 93 +++++++++++++++++++++
 drivers/raw/ioat/meson.build                |  8 ++
 drivers/raw/ioat/rte_ioat_rawdev.h          | 24 ++++++
 drivers/raw/ioat/rte_pmd_ioat_version.map   |  4 +
 drivers/raw/meson.build                     |  4 +-
 mk/rte.app.mk                               |  1 +
 17 files changed, 258 insertions(+), 2 deletions(-)  create mode 100644 doc/guides/rawdevs/ioat_rawdev.rst
 create mode 100644 drivers/raw/ioat/Makefile  create mode 100644 drivers/raw/ioat/ioat_rawdev.c  create mode 100644 drivers/raw/ioat/meson.build  create mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h
 create mode 100644 drivers/raw/ioat/rte_pmd_ioat_version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 0c3b48920..f28c526bc 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1068,6 +1068,11 @@ M: Tianfei zhang <tianfei.zhang@intel.com>
 F: drivers/raw/ifpga_rawdev/
 F: doc/guides/rawdevs/ifpga_rawdev.rst
 
+IOAT Rawdev
+M: Bruce Richardson <bruce.richardson@intel.com>
+F: drivers/raw/ioat/
+F: doc/guides/rawdevs/ioat_rawdev.rst
+
 NXP DPAA2 QDMA
 M: Nipun Gupta <nipun.gupta@nxp.com>
 F: drivers/raw/dpaa2_qdma/
@@ -1078,7 +1083,6 @@ M: Nipun Gupta <nipun.gupta@nxp.com>
 F: drivers/raw/dpaa2_cmdif/
 F: doc/guides/rawdevs/dpaa2_cmdif.rst
 
-
 Packet processing
 -----------------
 
diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c index 043a38a13..88549fb61 100644
--- a/app/test/test_rawdev.c
+++ b/app/test/test_rawdev.c
@@ -25,3 +25,11 @@ test_rawdev_selftest_skeleton(void)
 }
 
 REGISTER_TEST_COMMAND(rawdev_autotest, test_rawdev_selftest_skeleton);
+
+static int
+test_rawdev_selftest_ioat(void)
+{
+	return TEST_SKIPPED;
+}
+
+REGISTER_TEST_COMMAND(ioat_rawdev_autotest, test_rawdev_selftest_ioat);
diff --git a/config/common_armv8a_linux b/config/common_armv8a_linux index 72091de1c..481712ebc 100644
--- a/config/common_armv8a_linux
+++ b/config/common_armv8a_linux
@@ -34,5 +34,6 @@ CONFIG_RTE_ARCH_ARM64_MEMCPY=n  CONFIG_RTE_LIBRTE_FM10K_PMD=n  CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n  CONFIG_RTE_LIBRTE_AVP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
 
 CONFIG_RTE_SCHED_VECTOR=n
diff --git a/config/common_base b/config/common_base index fa1ae249a..e6b830923 100644
--- a/config/common_base
+++ b/config/common_base
@@ -747,6 +747,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n
 #
 CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
 
+#
+# Compile PMD for Intel IOAT raw device # 
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=y
+
 #
 # Compile librte_ring
 #
diff --git a/config/defconfig_arm-armv7a-linuxapp-gcc b/config/defconfig_arm-armv7a-linuxapp-gcc
index c9509b274..ee158ef9d 100644
--- a/config/defconfig_arm-armv7a-linuxapp-gcc
+++ b/config/defconfig_arm-armv7a-linuxapp-gcc
@@ -54,3 +54,4 @@ CONFIG_RTE_LIBRTE_QEDE_PMD=n  CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n  CONFIG_RTE_LIBRTE_AVP_PMD=n  CONFIG_RTE_LIBRTE_NFP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
diff --git a/config/defconfig_ppc_64-power8-linuxapp-gcc b/config/defconfig_ppc_64-power8-linuxapp-gcc
index 7e248b755..9f3670ec0 100644
--- a/config/defconfig_ppc_64-power8-linuxapp-gcc
+++ b/config/defconfig_ppc_64-power8-linuxapp-gcc
@@ -56,3 +56,4 @@ CONFIG_RTE_LIBRTE_ENIC_PMD=n  CONFIG_RTE_LIBRTE_FM10K_PMD=n  CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n  CONFIG_RTE_LIBRTE_AVP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
diff --git a/doc/guides/rawdevs/index.rst b/doc/guides/rawdevs/index.rst index 7c3bd9586..0a21989e4 100644
--- a/doc/guides/rawdevs/index.rst
+++ b/doc/guides/rawdevs/index.rst
@@ -14,3 +14,4 @@ application through rawdev API.
     dpaa2_cmdif
     dpaa2_qdma
     ifpga_rawdev
+    ioat_rawdev
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
new file mode 100644
index 000000000..0c612e73a
--- /dev/null
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -0,0 +1,63 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+.. include:: <isonum.txt>
+
+IOAT Rawdev Driver for Intel\ |reg| QuickData Technology 
+======================================================================
+
+The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ 
+|reg| QuickData Technology, part of Intel\ |reg| I/O Acceleration 
+Technology `(Intel I/OAT) 
+<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_.
+This PMD, when used on supported hardware, allows data copies, for 
+example, cloning packet data, to be accelerated by that hardware rather 
+than having to be done by software, freeing up CPU cycles for other tasks.
+
+Hardware Requirements
+----------------------
+
+On Linux, the presence of an Intel\ |reg| QuickData Technology hardware 
+can be detected by checking the output of the ``lspci`` command, where 
+the hardware will be often listed as "Crystal Beach DMA" or "CBDMA". 
+For example, on a system with Intel\ |reg| Xeon\ |reg| CPU E5-2699 v4 
+@2.20GHz, lspci shows:
+
+.. code-block:: console
+
+  # lspci | grep DMA
+  00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01)
+  00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01)
+  00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01)
+  00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01)
+  00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01)
+  00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01)
+  00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01)
+  00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01)
+
+On a system with Intel\ |reg| Xeon\ |reg| Gold 6154 CPU @ 3.00GHz, 
+lspci
+shows:
+
+.. code-block:: console
+
+  # lspci | grep DMA
+  00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+  00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+  00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+  00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+  00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+  00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+  00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+  00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+
+
+Compilation
+------------
+
+For builds done with ``make``, the driver compilation is enabled by the 
+``CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV`` build configuration option. This 
+is enabled by default in builds for x86 platforms, and disabled in 
+other configurations.
+
+For builds using ``meson`` and ``ninja``, the driver will be built when 
+the target platform is x86-based.
diff --git a/doc/guides/rel_notes/release_19_08.rst b/doc/guides/rel_notes/release_19_08.rst
index 3da266705..720c13f8b 100644
--- a/doc/guides/rel_notes/release_19_08.rst
+++ b/doc/guides/rel_notes/release_19_08.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright 2019 The DPDK contributors
 
+.. include:: <isonum.txt>
+
 DPDK Release 19.08
 ==================
 
@@ -99,6 +101,15 @@ New Features
   Updated ``librte_telemetry`` to fetch the global metrics from the
   ``librte_metrics`` library.
 
+* **Added Intel QuickData Technology PMD**
+
+  The PMD for Intel\ |reg|  QuickData Technology, part of  Intel\ |reg|  
+ I/O Acceleration Technology `(Intel I/OAT)  
+ <https://www.intel.com/content/www/us/en/wireless-network/accel-techno
+ logy.html>`_,  allows data copies to be done by hardware instead  of 
+ via software, reducing cycles spent copying large blocks of data in  
+ applications.
+
 
 Removed Items
 -------------
diff --git a/drivers/raw/Makefile b/drivers/raw/Makefile index 8e29b4a56..c1b85c8c7 100644
--- a/drivers/raw/Makefile
+++ b/drivers/raw/Makefile
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma  endif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev
+DIRS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat
 
 include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile new file mode 100644 index 000000000..7726e310a
--- /dev/null
+++ b/drivers/raw/ioat/Makefile
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2019 Intel 
+Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_pmd_ioat_rawdev.a
+
+# build flags
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+
+LDLIBS += -lrte_eal -lrte_rawdev
+LDLIBS += -lrte_pci -lrte_bus_pci
+
+# library version
+LIBABIVER := 1
+
+# versioning export map
+EXPORT_MAP := rte_pmd_ioat_version.map
+
+# library source files
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
+
+# export include files
+SYMLINK-y-include += rte_ioat_rawdev.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c new file mode 100644 index 000000000..d9fc3091a
--- /dev/null
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -0,0 +1,93 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include <rte_bus_pci.h>
+#include <rte_rawdev_pmd.h>
+
+#include "rte_ioat_rawdev.h"
+
+/* Dynamic log type identifier */
+int ioat_pmd_logtype;
+
+static struct rte_pci_driver ioat_pmd_drv;
+
+#define IOAT_VENDOR_ID		0x8086
+#define IOAT_DEVICE_ID		0x2021

In the second patch, the script dpdk-devbind.py supports CBDMAs in
both Broadwell and Skylake. But you only supports Skylake here.

Thanks,
Jiayu
+
+#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \
+	ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args)
+
+#define IOAT_PMD_DEBUG(fmt, args...)  IOAT_PMD_LOG(DEBUG, fmt, ## args)
+#define IOAT_PMD_INFO(fmt, args...)   IOAT_PMD_LOG(INFO, fmt, ## args)
+#define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
+#define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
+
+static int
+ioat_rawdev_create(const char *name, struct rte_pci_device *dev) {
+	RTE_SET_USED(name);
+	RTE_SET_USED(dev);
+	return 0;
+}
+
+static int
+ioat_rawdev_destroy(const char *name)
+{
+	RTE_SET_USED(name);
+	return 0;
+}
+
+static int
+ioat_rawdev_probe(struct rte_pci_driver *drv, struct rte_pci_device 
+*dev) {
+	char name[32];
+	int ret = 0;
+
+
+	rte_pci_device_name(&dev->addr, name, sizeof(name));
+	IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node);
+
+	dev->device.driver = &drv->driver;
+	ret = ioat_rawdev_create(name, dev);
+	return ret;
+}
+
+static int
+ioat_rawdev_remove(struct rte_pci_device *dev) {
+	char name[32];
+	int ret;
+
+	rte_pci_device_name(&dev->addr, name, sizeof(name));
+
+	IOAT_PMD_INFO("Closing %s on NUMA node %d",
+			name, dev->device.numa_node);
+
+	ret = ioat_rawdev_destroy(name);
+	return ret;
+}
+
+static const struct rte_pci_id pci_id_ioat_map[] = {
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID) },
+	{ .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver ioat_pmd_drv = {
+	.id_table = pci_id_ioat_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+		     RTE_PCI_DRV_IOVA_AS_VA,
+	.probe = ioat_rawdev_probe,
+	.remove = ioat_rawdev_remove,
+};
+
+RTE_PMD_REGISTER_PCI(IOAT_PMD_RAWDEV_NAME, ioat_pmd_drv); 
+RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_RAWDEV_NAME, pci_id_ioat_map); 
+RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_RAWDEV_NAME, "* igb_uio | 
+uio_pci_generic");
+
+RTE_INIT(ioat_pmd_init_log)
+{
+	ioat_pmd_logtype = rte_log_register(IOAT_PMD_LOG_NAME);
+	if (ioat_pmd_logtype >= 0)
+		rte_log_set_level(ioat_pmd_logtype, RTE_LOG_INFO); }
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build new file mode 100644 index 000000000..ba7620a68
--- /dev/null
+++ b/drivers/raw/ioat/meson.build
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause # Copyright 2019 Intel 
+Corporation
+
+build = dpdk_conf.has('RTE_ARCH_X86')
+sources = files('ioat_rawdev.c')
+deps += ['rawdev', 'bus_pci']
+
+install_headers('rte_ioat_rawdev.h')
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
new file mode 100644
index 000000000..e77406403
--- /dev/null
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_IOAT_RAWDEV_H_
+#define _RTE_IOAT_RAWDEV_H_
+
+/**
+ * @file rte_ioat_rawdev.h
+ *
+ * Definitions for using the ioat rawdev device driver
+ *
+ * @warning
+ * @b EXPERIMENTAL: these structures and APIs may change without prior 
+notice  */
+
+/** Name of the device driver */
+#define IOAT_PMD_RAWDEV_NAME rawdev_ioat
+/** String reported as the device driver name by rte_rawdev_info_get() 
+*/ #define IOAT_PMD_RAWDEV_NAME_STR "rawdev_ioat"
+/** Name used to adjust the log level for this driver */ #define 
+IOAT_PMD_LOG_NAME "rawdev.ioat"
+
+#endif
diff --git a/drivers/raw/ioat/rte_pmd_ioat_version.map b/drivers/raw/ioat/rte_pmd_ioat_version.map
new file mode 100644
index 000000000..9a61188cd
--- /dev/null
+++ b/drivers/raw/ioat/rte_pmd_ioat_version.map
@@ -0,0 +1,4 @@
+DPDK_19.08 {
+
+	local: *;
+};
diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build index a61cdccef..2af8a70d4 100644
--- a/drivers/raw/meson.build
+++ b/drivers/raw/meson.build
@@ -1,7 +1,9 @@
 # SPDX-License-Identifier: BSD-3-Clause  # Copyright 2018 NXP
 
-drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
+drivers = ['dpaa2_cmdif', 'dpaa2_qdma',
+	'ifpga_rawdev', 'ioat',
+	'skeleton_rawdev']
 std_deps = ['rawdev']
 config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
 driver_name_fmt = 'rte_pmd_@0@'
diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 81be289a8..2a534796f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -306,6 +306,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV)   += -lrte_pmd_ifpga_rawdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD)       += -lrte_pmd_ipn3ke
 endif # CONFIG_RTE_LIBRTE_IFPGA_BUS
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV)   += -lrte_pmd_ioat_rawdev
 endif # CONFIG_RTE_LIBRTE_RAWDEV
 
 endif # !CONFIG_RTE_BUILD_SHARED_LIBS
--
2.21.0

Re: [dpdk-dev] [PATCH v3 1/8] raw/ioat: add initial support for ioat rawdev driver

[Thomas Monjalon]

Hi Jiayu,

01/07/2019 09:38, Hu, Jiayu:
Hi Bruce,
 
-----Original Message-----
From: dev <dev-bounces@dpdk.org> On Behalf Of Bruce Richardson
Sent: Thursday, June 27, 2019 6:41 PM
To: dev@dpdk.org
Cc: thomas@monjalon.net; jerinj@marvell.com; Richardson, Bruce <bruce.richardson@intel.com>
Subject: [dpdk-dev] [PATCH v3 1/8] raw/ioat: add initial support for ioat rawdev driver
 
Add stubs for ioat rawdev driver support in DPDK, specifically:
 
* makefile and meson build hooks
* initial public header file
* rawdev main C file, with probe and release functions
* release note update announcing the driver
* initial documentation for the new section in the rawdev doc
* unit test stubs for device unit tests
 
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
 
---
V2: don't create a new file for unit testing, add to existing rawdev test
file, and place test cases in the driver selftest routine (added later
in set)
Add new section in document about identifying hardware using lspci
---
MAINTAINERS                                 |  6 +-
app/test/test_rawdev.c                      |  8 ++
config/common_armv8a_linux                  |  1 +
config/common_base                          |  5 ++
config/defconfig_arm-armv7a-linuxapp-gcc    |  1 +
config/defconfig_ppc_64-power8-linuxapp-gcc |  1 +
doc/guides/rawdevs/index.rst                |  1 +
doc/guides/rawdevs/ioat_rawdev.rst          | 63 ++++++++++++++
doc/guides/rel_notes/release_19_08.rst      | 11 +++
drivers/raw/Makefile                        |  1 +
drivers/raw/ioat/Makefile                   | 28 +++++++
drivers/raw/ioat/ioat_rawdev.c              | 93 +++++++++++++++++++++
drivers/raw/ioat/meson.build                |  8 ++
drivers/raw/ioat/rte_ioat_rawdev.h          | 24 ++++++
drivers/raw/ioat/rte_pmd_ioat_version.map   |  4 +
drivers/raw/meson.build                     |  4 +-
mk/rte.app.mk                               |  1 +
17 files changed, 258 insertions(+), 2 deletions(-)  create mode 100644 doc/guides/rawdevs/ioat_rawdev.rst
create mode 100644 drivers/raw/ioat/Makefile  create mode 100644 drivers/raw/ioat/ioat_rawdev.c  create mode 100644 drivers/raw/ioat/meson.build  create mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h
create mode 100644 drivers/raw/ioat/rte_pmd_ioat_version.map
 
diff --git a/MAINTAINERS b/MAINTAINERS
index 0c3b48920..f28c526bc 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1068,6 +1068,11 @@ M: Tianfei zhang <tianfei.zhang@intel.com>
F: drivers/raw/ifpga_rawdev/
F: doc/guides/rawdevs/ifpga_rawdev.rst
  
+IOAT Rawdev
+M: Bruce Richardson <bruce.richardson@intel.com>
+F: drivers/raw/ioat/
+F: doc/guides/rawdevs/ioat_rawdev.rst
+
NXP DPAA2 QDMA
M: Nipun Gupta <nipun.gupta@nxp.com>
F: drivers/raw/dpaa2_qdma/
@@ -1078,7 +1083,6 @@ M: Nipun Gupta <nipun.gupta@nxp.com>
F: drivers/raw/dpaa2_cmdif/
F: doc/guides/rawdevs/dpaa2_cmdif.rst
  
-
Packet processing
-----------------
  
diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c index 043a38a13..88549fb61 100644
--- a/app/test/test_rawdev.c
+++ b/app/test/test_rawdev.c
@@ -25,3 +25,11 @@ test_rawdev_selftest_skeleton(void)
}
  
REGISTER_TEST_COMMAND(rawdev_autotest, test_rawdev_selftest_skeleton);
+
+static int
+test_rawdev_selftest_ioat(void)
+{
+	return TEST_SKIPPED;
+}
+
+REGISTER_TEST_COMMAND(ioat_rawdev_autotest, test_rawdev_selftest_ioat);
diff --git a/config/common_armv8a_linux b/config/common_armv8a_linux index 72091de1c..481712ebc 100644
--- a/config/common_armv8a_linux
+++ b/config/common_armv8a_linux
@@ -34,5 +34,6 @@ CONFIG_RTE_ARCH_ARM64_MEMCPY=n  CONFIG_RTE_LIBRTE_FM10K_PMD=n  CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n  CONFIG_RTE_LIBRTE_AVP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
  
CONFIG_RTE_SCHED_VECTOR=n
diff --git a/config/common_base b/config/common_base index fa1ae249a..e6b830923 100644
--- a/config/common_base
+++ b/config/common_base
@@ -747,6 +747,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n
#
CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
  
+#
+# Compile PMD for Intel IOAT raw device # 
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=y
+
#
# Compile librte_ring
#
diff --git a/config/defconfig_arm-armv7a-linuxapp-gcc b/config/defconfig_arm-armv7a-linuxapp-gcc
index c9509b274..ee158ef9d 100644
--- a/config/defconfig_arm-armv7a-linuxapp-gcc
+++ b/config/defconfig_arm-armv7a-linuxapp-gcc
@@ -54,3 +54,4 @@ CONFIG_RTE_LIBRTE_QEDE_PMD=n  CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n  CONFIG_RTE_LIBRTE_AVP_PMD=n  CONFIG_RTE_LIBRTE_NFP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
diff --git a/config/defconfig_ppc_64-power8-linuxapp-gcc b/config/defconfig_ppc_64-power8-linuxapp-gcc
index 7e248b755..9f3670ec0 100644
--- a/config/defconfig_ppc_64-power8-linuxapp-gcc
+++ b/config/defconfig_ppc_64-power8-linuxapp-gcc
@@ -56,3 +56,4 @@ CONFIG_RTE_LIBRTE_ENIC_PMD=n  CONFIG_RTE_LIBRTE_FM10K_PMD=n  CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n  CONFIG_RTE_LIBRTE_AVP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
diff --git a/doc/guides/rawdevs/index.rst b/doc/guides/rawdevs/index.rst index 7c3bd9586..0a21989e4 100644
--- a/doc/guides/rawdevs/index.rst
+++ b/doc/guides/rawdevs/index.rst
@@ -14,3 +14,4 @@ application through rawdev API.
dpaa2_cmdif
dpaa2_qdma
ifpga_rawdev
+    ioat_rawdev
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
new file mode 100644
index 000000000..0c612e73a
--- /dev/null
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -0,0 +1,63 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+.. include:: <isonum.txt>
+
+IOAT Rawdev Driver for Intel\ |reg| QuickData Technology 
+======================================================================
+
+The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ 
+|reg| QuickData Technology, part of Intel\ |reg| I/O Acceleration 
+Technology `(Intel I/OAT) 
+<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_.
+This PMD, when used on supported hardware, allows data copies, for 
+example, cloning packet data, to be accelerated by that hardware rather 
+than having to be done by software, freeing up CPU cycles for other tasks.
+
+Hardware Requirements
+----------------------
+
+On Linux, the presence of an Intel\ |reg| QuickData Technology hardware 
+can be detected by checking the output of the ``lspci`` command, where 
+the hardware will be often listed as "Crystal Beach DMA" or "CBDMA". 
+For example, on a system with Intel\ |reg| Xeon\ |reg| CPU E5-2699 v4 
+@2.20GHz, lspci shows:
+
+.. code-block:: console
+
+  # lspci | grep DMA
+  00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01)
+  00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01)
+  00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01)
+  00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01)
+  00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01)
+  00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01)
+  00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01)
+  00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 
+ v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01)
+
+On a system with Intel\ |reg| Xeon\ |reg| Gold 6154 CPU @ 3.00GHz, 
+lspci
+shows:
+
+.. code-block:: console
+
+  # lspci | grep DMA
+  00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+  00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+  00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+  00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+  00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+  00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+  00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+  00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA 
+ Registers (rev 04)
+
+
+Compilation
+------------
+
+For builds done with ``make``, the driver compilation is enabled by the 
+``CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV`` build configuration option. This 
+is enabled by default in builds for x86 platforms, and disabled in 
+other configurations.
+
+For builds using ``meson`` and ``ninja``, the driver will be built when 
+the target platform is x86-based.
diff --git a/doc/guides/rel_notes/release_19_08.rst b/doc/guides/rel_notes/release_19_08.rst
index 3da266705..720c13f8b 100644
--- a/doc/guides/rel_notes/release_19_08.rst
+++ b/doc/guides/rel_notes/release_19_08.rst
@@ -1,6 +1,8 @@
..  SPDX-License-Identifier: BSD-3-Clause
Copyright 2019 The DPDK contributors
  
+.. include:: <isonum.txt>
+
DPDK Release 19.08
==================
  
@@ -99,6 +101,15 @@ New Features
Updated ``librte_telemetry`` to fetch the global metrics from the
``librte_metrics`` library.
  
+* **Added Intel QuickData Technology PMD**
+
+  The PMD for Intel\ |reg|  QuickData Technology, part of  Intel\ |reg|  
+ I/O Acceleration Technology `(Intel I/OAT)  
+ <https://www.intel.com/content/www/us/en/wireless-network/accel-techno
+ logy.html>`_,  allows data copies to be done by hardware instead  of 
+ via software, reducing cycles spent copying large blocks of data in  
+ applications.
+
  
Removed Items
-------------
diff --git a/drivers/raw/Makefile b/drivers/raw/Makefile index 8e29b4a56..c1b85c8c7 100644
--- a/drivers/raw/Makefile
+++ b/drivers/raw/Makefile
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif
DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma  endif
DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev
+DIRS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat
  
include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile new file mode 100644 index 000000000..7726e310a
--- /dev/null
+++ b/drivers/raw/ioat/Makefile
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2019 Intel 
+Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_pmd_ioat_rawdev.a
+
+# build flags
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+
+LDLIBS += -lrte_eal -lrte_rawdev
+LDLIBS += -lrte_pci -lrte_bus_pci
+
+# library version
+LIBABIVER := 1
+
+# versioning export map
+EXPORT_MAP := rte_pmd_ioat_version.map
+
+# library source files
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
+
+# export include files
+SYMLINK-y-include += rte_ioat_rawdev.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c new file mode 100644 index 000000000..d9fc3091a
--- /dev/null
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -0,0 +1,93 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include <rte_bus_pci.h>
+#include <rte_rawdev_pmd.h>
+
+#include "rte_ioat_rawdev.h"
+
+/* Dynamic log type identifier */
+int ioat_pmd_logtype;
+
+static struct rte_pci_driver ioat_pmd_drv;
+
+#define IOAT_VENDOR_ID		0x8086
+#define IOAT_DEVICE_ID		0x2021
 
In the second patch, the script dpdk-devbind.py supports CBDMAs in
both Broadwell and Skylake. But you only supports Skylake here.
 
Thanks,
Jiayu
+
+#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \
+	ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args)
+
+#define IOAT_PMD_DEBUG(fmt, args...)  IOAT_PMD_LOG(DEBUG, fmt, ## args)
+#define IOAT_PMD_INFO(fmt, args...)   IOAT_PMD_LOG(INFO, fmt, ## args)
+#define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
+#define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
+
+static int
+ioat_rawdev_create(const char *name, struct rte_pci_device *dev) {
+	RTE_SET_USED(name);
+	RTE_SET_USED(dev);
+	return 0;
+}
+
+static int
+ioat_rawdev_destroy(const char *name)
+{
+	RTE_SET_USED(name);
+	return 0;
+}
+
+static int
+ioat_rawdev_probe(struct rte_pci_driver *drv, struct rte_pci_device 
+*dev) {
+	char name[32];
+	int ret = 0;
+
+
+	rte_pci_device_name(&dev->addr, name, sizeof(name));
+	IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node);
+
+	dev->device.driver = &drv->driver;
+	ret = ioat_rawdev_create(name, dev);
+	return ret;
+}
+
+static int
+ioat_rawdev_remove(struct rte_pci_device *dev) {
+	char name[32];
+	int ret;
+
+	rte_pci_device_name(&dev->addr, name, sizeof(name));
+
+	IOAT_PMD_INFO("Closing %s on NUMA node %d",
+			name, dev->device.numa_node);
+
+	ret = ioat_rawdev_destroy(name);
+	return ret;
+}
+
+static const struct rte_pci_id pci_id_ioat_map[] = {
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID) },
+	{ .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver ioat_pmd_drv = {
+	.id_table = pci_id_ioat_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+		     RTE_PCI_DRV_IOVA_AS_VA,
+	.probe = ioat_rawdev_probe,
+	.remove = ioat_rawdev_remove,
+};
+
+RTE_PMD_REGISTER_PCI(IOAT_PMD_RAWDEV_NAME, ioat_pmd_drv); 
+RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_RAWDEV_NAME, pci_id_ioat_map); 
+RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_RAWDEV_NAME, "* igb_uio | 
+uio_pci_generic");
+
+RTE_INIT(ioat_pmd_init_log)
+{
+	ioat_pmd_logtype = rte_log_register(IOAT_PMD_LOG_NAME);
+	if (ioat_pmd_logtype >= 0)
+		rte_log_set_level(ioat_pmd_logtype, RTE_LOG_INFO); }
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build new file mode 100644 index 000000000..ba7620a68
--- /dev/null
+++ b/drivers/raw/ioat/meson.build
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause # Copyright 2019 Intel 
+Corporation
+
+build = dpdk_conf.has('RTE_ARCH_X86')
+sources = files('ioat_rawdev.c')
+deps += ['rawdev', 'bus_pci']
+
+install_headers('rte_ioat_rawdev.h')
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
new file mode 100644
index 000000000..e77406403
--- /dev/null
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_IOAT_RAWDEV_H_
+#define _RTE_IOAT_RAWDEV_H_
+
+/**
+ * @file rte_ioat_rawdev.h
+ *
+ * Definitions for using the ioat rawdev device driver
+ *
+ * @warning
+ * @b EXPERIMENTAL: these structures and APIs may change without prior 
+notice  */
+
+/** Name of the device driver */
+#define IOAT_PMD_RAWDEV_NAME rawdev_ioat
+/** String reported as the device driver name by rte_rawdev_info_get() 
+*/ #define IOAT_PMD_RAWDEV_NAME_STR "rawdev_ioat"
+/** Name used to adjust the log level for this driver */ #define 
+IOAT_PMD_LOG_NAME "rawdev.ioat"
+
+#endif
diff --git a/drivers/raw/ioat/rte_pmd_ioat_version.map b/drivers/raw/ioat/rte_pmd_ioat_version.map
new file mode 100644
index 000000000..9a61188cd
--- /dev/null
+++ b/drivers/raw/ioat/rte_pmd_ioat_version.map
@@ -0,0 +1,4 @@
+DPDK_19.08 {
+
+	local: *;
+};
diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build index a61cdccef..2af8a70d4 100644
--- a/drivers/raw/meson.build
+++ b/drivers/raw/meson.build
@@ -1,7 +1,9 @@
# SPDX-License-Identifier: BSD-3-Clause  # Copyright 2018 NXP
  
-drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
+drivers = ['dpaa2_cmdif', 'dpaa2_qdma',
+	'ifpga_rawdev', 'ioat',
+	'skeleton_rawdev']
std_deps = ['rawdev']
config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
driver_name_fmt = 'rte_pmd_@0@'
diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 81be289a8..2a534796f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -306,6 +306,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y)
_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV)   += -lrte_pmd_ifpga_rawdev
_LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD)       += -lrte_pmd_ipn3ke
endif # CONFIG_RTE_LIBRTE_IFPGA_BUS
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV)   += -lrte_pmd_ioat_rawdev
endif # CONFIG_RTE_LIBRTE_RAWDEV
  
endif # !CONFIG_RTE_BUILD_SHARED_LIBS
--
2.21.0

Re: [dpdk-dev] [PATCH v3 1/8] raw/ioat: add initial support for ioat rawdev driver

[Hu, Jiayu]

Sorry for the incorrect format in the last mail.
The comment is repeated below.

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Bruce Richardson
> Sent: Thursday, June 27, 2019 6:41 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; jerinj@marvell.com; Richardson, Bruce
> <bruce.richardson@intel.com>
> Subject: [dpdk-dev] [PATCH v3 1/8] raw/ioat: add initial support for ioat
> rawdev driver
> 
> Add stubs for ioat rawdev driver support in DPDK, specifically:
> 
>   * makefile and meson build hooks
>   * initial public header file
>   * rawdev main C file, with probe and release functions
>   * release note update announcing the driver
>   * initial documentation for the new section in the rawdev doc
>   * unit test stubs for device unit tests
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> 
> ---
> V2: don't create a new file for unit testing, add to existing rawdev test
>     file, and place test cases in the driver selftest routine (added later
>     in set)
>     Add new section in document about identifying hardware using lspci
> ---
>  MAINTAINERS                                 |  6 +-
>  app/test/test_rawdev.c                      |  8 ++
>  config/common_armv8a_linux                  |  1 +
>  config/common_base                          |  5 ++
>  config/defconfig_arm-armv7a-linuxapp-gcc    |  1 +
>  config/defconfig_ppc_64-power8-linuxapp-gcc |  1 +
>  doc/guides/rawdevs/index.rst                |  1 +
>  doc/guides/rawdevs/ioat_rawdev.rst          | 63 ++++++++++++++
>  doc/guides/rel_notes/release_19_08.rst      | 11 +++
>  drivers/raw/Makefile                        |  1 +
>  drivers/raw/ioat/Makefile                   | 28 +++++++
>  drivers/raw/ioat/ioat_rawdev.c              | 93 +++++++++++++++++++++
>  drivers/raw/ioat/meson.build                |  8 ++
>  drivers/raw/ioat/rte_ioat_rawdev.h          | 24 ++++++
>  drivers/raw/ioat/rte_pmd_ioat_version.map   |  4 +
>  drivers/raw/meson.build                     |  4 +-
>  mk/rte.app.mk                               |  1 +
>  17 files changed, 258 insertions(+), 2 deletions(-)
>  create mode 100644 doc/guides/rawdevs/ioat_rawdev.rst
>  create mode 100644 drivers/raw/ioat/Makefile
>  create mode 100644 drivers/raw/ioat/ioat_rawdev.c
>  create mode 100644 drivers/raw/ioat/meson.build
>  create mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h
>  create mode 100644 drivers/raw/ioat/rte_pmd_ioat_version.map
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 0c3b48920..f28c526bc 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1068,6 +1068,11 @@ M: Tianfei zhang <tianfei.zhang@intel.com>
>  F: drivers/raw/ifpga_rawdev/
>  F: doc/guides/rawdevs/ifpga_rawdev.rst
> 
> +IOAT Rawdev
> +M: Bruce Richardson <bruce.richardson@intel.com>
> +F: drivers/raw/ioat/
> +F: doc/guides/rawdevs/ioat_rawdev.rst
> +
>  NXP DPAA2 QDMA
>  M: Nipun Gupta <nipun.gupta@nxp.com>
>  F: drivers/raw/dpaa2_qdma/
> @@ -1078,7 +1083,6 @@ M: Nipun Gupta <nipun.gupta@nxp.com>
>  F: drivers/raw/dpaa2_cmdif/
>  F: doc/guides/rawdevs/dpaa2_cmdif.rst
> 
> -
>  Packet processing
>  -----------------
> 
> diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
> index 043a38a13..88549fb61 100644
> --- a/app/test/test_rawdev.c
> +++ b/app/test/test_rawdev.c
> @@ -25,3 +25,11 @@ test_rawdev_selftest_skeleton(void)
>  }
> 
>  REGISTER_TEST_COMMAND(rawdev_autotest,
> test_rawdev_selftest_skeleton);
> +
> +static int
> +test_rawdev_selftest_ioat(void)
> +{
> +	return TEST_SKIPPED;
> +}
> +
> +REGISTER_TEST_COMMAND(ioat_rawdev_autotest,
> test_rawdev_selftest_ioat);
> diff --git a/config/common_armv8a_linux b/config/common_armv8a_linux
> index 72091de1c..481712ebc 100644
> --- a/config/common_armv8a_linux
> +++ b/config/common_armv8a_linux
> @@ -34,5 +34,6 @@ CONFIG_RTE_ARCH_ARM64_MEMCPY=n
>  CONFIG_RTE_LIBRTE_FM10K_PMD=n
>  CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
>  CONFIG_RTE_LIBRTE_AVP_PMD=n
> +CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
> 
>  CONFIG_RTE_SCHED_VECTOR=n
> diff --git a/config/common_base b/config/common_base
> index fa1ae249a..e6b830923 100644
> --- a/config/common_base
> +++ b/config/common_base
> @@ -747,6 +747,11 @@
> CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n
>  #
>  CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
> 
> +#
> +# Compile PMD for Intel IOAT raw device
> +#
> +CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=y
> +
>  #
>  # Compile librte_ring
>  #
> diff --git a/config/defconfig_arm-armv7a-linuxapp-gcc
> b/config/defconfig_arm-armv7a-linuxapp-gcc
> index c9509b274..ee158ef9d 100644
> --- a/config/defconfig_arm-armv7a-linuxapp-gcc
> +++ b/config/defconfig_arm-armv7a-linuxapp-gcc
> @@ -54,3 +54,4 @@ CONFIG_RTE_LIBRTE_QEDE_PMD=n
>  CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
>  CONFIG_RTE_LIBRTE_AVP_PMD=n
>  CONFIG_RTE_LIBRTE_NFP_PMD=n
> +CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
> diff --git a/config/defconfig_ppc_64-power8-linuxapp-gcc
> b/config/defconfig_ppc_64-power8-linuxapp-gcc
> index 7e248b755..9f3670ec0 100644
> --- a/config/defconfig_ppc_64-power8-linuxapp-gcc
> +++ b/config/defconfig_ppc_64-power8-linuxapp-gcc
> @@ -56,3 +56,4 @@ CONFIG_RTE_LIBRTE_ENIC_PMD=n
>  CONFIG_RTE_LIBRTE_FM10K_PMD=n
>  CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
>  CONFIG_RTE_LIBRTE_AVP_PMD=n
> +CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
> diff --git a/doc/guides/rawdevs/index.rst b/doc/guides/rawdevs/index.rst
> index 7c3bd9586..0a21989e4 100644
> --- a/doc/guides/rawdevs/index.rst
> +++ b/doc/guides/rawdevs/index.rst
> @@ -14,3 +14,4 @@ application through rawdev API.
>      dpaa2_cmdif
>      dpaa2_qdma
>      ifpga_rawdev
> +    ioat_rawdev
> diff --git a/doc/guides/rawdevs/ioat_rawdev.rst
> b/doc/guides/rawdevs/ioat_rawdev.rst
> new file mode 100644
> index 000000000..0c612e73a
> --- /dev/null
> +++ b/doc/guides/rawdevs/ioat_rawdev.rst
> @@ -0,0 +1,63 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2019 Intel Corporation.
> +
> +.. include:: <isonum.txt>
> +
> +IOAT Rawdev Driver for Intel\ |reg| QuickData Technology
> +===============================================================
> =======
> +
> +The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg|
> +QuickData Technology, part of Intel\ |reg| I/O Acceleration Technology
> +`(Intel I/OAT)
> +<https://www.intel.com/content/www/us/en/wireless-network/accel-
> technology.html>`_.
> +This PMD, when used on supported hardware, allows data copies, for
> example,
> +cloning packet data, to be accelerated by that hardware rather than having
> to
> +be done by software, freeing up CPU cycles for other tasks.
> +
> +Hardware Requirements
> +----------------------
> +
> +On Linux, the presence of an Intel\ |reg| QuickData Technology hardware
> can
> +be detected by checking the output of the ``lspci`` command, where the
> +hardware will be often listed as "Crystal Beach DMA" or "CBDMA". For
> +example, on a system with Intel\ |reg| Xeon\ |reg| CPU E5-2699 v4
> @2.20GHz,
> +lspci shows:
> +
> +.. code-block:: console
> +
> +  # lspci | grep DMA
> +  00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon
> E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01)
> +  00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon
> E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01)
> +  00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon
> E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01)
> +  00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon
> E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01)
> +  00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon
> E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01)
> +  00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon
> E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01)
> +  00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon
> E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01)
> +  00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon
> E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01)
> +
> +On a system with Intel\ |reg| Xeon\ |reg| Gold 6154 CPU @ 3.00GHz, lspci
> +shows:
> +
> +.. code-block:: console
> +
> +  # lspci | grep DMA
> +  00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers
> (rev 04)
> +  00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers
> (rev 04)
> +  00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers
> (rev 04)
> +  00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers
> (rev 04)
> +  00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers
> (rev 04)
> +  00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers
> (rev 04)
> +  00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers
> (rev 04)
> +  00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers
> (rev 04)
> +
> +
> +Compilation
> +------------
> +
> +For builds done with ``make``, the driver compilation is enabled by the
> +``CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV`` build configuration option.
> This is
> +enabled by default in builds for x86 platforms, and disabled in other
> +configurations.
> +
> +For builds using ``meson`` and ``ninja``, the driver will be built when the
> +target platform is x86-based.
> diff --git a/doc/guides/rel_notes/release_19_08.rst
> b/doc/guides/rel_notes/release_19_08.rst
> index 3da266705..720c13f8b 100644
> --- a/doc/guides/rel_notes/release_19_08.rst
> +++ b/doc/guides/rel_notes/release_19_08.rst
> @@ -1,6 +1,8 @@
>  ..  SPDX-License-Identifier: BSD-3-Clause
>      Copyright 2019 The DPDK contributors
> 
> +.. include:: <isonum.txt>
> +
>  DPDK Release 19.08
>  ==================
> 
> @@ -99,6 +101,15 @@ New Features
>    Updated ``librte_telemetry`` to fetch the global metrics from the
>    ``librte_metrics`` library.
> 
> +* **Added Intel QuickData Technology PMD**
> +
> +  The PMD for Intel\ |reg|  QuickData Technology, part of
> +  Intel\ |reg|  I/O Acceleration Technology `(Intel I/OAT)
> +  <https://www.intel.com/content/www/us/en/wireless-network/accel-
> technology.html>`_,
> +  allows data copies to be done by hardware instead
> +  of via software, reducing cycles spent copying large blocks of data in
> +  applications.
> +
> 
>  Removed Items
>  -------------
> diff --git a/drivers/raw/Makefile b/drivers/raw/Makefile
> index 8e29b4a56..c1b85c8c7 100644
> --- a/drivers/raw/Makefile
> +++ b/drivers/raw/Makefile
> @@ -10,5 +10,6 @@ DIRS-
> $(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif
>  DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) +=
> dpaa2_qdma
>  endif
>  DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev
> +DIRS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat
> 
>  include $(RTE_SDK)/mk/rte.subdir.mk
> diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
> new file mode 100644
> index 000000000..7726e310a
> --- /dev/null
> +++ b/drivers/raw/ioat/Makefile
> @@ -0,0 +1,28 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2019 Intel Corporation
> +
> +include $(RTE_SDK)/mk/rte.vars.mk
> +
> +# library name
> +LIB = librte_pmd_ioat_rawdev.a
> +
> +# build flags
> +CFLAGS += -O3
> +CFLAGS += $(WERROR_FLAGS)
> +
> +LDLIBS += -lrte_eal -lrte_rawdev
> +LDLIBS += -lrte_pci -lrte_bus_pci
> +
> +# library version
> +LIBABIVER := 1
> +
> +# versioning export map
> +EXPORT_MAP := rte_pmd_ioat_version.map
> +
> +# library source files
> +SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
> +
> +# export include files
> +SYMLINK-y-include += rte_ioat_rawdev.h
> +
> +include $(RTE_SDK)/mk/rte.lib.mk
> diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
> new file mode 100644
> index 000000000..d9fc3091a
> --- /dev/null
> +++ b/drivers/raw/ioat/ioat_rawdev.c
> @@ -0,0 +1,93 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2019 Intel Corporation
> + */
> +
> +#include <rte_bus_pci.h>
> +#include <rte_rawdev_pmd.h>
> +
> +#include "rte_ioat_rawdev.h"
> +
> +/* Dynamic log type identifier */
> +int ioat_pmd_logtype;
> +
> +static struct rte_pci_driver ioat_pmd_drv;
> +
> +#define IOAT_VENDOR_ID		0x8086
> +#define IOAT_DEVICE_ID		0x2021

In the second patch, the script dpdk-devbind.py supports CBDMAs in
both Broadwell and Skylake. But you only supports Skylake here.

Thanks,
Jiayu
> +
> +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \
> +	ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args)
> +
> +#define IOAT_PMD_DEBUG(fmt, args...)  IOAT_PMD_LOG(DEBUG, fmt, ##
> args)
> +#define IOAT_PMD_INFO(fmt, args...)   IOAT_PMD_LOG(INFO, fmt, ## args)
> +#define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
> +#define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt,
> ## args)
> +
> +static int
> +ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
> +{
> +	RTE_SET_USED(name);
> +	RTE_SET_USED(dev);
> +	return 0;
> +}
> +
> +static int
> +ioat_rawdev_destroy(const char *name)
> +{
> +	RTE_SET_USED(name);
> +	return 0;
> +}
> +
> +static int
> +ioat_rawdev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev)
> +{
> +	char name[32];
> +	int ret = 0;
> +
> +
> +	rte_pci_device_name(&dev->addr, name, sizeof(name));
> +	IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev-
> >device.numa_node);
> +
> +	dev->device.driver = &drv->driver;
> +	ret = ioat_rawdev_create(name, dev);
> +	return ret;
> +}
> +
> +static int
> +ioat_rawdev_remove(struct rte_pci_device *dev)
> +{
> +	char name[32];
> +	int ret;
> +
> +	rte_pci_device_name(&dev->addr, name, sizeof(name));
> +
> +	IOAT_PMD_INFO("Closing %s on NUMA node %d",
> +			name, dev->device.numa_node);
> +
> +	ret = ioat_rawdev_destroy(name);
> +	return ret;
> +}
> +
> +static const struct rte_pci_id pci_id_ioat_map[] = {
> +	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID) },
> +	{ .vendor_id = 0, /* sentinel */ },
> +};
> +
> +static struct rte_pci_driver ioat_pmd_drv = {
> +	.id_table = pci_id_ioat_map,
> +	.drv_flags = RTE_PCI_DRV_NEED_MAPPING |
> RTE_PCI_DRV_INTR_LSC |
> +		     RTE_PCI_DRV_IOVA_AS_VA,
> +	.probe = ioat_rawdev_probe,
> +	.remove = ioat_rawdev_remove,
> +};
> +
> +RTE_PMD_REGISTER_PCI(IOAT_PMD_RAWDEV_NAME, ioat_pmd_drv);
> +RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_RAWDEV_NAME,
> pci_id_ioat_map);
> +RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_RAWDEV_NAME, "* igb_uio |
> uio_pci_generic");
> +
> +RTE_INIT(ioat_pmd_init_log)
> +{
> +	ioat_pmd_logtype = rte_log_register(IOAT_PMD_LOG_NAME);
> +	if (ioat_pmd_logtype >= 0)
> +		rte_log_set_level(ioat_pmd_logtype, RTE_LOG_INFO);
> +}
> diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
> new file mode 100644
> index 000000000..ba7620a68
> --- /dev/null
> +++ b/drivers/raw/ioat/meson.build
> @@ -0,0 +1,8 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright 2019 Intel Corporation
> +
> +build = dpdk_conf.has('RTE_ARCH_X86')
> +sources = files('ioat_rawdev.c')
> +deps += ['rawdev', 'bus_pci']
> +
> +install_headers('rte_ioat_rawdev.h')
> diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h
> b/drivers/raw/ioat/rte_ioat_rawdev.h
> new file mode 100644
> index 000000000..e77406403
> --- /dev/null
> +++ b/drivers/raw/ioat/rte_ioat_rawdev.h
> @@ -0,0 +1,24 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2019 Intel Corporation
> + */
> +
> +#ifndef _RTE_IOAT_RAWDEV_H_
> +#define _RTE_IOAT_RAWDEV_H_
> +
> +/**
> + * @file rte_ioat_rawdev.h
> + *
> + * Definitions for using the ioat rawdev device driver
> + *
> + * @warning
> + * @b EXPERIMENTAL: these structures and APIs may change without prior
> notice
> + */
> +
> +/** Name of the device driver */
> +#define IOAT_PMD_RAWDEV_NAME rawdev_ioat
> +/** String reported as the device driver name by rte_rawdev_info_get() */
> +#define IOAT_PMD_RAWDEV_NAME_STR "rawdev_ioat"
> +/** Name used to adjust the log level for this driver */
> +#define IOAT_PMD_LOG_NAME "rawdev.ioat"
> +
> +#endif
> diff --git a/drivers/raw/ioat/rte_pmd_ioat_version.map
> b/drivers/raw/ioat/rte_pmd_ioat_version.map
> new file mode 100644
> index 000000000..9a61188cd
> --- /dev/null
> +++ b/drivers/raw/ioat/rte_pmd_ioat_version.map
> @@ -0,0 +1,4 @@
> +DPDK_19.08 {
> +
> +	local: *;
> +};
> diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build
> index a61cdccef..2af8a70d4 100644
> --- a/drivers/raw/meson.build
> +++ b/drivers/raw/meson.build
> @@ -1,7 +1,9 @@
>  # SPDX-License-Identifier: BSD-3-Clause
>  # Copyright 2018 NXP
> 
> -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
> +drivers = ['dpaa2_cmdif', 'dpaa2_qdma',
> +	'ifpga_rawdev', 'ioat',
> +	'skeleton_rawdev']
>  std_deps = ['rawdev']
>  config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
>  driver_name_fmt = 'rte_pmd_@0@'
> diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> index 81be289a8..2a534796f 100644
> --- a/mk/rte.app.mk
> +++ b/mk/rte.app.mk
> @@ -306,6 +306,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y)
>  _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV)   += -
> lrte_pmd_ifpga_rawdev
>  _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD)       += -lrte_pmd_ipn3ke
>  endif # CONFIG_RTE_LIBRTE_IFPGA_BUS
> +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV)   += -
> lrte_pmd_ioat_rawdev
>  endif # CONFIG_RTE_LIBRTE_RAWDEV
> 
>  endif # !CONFIG_RTE_BUILD_SHARED_LIBS
> --
> 2.21.0

Re: [dpdk-dev] [PATCH v3 1/8] raw/ioat: add initial support for ioat rawdev driver

[Bruce Richardson]

On Mon, Jul 01, 2019 at 09:29:21AM +0100, Hu, Jiayu wrote:
> Sorry for the incorrect format in the last mail.
> The comment is repeated below.
> 
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Bruce Richardson
> > Sent: Thursday, June 27, 2019 6:41 PM
> > To: dev@dpdk.org
> > Cc: thomas@monjalon.net; jerinj@marvell.com; Richardson, Bruce
> > <bruce.richardson@intel.com>
> > Subject: [dpdk-dev] [PATCH v3 1/8] raw/ioat: add initial support for ioat
> > rawdev driver
> > 
> > Add stubs for ioat rawdev driver support in DPDK, specifically:
> > 
> >   * makefile and meson build hooks
> >   * initial public header file
> >   * rawdev main C file, with probe and release functions
> >   * release note update announcing the driver
> >   * initial documentation for the new section in the rawdev doc
> >   * unit test stubs for device unit tests
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > 
> > ---
<snip>
> +
> > +static struct rte_pci_driver ioat_pmd_drv;
> > +
> > +#define IOAT_VENDOR_ID		0x8086
> > +#define IOAT_DEVICE_ID		0x2021
> 
> In the second patch, the script dpdk-devbind.py supports CBDMAs in
> both Broadwell and Skylake. But you only supports Skylake here.
>

Will fix in v4, thanks.

/Bruce 

[dpdk-dev] [PATCH v3 2/8] usertools/dpdk-devbind.py: add support for IOAT devices

[Bruce Richardson]

In order to allow binding/unbinding of devices for use by the
ioat_rawdev, we need to update the devbind script to add a new class
of device, and add device ids for the specific HW instances.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---

V2: rather than adding a DMA section, add to "misc (rawdev)" section where
    other device types, e.g. ntb can also do.

NOTE: this set largely overlaps with the equivalent changes made by the
patchset adding NTB support [1]. Since it's unclear which set will be added
first, this set is based off the latest head.

[1] http://patches.dpdk.org/patch/55127/
---
 doc/guides/rawdevs/ioat_rawdev.rst | 11 +++++++++++
 usertools/dpdk-devbind.py          | 10 ++++++++++
 2 files changed, 21 insertions(+)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 0c612e73a..1a4b0e03e 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -61,3 +61,14 @@ configurations.
 
 For builds using ``meson`` and ``ninja``, the driver will be built when the
 target platform is x86-based.
+
+Device Setup
+-------------
+
+The Intel\ |reg| QuickData Technology HW devices will need to be bound to a
+user-space IO driver for use. The script ``dpdk-devbind.py`` script
+included with DPDK can be used to view the state of the devices and to bind
+them to a suitable DPDK-supported kernel driver. When querying the status
+of the devices, they will appear under the category of "Misc (rawdev)
+devices", i.e. the command ``dpdk-devbind.py --status-dev misc`` can be
+used to see the state of those devices alone.
diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py
index 9e79f0d28..5c1cd3548 100755
--- a/usertools/dpdk-devbind.py
+++ b/usertools/dpdk-devbind.py
@@ -36,11 +36,17 @@
 octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc',
               'SVendor': None, 'SDevice': None}
 
+intel_ioat_bdw = {'Class': '08', 'Vendor': '8086', 'Device': '6f20,6f21,6f22,6f23,6f24,6f25,6f26,6f27,6f2e,6f2f',
+              'SVendor': None, 'SDevice': None}
+intel_ioat_skx = {'Class': '08', 'Vendor': '8086', 'Device': '2021',
+              'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 crypto_devices = [encryption_class, intel_processor_class]
 eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso]
 mempool_devices = [cavium_fpa, octeontx2_npa]
 compress_devices = [cavium_zip]
+misc_devices = [intel_ioat_bdw, intel_ioat_skx]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
@@ -595,6 +601,8 @@ def show_status():
     if status_dev == "compress" or status_dev == "all":
         show_device_status(compress_devices , "Compress")
 
+    if status_dev == "misc" or status_dev == "all":
+        show_device_status(misc_devices, "Misc (rawdev)")
 
 def parse_args():
     '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +678,7 @@ def do_arg_actions():
             get_device_details(eventdev_devices)
             get_device_details(mempool_devices)
             get_device_details(compress_devices)
+            get_device_details(misc_devices)
         show_status()
 
 
@@ -690,6 +699,7 @@ def main():
     get_device_details(eventdev_devices)
     get_device_details(mempool_devices)
     get_device_details(compress_devices)
+    get_device_details(misc_devices)
     do_arg_actions()
 
 if __name__ == "__main__":
-- 
2.21.0

Re: [dpdk-dev] [PATCH v3 2/8] usertools/dpdk-devbind.py: add support for IOAT devices

[Burakov, Anatoly]

On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> In order to allow binding/unbinding of devices for use by the
> ioat_rawdev, we need to update the devbind script to add a new class
> of device, and add device ids for the specific HW instances.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> 
> ---

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

[dpdk-dev] [PATCH v3 3/8] raw/ioat: add register definition file

[Bruce Richardson]

Add in the list of registers for the device. File is taken from the SPDK
project:

  https://github.com/spdk/spdk/blob/master/include/spdk/ioat_spec.h

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 drivers/raw/ioat/Makefile        |   1 +
 drivers/raw/ioat/meson.build     |   3 +-
 drivers/raw/ioat/rte_ioat_spec.h | 301 +++++++++++++++++++++++++++++++
 3 files changed, 304 insertions(+), 1 deletion(-)
 create mode 100644 drivers/raw/ioat/rte_ioat_spec.h

diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
index 7726e310a..1e10938f3 100644
--- a/drivers/raw/ioat/Makefile
+++ b/drivers/raw/ioat/Makefile
@@ -24,5 +24,6 @@ SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
 
 # export include files
 SYMLINK-y-include += rte_ioat_rawdev.h
+SYMLINK-y-include += rte_ioat_spec.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
index ba7620a68..ca23e23fc 100644
--- a/drivers/raw/ioat/meson.build
+++ b/drivers/raw/ioat/meson.build
@@ -5,4 +5,5 @@ build = dpdk_conf.has('RTE_ARCH_X86')
 sources = files('ioat_rawdev.c')
 deps += ['rawdev', 'bus_pci']
 
-install_headers('rte_ioat_rawdev.h')
+install_headers('rte_ioat_rawdev.h',
+		'rte_ioat_spec.h')
diff --git a/drivers/raw/ioat/rte_ioat_spec.h b/drivers/raw/ioat/rte_ioat_spec.h
new file mode 100644
index 000000000..305e36ded
--- /dev/null
+++ b/drivers/raw/ioat/rte_ioat_spec.h
@@ -0,0 +1,301 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) Intel Corporation
+ */
+
+/**
+ * \file
+ * I/OAT specification definitions
+ *
+ * Taken from ioat_spec.h from SPDK project, with prefix renames and
+ * other minor changes.
+ */
+
+#ifndef RTE_IOAT_SPEC_H
+#define RTE_IOAT_SPEC_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+#define RTE_IOAT_PCI_CHANERR_INT_OFFSET	0x180
+
+#define RTE_IOAT_INTRCTRL_MASTER_INT_EN	0x01
+
+#define RTE_IOAT_VER_3_0                0x30
+#define RTE_IOAT_VER_3_3                0x33
+
+/* DMA Channel Registers */
+#define RTE_IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK	0xF000
+#define RTE_IOAT_CHANCTRL_COMPL_DCA_EN		0x0200
+#define RTE_IOAT_CHANCTRL_CHANNEL_IN_USE		0x0100
+#define RTE_IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL	0x0020
+#define RTE_IOAT_CHANCTRL_ERR_INT_EN		0x0010
+#define RTE_IOAT_CHANCTRL_ANY_ERR_ABORT_EN		0x0008
+#define RTE_IOAT_CHANCTRL_ERR_COMPLETION_EN		0x0004
+#define RTE_IOAT_CHANCTRL_INT_REARM			0x0001
+
+/* DMA Channel Capabilities */
+#define	RTE_IOAT_DMACAP_PB		(1 << 0)
+#define	RTE_IOAT_DMACAP_DCA		(1 << 4)
+#define	RTE_IOAT_DMACAP_BFILL		(1 << 6)
+#define	RTE_IOAT_DMACAP_XOR		(1 << 8)
+#define	RTE_IOAT_DMACAP_PQ		(1 << 9)
+#define	RTE_IOAT_DMACAP_DMA_DIF	(1 << 10)
+
+struct rte_ioat_registers {
+	uint8_t		chancnt;
+	uint8_t		xfercap;
+	uint8_t		genctrl;
+	uint8_t		intrctrl;
+	uint32_t	attnstatus;
+	uint8_t		cbver;		/* 0x08 */
+	uint8_t		reserved4[0x3]; /* 0x09 */
+	uint16_t	intrdelay;	/* 0x0C */
+	uint16_t	cs_status;	/* 0x0E */
+	uint32_t	dmacapability;	/* 0x10 */
+	uint8_t		reserved5[0x6C]; /* 0x14 */
+	uint16_t	chanctrl;	/* 0x80 */
+	uint8_t		reserved6[0x2];	/* 0x82 */
+	uint8_t		chancmd;	/* 0x84 */
+	uint8_t		reserved3[1];	/* 0x85 */
+	uint16_t	dmacount;	/* 0x86 */
+	uint64_t	chansts;	/* 0x88 */
+	uint64_t	chainaddr;	/* 0x90 */
+	uint64_t	chancmp;	/* 0x98 */
+	uint8_t		reserved2[0x8];	/* 0xA0 */
+	uint32_t	chanerr;	/* 0xA8 */
+	uint32_t	chanerrmask;	/* 0xAC */
+} __attribute__((packed));
+
+#define RTE_IOAT_CHANCMD_RESET			0x20
+#define RTE_IOAT_CHANCMD_SUSPEND		0x04
+
+#define RTE_IOAT_CHANSTS_STATUS		0x7ULL
+#define RTE_IOAT_CHANSTS_ACTIVE		0x0
+#define RTE_IOAT_CHANSTS_IDLE			0x1
+#define RTE_IOAT_CHANSTS_SUSPENDED		0x2
+#define RTE_IOAT_CHANSTS_HALTED		0x3
+#define RTE_IOAT_CHANSTS_ARMED			0x4
+
+#define RTE_IOAT_CHANSTS_UNAFFILIATED_ERROR	0x8ULL
+#define RTE_IOAT_CHANSTS_SOFT_ERROR		0x10ULL
+
+#define RTE_IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK	(~0x3FULL)
+
+#define RTE_IOAT_CHANCMP_ALIGN			8	/* CHANCMP address must be 64-bit aligned */
+
+struct rte_ioat_generic_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t reserved2: 1;
+			uint32_t src_page_break: 1;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t op_specific[4];
+};
+
+struct rte_ioat_dma_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t null: 1;
+			uint32_t src_page_break: 1;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+#define RTE_IOAT_OP_COPY 0x00
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t reserved;
+	uint64_t reserved2;
+	uint64_t user1;
+	uint64_t user2;
+};
+
+struct rte_ioat_fill_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t reserved: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t reserved2: 2;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t reserved3: 15;
+#define RTE_IOAT_OP_FILL 0x01
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_data;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t reserved;
+	uint64_t next_dest_addr;
+	uint64_t user1;
+	uint64_t user2;
+};
+
+struct rte_ioat_xor_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_count: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+#define RTE_IOAT_OP_XOR 0x87
+#define RTE_IOAT_OP_XOR_VAL 0x88
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t src_addr3;
+	uint64_t src_addr4;
+	uint64_t src_addr5;
+};
+
+struct rte_ioat_xor_ext_hw_desc {
+	uint64_t src_addr6;
+	uint64_t src_addr7;
+	uint64_t src_addr8;
+	uint64_t next;
+	uint64_t reserved[4];
+};
+
+struct rte_ioat_pq_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_count: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t p_disable: 1;
+			uint32_t q_disable: 1;
+			uint32_t reserved: 11;
+#define RTE_IOAT_OP_PQ 0x89
+#define RTE_IOAT_OP_PQ_VAL 0x8a
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t p_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t src_addr3;
+	uint8_t  coef[8];
+	uint64_t q_addr;
+};
+
+struct rte_ioat_pq_ext_hw_desc {
+	uint64_t src_addr4;
+	uint64_t src_addr5;
+	uint64_t src_addr6;
+	uint64_t next;
+	uint64_t src_addr7;
+	uint64_t src_addr8;
+	uint64_t reserved[2];
+};
+
+struct rte_ioat_pq_update_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_cnt: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t p_disable: 1;
+			uint32_t q_disable: 1;
+			uint32_t reserved: 3;
+			uint32_t coef: 8;
+#define RTE_IOAT_OP_PQ_UP 0x8b
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t p_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t p_src;
+	uint64_t q_src;
+	uint64_t q_addr;
+};
+
+struct rte_ioat_raw_hw_desc {
+	uint64_t field[8];
+};
+
+union rte_ioat_hw_desc {
+	struct rte_ioat_raw_hw_desc raw;
+	struct rte_ioat_generic_hw_desc generic;
+	struct rte_ioat_dma_hw_desc dma;
+	struct rte_ioat_fill_hw_desc fill;
+	struct rte_ioat_xor_hw_desc xor_desc;
+	struct rte_ioat_xor_ext_hw_desc xor_ext;
+	struct rte_ioat_pq_hw_desc pq;
+	struct rte_ioat_pq_ext_hw_desc pq_ext;
+	struct rte_ioat_pq_update_hw_desc pq_update;
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_IOAT_SPEC_H */
-- 
2.21.0

Re: [dpdk-dev] [PATCH v3 3/8] raw/ioat: add register definition file

[Burakov, Anatoly]

On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> Add in the list of registers for the device. File is taken from the SPDK
> project:
> 
>    https://github.com/spdk/spdk/blob/master/include/spdk/ioat_spec.h
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---

The indentation is slightly inconsistent across the file, but LGTM

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3 3/8] raw/ioat: add register definition file

[Bruce Richardson]

On Thu, Jun 27, 2019 at 01:01:47PM +0100, Burakov, Anatoly wrote:
> On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> > Add in the list of registers for the device. File is taken from the SPDK
> > project:
> > 
> >    https://github.com/spdk/spdk/blob/master/include/spdk/ioat_spec.h
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
> 
> The indentation is slightly inconsistent across the file, but LGTM
> 
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> 
I've tried to keep the delta to the original file to a minimum, so didn't
look at whitespace. I think it's ok as-is.

[dpdk-dev] [PATCH v3 4/8] raw/ioat: create device on probe and destroy on release

[Bruce Richardson]

Add the create/destroy driver functions so that we can actually allocate
a rawdev and destroy it when done. No rawdev API functions are actually
implemented at this point.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/rawdevs/ioat_rawdev.rst | 11 ++++
 drivers/raw/ioat/ioat_rawdev.c     | 93 +++++++++++++++++++++++++++++-
 drivers/raw/ioat/rte_ioat_rawdev.h | 20 +++++++
 3 files changed, 121 insertions(+), 3 deletions(-)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 1a4b0e03e..4b7fe8a8f 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -72,3 +72,14 @@ them to a suitable DPDK-supported kernel driver. When querying the status
 of the devices, they will appear under the category of "Misc (rawdev)
 devices", i.e. the command ``dpdk-devbind.py --status-dev misc`` can be
 used to see the state of those devices alone.
+
+Device Probing and Initialization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once bound to a suitable kernel device driver, the HW devices will be found
+as part of the PCI scan done at application initialization time. No vdev
+parameters need to be passed to create or initialize the device.
+
+Once probed successfully, the device will appear as a ``rawdev``, that is a
+"raw device type" inside DPDK, and can be accessed using APIs from the
+``rte_rawdev`` library.
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index d9fc3091a..d13391dd5 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -2,6 +2,7 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <rte_cycles.h>
 #include <rte_bus_pci.h>
 #include <rte_rawdev_pmd.h>
 
@@ -26,15 +27,101 @@ static struct rte_pci_driver ioat_pmd_drv;
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
-	RTE_SET_USED(name);
-	RTE_SET_USED(dev);
+	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+	};
+
+	struct rte_rawdev *rawdev = NULL;
+	struct rte_ioat_rawdev *ioat = NULL;
+	int ret = 0;
+	int retry = 0;
+
+	if (!name) {
+		IOAT_PMD_ERR("Invalid name of the device!");
+		ret = -EINVAL;
+		goto cleanup;
+	}
+
+	/* Allocate device structure */
+	rawdev = rte_rawdev_pmd_allocate(name, sizeof(struct rte_ioat_rawdev),
+					 dev->device.numa_node);
+	if (rawdev == NULL) {
+		IOAT_PMD_ERR("Unable to allocate raw device");
+		ret = -EINVAL;
+		goto cleanup;
+	}
+
+	rawdev->dev_ops = &ioat_rawdev_ops;
+	rawdev->device = &dev->device;
+	rawdev->driver_name = dev->device.driver->name;
+
+	ioat = rawdev->dev_private;
+	ioat->rawdev = rawdev;
+	ioat->regs = dev->mem_resource[0].addr;
+	ioat->ring_size = 0;
+	ioat->desc_ring = NULL;
+	ioat->status_addr = rte_malloc_virt2iova(ioat) +
+			offsetof(struct rte_ioat_rawdev, status);
+
+	/* do device initialization - reset and set error behaviour */
+	if (ioat->regs->chancnt != 1)
+		IOAT_PMD_ERR("%s: Channel count == %d\n", __func__,
+				ioat->regs->chancnt);
+
+	if (ioat->regs->chanctrl & 0x100) { /* locked by someone else */
+		IOAT_PMD_WARN("%s: Channel appears locked\n", __func__);
+		ioat->regs->chanctrl = 0;
+	}
+
+	ioat->regs->chancmd = RTE_IOAT_CHANCMD_SUSPEND;
+	rte_delay_ms(1);
+	ioat->regs->chancmd = RTE_IOAT_CHANCMD_RESET;
+	rte_delay_ms(1);
+	while (ioat->regs->chancmd & RTE_IOAT_CHANCMD_RESET) {
+		ioat->regs->chainaddr = 0;
+		rte_delay_ms(1);
+		if (++retry >= 200) {
+			IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=0x%"PRIx8", CHANSTS=0x%"PRIx64", CHANERR=0x%"PRIx32"\n",
+					__func__,
+					ioat->regs->chancmd,
+					ioat->regs->chansts,
+					ioat->regs->chanerr);
+			ret = -EIO;
+		}
+	}
+	ioat->regs->chanctrl = RTE_IOAT_CHANCTRL_ANY_ERR_ABORT_EN |
+			RTE_IOAT_CHANCTRL_ERR_COMPLETION_EN;
+
 	return 0;
+
+cleanup:
+	if (rawdev)
+		rte_rawdev_pmd_release(rawdev);
+
+	return ret;
 }
 
 static int
 ioat_rawdev_destroy(const char *name)
 {
-	RTE_SET_USED(name);
+	int ret;
+	struct rte_rawdev *rdev;
+
+	if (!name) {
+		IOAT_PMD_ERR("Invalid device name");
+		return -EINVAL;
+	}
+
+	rdev = rte_rawdev_pmd_get_named_dev(name);
+	if (!rdev) {
+		IOAT_PMD_ERR("Invalid device name (%s)", name);
+		return -EINVAL;
+	}
+
+	/* rte_rawdev_close is called by pmd_release */
+	ret = rte_rawdev_pmd_release(rdev);
+	if (ret)
+		IOAT_PMD_DEBUG("Device cleanup failed");
+
 	return 0;
 }
 
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index e77406403..c3216a174 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -14,6 +14,9 @@
  * @b EXPERIMENTAL: these structures and APIs may change without prior notice
  */
 
+#include <rte_memory.h>
+#include <rte_ioat_spec.h>
+
 /** Name of the device driver */
 #define IOAT_PMD_RAWDEV_NAME rawdev_ioat
 /** String reported as the device driver name by rte_rawdev_info_get() */
@@ -21,4 +24,21 @@
 /** Name used to adjust the log level for this driver */
 #define IOAT_PMD_LOG_NAME "rawdev.ioat"
 
+/**
+ * @internal
+ * Structure representing a device instance
+ */
+struct rte_ioat_rawdev {
+	struct rte_rawdev *rawdev;
+	volatile struct rte_ioat_registers *regs;
+	phys_addr_t status_addr;
+	phys_addr_t ring_addr;
+
+	unsigned short ring_size;
+	struct rte_ioat_desc *desc_ring;
+
+	/* to report completions, the device will write status back here */
+	volatile uint64_t status __rte_cache_aligned;
+};
+
 #endif
-- 
2.21.0

Re: [dpdk-dev] [PATCH v3 4/8] raw/ioat: create device on probe and destroy on release

[Burakov, Anatoly]

On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> Add the create/destroy driver functions so that we can actually allocate
> a rawdev and destroy it when done. No rawdev API functions are actually
> implemented at this point.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---

<snip>

>   ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
>   {
> -	RTE_SET_USED(name);
> -	RTE_SET_USED(dev);
> +	static const struct rte_rawdev_ops ioat_rawdev_ops = {
> +	};
> +
> +	struct rte_rawdev *rawdev = NULL;
> +	struct rte_ioat_rawdev *ioat = NULL;
> +	int ret = 0;
> +	int retry = 0;
> +
> +	if (!name) {
> +		IOAT_PMD_ERR("Invalid name of the device!");
> +		ret = -EINVAL;
> +		goto cleanup;
> +	}

Is checking `dev` not necessary here?

> +
> +	/* Allocate device structure */
> +	rawdev = rte_rawdev_pmd_allocate(name, sizeof(struct rte_ioat_rawdev),
> +					 dev->device.numa_node);
> +	if (rawdev == NULL) {
> +		IOAT_PMD_ERR("Unable to allocate raw device");
> +		ret = -EINVAL;
> +		goto cleanup;

EINVAL is supposed to be used to indicate invalid arguments. Inability 
to allocate is not an "invalid arguments" condition. Does 
rte_rawdev_pmd_allocate() set its own errno value? If so, perhaps it 
would be worth passing it on? If not, perhaps -ENOMEM would be a better 
return value?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3 4/8] raw/ioat: create device on probe and destroy on release

[Bruce Richardson]

On Thu, Jun 27, 2019 at 01:09:03PM +0100, Burakov, Anatoly wrote:
> On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> > Add the create/destroy driver functions so that we can actually allocate
> > a rawdev and destroy it when done. No rawdev API functions are actually
> > implemented at this point.
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
> 
> <snip>
> 
> >   ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
> >   {
> > -	RTE_SET_USED(name);
> > -	RTE_SET_USED(dev);
> > +	static const struct rte_rawdev_ops ioat_rawdev_ops = {
> > +	};
> > +
> > +	struct rte_rawdev *rawdev = NULL;
> > +	struct rte_ioat_rawdev *ioat = NULL;
> > +	int ret = 0;
> > +	int retry = 0;
> > +
> > +	if (!name) {
> > +		IOAT_PMD_ERR("Invalid name of the device!");
> > +		ret = -EINVAL;
> > +		goto cleanup;
> > +	}
> 
> Is checking `dev` not necessary here?
> 

No, the only place it's called from is already using the PCI device
structure. I don't think the probe function can ever be called with a NULL
parameter.

> > +
> > +	/* Allocate device structure */
> > +	rawdev = rte_rawdev_pmd_allocate(name, sizeof(struct rte_ioat_rawdev),
> > +					 dev->device.numa_node);
> > +	if (rawdev == NULL) {
> > +		IOAT_PMD_ERR("Unable to allocate raw device");
> > +		ret = -EINVAL;
> > +		goto cleanup;
> 
> EINVAL is supposed to be used to indicate invalid arguments. Inability to
> allocate is not an "invalid arguments" condition. Does
> rte_rawdev_pmd_allocate() set its own errno value? If so, perhaps it would
> be worth passing it on? If not, perhaps -ENOMEM would be a better return
> value?
> 
Fixing in v4.

Re: [dpdk-dev] [PATCH v3 4/8] raw/ioat: create device on probe and destroy on release

[Burakov, Anatoly]

On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> Add the create/destroy driver functions so that we can actually allocate
> a rawdev and destroy it when done. No rawdev API functions are actually
> implemented at this point.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---

<snip>

> +	rawdev->driver_name = dev->device.driver->name;
> +
> +	ioat = rawdev->dev_private;
> +	ioat->rawdev = rawdev;
> +	ioat->regs = dev->mem_resource[0].addr;
> +	ioat->ring_size = 0;
> +	ioat->desc_ring = NULL;
> +	ioat->status_addr = rte_malloc_virt2iova(ioat) +
> +			offsetof(struct rte_ioat_rawdev, status);

While reviewing other patch, i remembered that i've seen this here. You 
can't make any guarantees about IOVA addresses in rte_malloc-allocated 
memory. Are you sure you don't require IOVA-contiguous memory here?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3 4/8] raw/ioat: create device on probe and destroy on release

[Bruce Richardson]

On Thu, Jun 27, 2019 at 01:28:04PM +0100, Burakov, Anatoly wrote:
> On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> > Add the create/destroy driver functions so that we can actually allocate
> > a rawdev and destroy it when done. No rawdev API functions are actually
> > implemented at this point.
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
> 
> <snip>
> 
> > +	rawdev->driver_name = dev->device.driver->name;
> > +
> > +	ioat = rawdev->dev_private;
> > +	ioat->rawdev = rawdev;
> > +	ioat->regs = dev->mem_resource[0].addr;
> > +	ioat->ring_size = 0;
> > +	ioat->desc_ring = NULL;
> > +	ioat->status_addr = rte_malloc_virt2iova(ioat) +
> > +			offsetof(struct rte_ioat_rawdev, status);
> 
> While reviewing other patch, i remembered that i've seen this here. You
> can't make any guarantees about IOVA addresses in rte_malloc-allocated
> memory. Are you sure you don't require IOVA-contiguous memory here?
> 
Presumably we can guarantee that for structures less than 1 page in size,
this will work? I believe the device structure should be within that page
limit.

Re: [dpdk-dev] [PATCH v3 4/8] raw/ioat: create device on probe and destroy on release

[Burakov, Anatoly]

On 28-Jun-19 1:46 PM, Bruce Richardson wrote:
> On Thu, Jun 27, 2019 at 01:28:04PM +0100, Burakov, Anatoly wrote:
>> On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
>>> Add the create/destroy driver functions so that we can actually allocate
>>> a rawdev and destroy it when done. No rawdev API functions are actually
>>> implemented at this point.
>>>
>>> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>>> ---
>>
>> <snip>
>>
>>> +	rawdev->driver_name = dev->device.driver->name;
>>> +
>>> +	ioat = rawdev->dev_private;
>>> +	ioat->rawdev = rawdev;
>>> +	ioat->regs = dev->mem_resource[0].addr;
>>> +	ioat->ring_size = 0;
>>> +	ioat->desc_ring = NULL;
>>> +	ioat->status_addr = rte_malloc_virt2iova(ioat) +
>>> +			offsetof(struct rte_ioat_rawdev, status);
>>
>> While reviewing other patch, i remembered that i've seen this here. You
>> can't make any guarantees about IOVA addresses in rte_malloc-allocated
>> memory. Are you sure you don't require IOVA-contiguous memory here?
>>
> Presumably we can guarantee that for structures less than 1 page in size,
> this will work? I believe the device structure should be within that page
> limit.
> 

No, we can't. That would only be true if you were allocating 
IOVA-contiguous memory. Otherwise there's nothing stopping the allocator 
to allocate even a few kilobytes across page boundary.

You can only ever guarantee that *one cache line* will not cross the 
page boundary with rte_malloc. With rte_memzone and IOVA_CONTIG flag, 
you'll be able to guarantee IOVA-contiguousness in all cases (or 
allocation failure).

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3 4/8] raw/ioat: create device on probe and destroy on release

[Bruce Richardson]

On Fri, Jun 28, 2019 at 01:59:26PM +0100, Burakov, Anatoly wrote:
> On 28-Jun-19 1:46 PM, Bruce Richardson wrote:
> > On Thu, Jun 27, 2019 at 01:28:04PM +0100, Burakov, Anatoly wrote:
> > > On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> > > > Add the create/destroy driver functions so that we can actually allocate
> > > > a rawdev and destroy it when done. No rawdev API functions are actually
> > > > implemented at this point.
> > > > 
> > > > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > > > ---
> > > 
> > > <snip>
> > > 
> > > > +	rawdev->driver_name = dev->device.driver->name;
> > > > +
> > > > +	ioat = rawdev->dev_private;
> > > > +	ioat->rawdev = rawdev;
> > > > +	ioat->regs = dev->mem_resource[0].addr;
> > > > +	ioat->ring_size = 0;
> > > > +	ioat->desc_ring = NULL;
> > > > +	ioat->status_addr = rte_malloc_virt2iova(ioat) +
> > > > +			offsetof(struct rte_ioat_rawdev, status);
> > > 
> > > While reviewing other patch, i remembered that i've seen this here. You
> > > can't make any guarantees about IOVA addresses in rte_malloc-allocated
> > > memory. Are you sure you don't require IOVA-contiguous memory here?
> > > 
> > Presumably we can guarantee that for structures less than 1 page in size,
> > this will work? I believe the device structure should be within that page
> > limit.
> > 
> 
> No, we can't. That would only be true if you were allocating IOVA-contiguous
> memory. Otherwise there's nothing stopping the allocator to allocate even a
> few kilobytes across page boundary.
> 
> You can only ever guarantee that *one cache line* will not cross the page
> boundary with rte_malloc. With rte_memzone and IOVA_CONTIG flag, you'll be
> able to guarantee IOVA-contiguousness in all cases (or allocation failure).
> 
Ok, so I either need to move this field to the start of the structure, i.e.
have offset zero, or else use contiguous allocation. Will fix in next
version.

/Bruce

Re: [dpdk-dev] [PATCH v3 4/8] raw/ioat: create device on probe and destroy on release

[Burakov, Anatoly]

On 28-Jun-19 2:15 PM, Bruce Richardson wrote:
> On Fri, Jun 28, 2019 at 01:59:26PM +0100, Burakov, Anatoly wrote:
>> On 28-Jun-19 1:46 PM, Bruce Richardson wrote:
>>> On Thu, Jun 27, 2019 at 01:28:04PM +0100, Burakov, Anatoly wrote:
>>>> On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
>>>>> Add the create/destroy driver functions so that we can actually allocate
>>>>> a rawdev and destroy it when done. No rawdev API functions are actually
>>>>> implemented at this point.
>>>>>
>>>>> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>>>>> ---
>>>>
>>>> <snip>
>>>>
>>>>> +	rawdev->driver_name = dev->device.driver->name;
>>>>> +
>>>>> +	ioat = rawdev->dev_private;
>>>>> +	ioat->rawdev = rawdev;
>>>>> +	ioat->regs = dev->mem_resource[0].addr;
>>>>> +	ioat->ring_size = 0;
>>>>> +	ioat->desc_ring = NULL;
>>>>> +	ioat->status_addr = rte_malloc_virt2iova(ioat) +
>>>>> +			offsetof(struct rte_ioat_rawdev, status);
>>>>
>>>> While reviewing other patch, i remembered that i've seen this here. You
>>>> can't make any guarantees about IOVA addresses in rte_malloc-allocated
>>>> memory. Are you sure you don't require IOVA-contiguous memory here?
>>>>
>>> Presumably we can guarantee that for structures less than 1 page in size,
>>> this will work? I believe the device structure should be within that page
>>> limit.
>>>
>>
>> No, we can't. That would only be true if you were allocating IOVA-contiguous
>> memory. Otherwise there's nothing stopping the allocator to allocate even a
>> few kilobytes across page boundary.
>>
>> You can only ever guarantee that *one cache line* will not cross the page
>> boundary with rte_malloc. With rte_memzone and IOVA_CONTIG flag, you'll be
>> able to guarantee IOVA-contiguousness in all cases (or allocation failure).
>>
> Ok, so I either need to move this field to the start of the structure, i.e.
> have offset zero, or else use contiguous allocation. Will fix in next
> version.
> 
> /Bruce
> 

The latter is probably more explicit in intention, i'd rather the code 
not rely on details of rte_malloc implementation :)

-- 
Thanks,
Anatoly

[dpdk-dev] [PATCH v3 5/8] raw/ioat: add device info function

[Bruce Richardson]

Add in the "info_get" function to the driver, to allow us to query the
device. This allows us to have the unit test pick up the presence of
supported hardware or not.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---

V2: Test case is placed in driver self-test routine
---
 app/test/test_rawdev.c             | 11 ++++++++++
 doc/guides/rawdevs/ioat_rawdev.rst | 34 ++++++++++++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev.c     | 11 ++++++++++
 drivers/raw/ioat/rte_ioat_rawdev.h | 11 ++++++++++
 4 files changed, 67 insertions(+)

diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
index 88549fb61..4db762b4c 100644
--- a/app/test/test_rawdev.c
+++ b/app/test/test_rawdev.c
@@ -29,6 +29,17 @@ REGISTER_TEST_COMMAND(rawdev_autotest, test_rawdev_selftest_skeleton);
 static int
 test_rawdev_selftest_ioat(void)
 {
+	const int count = rte_rawdev_count();
+	int i;
+
+	for (i = 0; i < count; i++) {
+		struct rte_rawdev_info info = { .dev_private = NULL };
+		if (rte_rawdev_info_get(i, &info) == 0 &&
+				strstr(info.driver_name, "ioat") != NULL)
+			return 0;
+	}
+
+	printf("No IOAT rawdev found, skipping tests\n");
 	return TEST_SKIPPED;
 }
 
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 4b7fe8a8f..0ce984490 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -83,3 +83,37 @@ parameters need to be passed to create or initialize the device.
 Once probed successfully, the device will appear as a ``rawdev``, that is a
 "raw device type" inside DPDK, and can be accessed using APIs from the
 ``rte_rawdev`` library.
+
+Using IOAT Rawdev Devices
+--------------------------
+
+To use the devices from an application, the rawdev API can be used, along
+with definitions taken from the device-specific header file
+``rte_ioat_rawdev.h``. This header is needed to get the definition of
+structure parameters used by some of the rawdev APIs for IOAT rawdev
+devices, as well as providing key functions for using the device for memory
+copies.
+
+Getting Device Information
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Basic information about each rawdev device can be got using the
+``rte_rawdev_info_get()`` API. For most applications, this API will be
+needed to verify that the rawdev in question is of the expected type. For
+example, the following code in ``test_ioat_rawdev.c`` is used to identify
+the IOAT rawdev device for use for the tests:
+
+.. code-block:: C
+
+        for (i = 0; i < count && !found; i++) {
+                struct rte_rawdev_info info = { .dev_private = NULL };
+                found = (rte_rawdev_info_get(i, &info) == 0 &&
+                                strcmp(info.driver_name,
+                                                IOAT_PMD_RAWDEV_NAME_STR) == 0);
+        }
+
+When calling the ``rte_rawdev_info_get()`` API for an IOAT rawdev device,
+the ``dev_private`` field in the ``rte_rawdev_info`` struct should either
+be NULL, or else be set to point to a structure of type
+``rte_ioat_rawdev_config``, in which case the size of the configured device
+input ring will be returned in that structure.
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index d13391dd5..08e7586c6 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -24,10 +24,21 @@ static struct rte_pci_driver ioat_pmd_drv;
 #define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
 #define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
 
+static void
+ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
+{
+	struct rte_ioat_rawdev_config *cfg = dev_info;
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+
+	if (cfg != NULL)
+		cfg->ring_size = ioat->ring_size;
+}
+
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
 	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+			.dev_info_get = ioat_dev_info_get,
 	};
 
 	struct rte_rawdev *rawdev = NULL;
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index c3216a174..7e0d72ca3 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -24,6 +24,17 @@
 /** Name used to adjust the log level for this driver */
 #define IOAT_PMD_LOG_NAME "rawdev.ioat"
 
+/**
+ * Configuration structure for an ioat rawdev instance
+ *
+ * This structure is to be passed as the ".dev_private" parameter when
+ * calling the rte_rawdev_get_info() and rte_rawdev_configure() APIs on
+ * an ioat rawdev instance.
+ */
+struct rte_ioat_rawdev_config {
+	unsigned short ring_size;
+};
+
 /**
  * @internal
  * Structure representing a device instance
-- 
2.21.0

Re: [dpdk-dev] [PATCH v3 5/8] raw/ioat: add device info function

[Burakov, Anatoly]

On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> Add in the "info_get" function to the driver, to allow us to query the
> device. This allows us to have the unit test pick up the presence of
> supported hardware or not.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> 
> ---
> 
> V2: Test case is placed in driver self-test routine
> ---
>   app/test/test_rawdev.c             | 11 ++++++++++
>   doc/guides/rawdevs/ioat_rawdev.rst | 34 ++++++++++++++++++++++++++++++
>   drivers/raw/ioat/ioat_rawdev.c     | 11 ++++++++++
>   drivers/raw/ioat/rte_ioat_rawdev.h | 11 ++++++++++
>   4 files changed, 67 insertions(+)
> 
> diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
> index 88549fb61..4db762b4c 100644
> --- a/app/test/test_rawdev.c
> +++ b/app/test/test_rawdev.c
> @@ -29,6 +29,17 @@ REGISTER_TEST_COMMAND(rawdev_autotest, test_rawdev_selftest_skeleton);
>   static int
>   test_rawdev_selftest_ioat(void)
>   {
> +	const int count = rte_rawdev_count();
> +	int i;
> +
> +	for (i = 0; i < count; i++) {
> +		struct rte_rawdev_info info = { .dev_private = NULL };
> +		if (rte_rawdev_info_get(i, &info) == 0 &&
> +				strstr(info.driver_name, "ioat") != NULL)
> +			return 0;

TEST_SUCCESS? Also, didn't "ioat" have a macro for its name?

> +	}
> +
> +	printf("No IOAT rawdev found, skipping tests\n");
>   	return TEST_SKIPPED;
>   }
>   
> diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
> index 4b7fe8a8f..0ce984490 100644
> --- a/doc/guides/rawdevs/ioat_rawdev.rst
> +++ b/doc/guides/rawdevs/ioat_rawdev.rst
> @@ -83,3 +83,37 @@ parameters need to be passed to create or initialize the device.
>   Once probed successfully, the device will appear as a ``rawdev``, that is a
>   "raw device type" inside DPDK, and can be accessed using APIs from the
>   ``rte_rawdev`` library.
> +
> +Using IOAT Rawdev Devices
> +--------------------------
> +
> +To use the devices from an application, the rawdev API can be used, along
> +with definitions taken from the device-specific header file
> +``rte_ioat_rawdev.h``. This header is needed to get the definition of
> +structure parameters used by some of the rawdev APIs for IOAT rawdev
> +devices, as well as providing key functions for using the device for memory
> +copies.
> +
> +Getting Device Information
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Basic information about each rawdev device can be got using the

According to certain unreliable sources [1], usage of "can be got" has 
not been used since the end of 19th century. I didn't think you were 
*that* old, Bruce!

> +``rte_rawdev_info_get()`` API. For most applications, this API will be
> +needed to verify that the rawdev in question is of the expected type. For
> +example, the following code in ``test_ioat_rawdev.c`` is used to identify
> +the IOAT rawdev device for use for the tests:
> +
> +.. code-block:: C
> +
> +        for (i = 0; i < count && !found; i++) {
> +                struct rte_rawdev_info info = { .dev_private = NULL };
> +                found = (rte_rawdev_info_get(i, &info) == 0 &&
> +                                strcmp(info.driver_name,
> +                                                IOAT_PMD_RAWDEV_NAME_STR) == 0);
> +        }

The code here doesn't match the actual implementation in the autotest.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3 5/8] raw/ioat: add device info function

[Bruce Richardson]

On Thu, Jun 27, 2019 at 01:16:01PM +0100, Burakov, Anatoly wrote:
> On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> > Add in the "info_get" function to the driver, to allow us to query the
> > device. This allows us to have the unit test pick up the presence of
> > supported hardware or not.
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > 
> > ---
> > 
> > V2: Test case is placed in driver self-test routine
> > ---
> >   app/test/test_rawdev.c             | 11 ++++++++++
> >   doc/guides/rawdevs/ioat_rawdev.rst | 34 ++++++++++++++++++++++++++++++
> >   drivers/raw/ioat/ioat_rawdev.c     | 11 ++++++++++
> >   drivers/raw/ioat/rte_ioat_rawdev.h | 11 ++++++++++
> >   4 files changed, 67 insertions(+)
> > 
> > diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
> > index 88549fb61..4db762b4c 100644
> > --- a/app/test/test_rawdev.c
> > +++ b/app/test/test_rawdev.c
> > @@ -29,6 +29,17 @@ REGISTER_TEST_COMMAND(rawdev_autotest, test_rawdev_selftest_skeleton);
> >   static int
> >   test_rawdev_selftest_ioat(void)
> >   {
> > +	const int count = rte_rawdev_count();
> > +	int i;
> > +
> > +	for (i = 0; i < count; i++) {
> > +		struct rte_rawdev_info info = { .dev_private = NULL };
> > +		if (rte_rawdev_info_get(i, &info) == 0 &&
> > +				strstr(info.driver_name, "ioat") != NULL)
> > +			return 0;
> 
> TEST_SUCCESS? Also, didn't "ioat" have a macro for its name?
> 

Ack on the TEST_SUCCESS part.

For the seond point, yes, there is a macro for the name, but to get it one
has to include the header file for the driver, which will not be available
if the driver is not being built. Therefore, I just use a basic check for
the "ioat" name, rather than cluttering up the code with lots of
conditionals for handling cases where the driver won't be built.

/Bruce

[dpdk-dev] [PATCH v3 6/8] raw/ioat: add configure, start and stop functions

[Bruce Richardson]

Allow initializing a driver instance. Include selftest to validate these
functions.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---

V3: don't add a new descriptor format struct, reuse from rte_ioat_spec.h
V2: test cases placed in self-test routine
---
 app/test/test_rawdev.c              |  2 +-
 doc/guides/rawdevs/ioat_rawdev.rst  | 32 ++++++++++++
 drivers/raw/ioat/Makefile           |  1 +
 drivers/raw/ioat/ioat_rawdev.c      | 78 +++++++++++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev_test.c | 41 +++++++++++++++
 drivers/raw/ioat/meson.build        |  3 +-
 drivers/raw/ioat/rte_ioat_rawdev.h  |  4 +-
 7 files changed, 158 insertions(+), 3 deletions(-)
 create mode 100644 drivers/raw/ioat/ioat_rawdev_test.c

diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
index 4db762b4c..731e51717 100644
--- a/app/test/test_rawdev.c
+++ b/app/test/test_rawdev.c
@@ -36,7 +36,7 @@ test_rawdev_selftest_ioat(void)
 		struct rte_rawdev_info info = { .dev_private = NULL };
 		if (rte_rawdev_info_get(i, &info) == 0 &&
 				strstr(info.driver_name, "ioat") != NULL)
-			return 0;
+			return rte_rawdev_selftest(i);
 	}
 
 	printf("No IOAT rawdev found, skipping tests\n");
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 0ce984490..9ab97e2aa 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -117,3 +117,35 @@ the ``dev_private`` field in the ``rte_rawdev_info`` struct should either
 be NULL, or else be set to point to a structure of type
 ``rte_ioat_rawdev_config``, in which case the size of the configured device
 input ring will be returned in that structure.
+
+Device Configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+Configuring an IOAT rawdev device is done using the
+``rte_rawdev_configure()`` API, which takes the same structure parameters
+as the, previously referenced, ``rte_rawdev_info_get()`` API. The main
+difference is that, because the parameter is used as input rather than
+output, the ``dev_private`` structure element cannot be NULL, and must
+point to a valid ``rte_ioat_rawdev_config`` structure, containing the ring
+size to be used by the device. The ring size must be a power of two,
+between 64 and 4096.
+
+The following code shows how the device is configured in
+``test_ioat_rawdev.c``:
+
+.. code-block:: C
+
+   #define IOAT_TEST_RINGSIZE 512
+        struct rte_ioat_rawdev_config p = { .ring_size = -1 };
+        struct rte_rawdev_info info = { .dev_private = &p };
+
+        /* ... */
+
+        p.ring_size = IOAT_TEST_RINGSIZE;
+        if (rte_rawdev_configure(dev_id, &info) != 0) {
+                printf("Error with rte_rawdev_configure()\n");
+                return -1;
+        }
+
+Once configured, the device can then be made ready for use by calling the
+``rte_rawdev_start()`` API.
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
index 1e10938f3..b1af9c666 100644
--- a/drivers/raw/ioat/Makefile
+++ b/drivers/raw/ioat/Makefile
@@ -21,6 +21,7 @@ EXPORT_MAP := rte_pmd_ioat_version.map
 
 # library source files
 SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev_test.c
 
 # export include files
 SYMLINK-y-include += rte_ioat_rawdev.h
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index 08e7586c6..fd1653da2 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -24,6 +24,78 @@ static struct rte_pci_driver ioat_pmd_drv;
 #define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
 #define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
 
+#define DESC_SZ sizeof(struct rte_ioat_generic_hw_desc)
+#define COMPLETION_SZ sizeof(__m128i)
+
+static int
+ioat_dev_configure(const struct rte_rawdev *dev, rte_rawdev_obj_t config)
+{
+	struct rte_ioat_rawdev_config *params = config;
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+	unsigned short i;
+
+	if (dev->started)
+		return -EBUSY;
+
+	if (params == NULL)
+		return -EINVAL;
+
+	if (params->ring_size > 4096 || params->ring_size < 64 ||
+			!rte_is_power_of_2(params->ring_size))
+		return -EINVAL;
+
+	ioat->ring_size = params->ring_size;
+	if (ioat->desc_ring != NULL) {
+		rte_free(ioat->desc_ring);
+		ioat->desc_ring = NULL;
+	}
+
+	/* allocate one block of memory for both descriptors
+	 * and completion handles.
+	 */
+	ioat->desc_ring = rte_zmalloc_socket(NULL,
+			(DESC_SZ + COMPLETION_SZ) * ioat->ring_size,
+			0, /* alignment, default to 64Byte */
+			dev->device->numa_node);
+	if (ioat->desc_ring == NULL)
+		return -ENOMEM;
+	ioat->hdls = (void *)&ioat->desc_ring[ioat->ring_size];
+
+	ioat->ring_addr = rte_malloc_virt2iova(ioat->desc_ring);
+
+	/* configure descriptor ring - each one points to next */
+	for (i = 0; i < ioat->ring_size; i++) {
+		ioat->desc_ring[i].next = ioat->ring_addr +
+				(((i + 1) % ioat->ring_size) * DESC_SZ);
+	}
+
+	return 0;
+}
+
+static int
+ioat_dev_start(struct rte_rawdev *dev)
+{
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+
+	if (ioat->ring_size == 0 || ioat->desc_ring == NULL)
+		return -EBUSY;
+
+	/* inform hardware of where the descriptor ring is */
+	ioat->regs->chainaddr = ioat->ring_addr;
+	/* inform hardware of where to write the status/completions */
+	ioat->regs->chancmp = ioat->status_addr;
+
+	/* prime the status register to be set to the last element */
+	ioat->status =  ioat->ring_addr + ((ioat->ring_size - 1) * DESC_SZ);
+	return 0;
+}
+
+static void
+ioat_dev_stop(struct rte_rawdev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
 static void
 ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
 {
@@ -34,11 +106,17 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
 		cfg->ring_size = ioat->ring_size;
 }
 
+int ioat_rawdev_test(uint16_t dev_id);
+
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
 	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+			.dev_configure = ioat_dev_configure,
+			.dev_start = ioat_dev_start,
+			.dev_stop = ioat_dev_stop,
 			.dev_info_get = ioat_dev_info_get,
+			.dev_selftest = ioat_rawdev_test,
 	};
 
 	struct rte_rawdev *rawdev = NULL;
diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c
new file mode 100644
index 000000000..5375da26c
--- /dev/null
+++ b/drivers/raw/ioat/ioat_rawdev_test.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include "rte_rawdev.h"
+#include "rte_ioat_rawdev.h"
+
+int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */
+
+int
+ioat_rawdev_test(uint16_t dev_id)
+{
+#define IOAT_TEST_RINGSIZE 512
+	struct rte_ioat_rawdev_config p = { .ring_size = -1 };
+	struct rte_rawdev_info info = { .dev_private = &p };
+
+	rte_rawdev_info_get(dev_id, &info);
+	if (p.ring_size != 0) {
+		printf("Error, initial ring size is non-zero (%d)\n",
+				(int)p.ring_size);
+		return -1;
+	}
+
+	p.ring_size = IOAT_TEST_RINGSIZE;
+	if (rte_rawdev_configure(dev_id, &info) != 0) {
+		printf("Error with rte_rawdev_configure()\n");
+		return -1;
+	}
+	rte_rawdev_info_get(dev_id, &info);
+	if (p.ring_size != IOAT_TEST_RINGSIZE) {
+		printf("Error, ring size is not %d (%d)\n",
+				IOAT_TEST_RINGSIZE, (int)p.ring_size);
+		return -1;
+	}
+
+	if (rte_rawdev_start(dev_id) != 0) {
+		printf("Error with rte_rawdev_start()\n");
+		return -1;
+	}
+	return 0;
+}
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
index ca23e23fc..40fff6654 100644
--- a/drivers/raw/ioat/meson.build
+++ b/drivers/raw/ioat/meson.build
@@ -2,7 +2,8 @@
 # Copyright 2019 Intel Corporation
 
 build = dpdk_conf.has('RTE_ARCH_X86')
-sources = files('ioat_rawdev.c')
+sources = files('ioat_rawdev.c',
+		'ioat_rawdev_test.c')
 deps += ['rawdev', 'bus_pci']
 
 install_headers('rte_ioat_rawdev.h',
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index 7e0d72ca3..c09fd0791 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -14,6 +14,7 @@
  * @b EXPERIMENTAL: these structures and APIs may change without prior notice
  */
 
+#include <x86intrin.h>
 #include <rte_memory.h>
 #include <rte_ioat_spec.h>
 
@@ -46,7 +47,8 @@ struct rte_ioat_rawdev {
 	phys_addr_t ring_addr;
 
 	unsigned short ring_size;
-	struct rte_ioat_desc *desc_ring;
+	struct rte_ioat_generic_hw_desc *desc_ring;
+	__m128i *hdls; /* completion handles for returning to user */
 
 	/* to report completions, the device will write status back here */
 	volatile uint64_t status __rte_cache_aligned;
-- 
2.21.0

Re: [dpdk-dev] [PATCH v3 6/8] raw/ioat: add configure, start and stop functions

[Burakov, Anatoly]

On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> Allow initializing a driver instance. Include selftest to validate these
> functions.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> 
> ---
> 
> V3: don't add a new descriptor format struct, reuse from rte_ioat_spec.h
> V2: test cases placed in self-test routine
> ---
>   app/test/test_rawdev.c              |  2 +-
>   doc/guides/rawdevs/ioat_rawdev.rst  | 32 ++++++++++++
>   drivers/raw/ioat/Makefile           |  1 +
>   drivers/raw/ioat/ioat_rawdev.c      | 78 +++++++++++++++++++++++++++++
>   drivers/raw/ioat/ioat_rawdev_test.c | 41 +++++++++++++++
>   drivers/raw/ioat/meson.build        |  3 +-
>   drivers/raw/ioat/rte_ioat_rawdev.h  |  4 +-
>   7 files changed, 158 insertions(+), 3 deletions(-)
>   create mode 100644 drivers/raw/ioat/ioat_rawdev_test.c
> 
> diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
> index 4db762b4c..731e51717 100644
> --- a/app/test/test_rawdev.c
> +++ b/app/test/test_rawdev.c
> @@ -36,7 +36,7 @@ test_rawdev_selftest_ioat(void)
>   		struct rte_rawdev_info info = { .dev_private = NULL };
>   		if (rte_rawdev_info_get(i, &info) == 0 &&
>   				strstr(info.driver_name, "ioat") != NULL)
> -			return 0;
> +			return rte_rawdev_selftest(i);

Even though it doesn't matter in practice, technically, we can't pass a 
raw return value to the test caller. It should be TEST_SUCCESS or 
TEST_FAILURE.

>   	}
>   
>   	printf("No IOAT rawdev found, skipping tests\n");
> diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
> index 0ce984490..9ab97e2aa 100644
> --- a/doc/guides/rawdevs/ioat_rawdev.rst
> +++ b/doc/guides/rawdevs/ioat_rawdev.rst
> @@ -117,3 +117,35 @@ the ``dev_private`` field in the ``rte_rawdev_info`` struct should either
>   be NULL, or else be set to point to a structure of type
>   ``rte_ioat_rawdev_config``, in which case the size of the configured device
>   input ring will be returned in that structure.
> +
> +Device Configuration
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +Configuring an IOAT rawdev device is done using the
> +``rte_rawdev_configure()`` API, which takes the same structure parameters
> +as the, previously referenced, ``rte_rawdev_info_get()`` API. The main
> +difference is that, because the parameter is used as input rather than
> +output, the ``dev_private`` structure element cannot be NULL, and must
> +point to a valid ``rte_ioat_rawdev_config`` structure, containing the ring
> +size to be used by the device. The ring size must be a power of two,
> +between 64 and 4096.

<snip>

> +	if (params->ring_size > 4096 || params->ring_size < 64 ||
> +			!rte_is_power_of_2(params->ring_size))
> +		return -EINVAL;
> +
> +	ioat->ring_size = params->ring_size;
> +	if (ioat->desc_ring != NULL) {
> +		rte_free(ioat->desc_ring);
> +		ioat->desc_ring = NULL;
> +	}
> +
> +	/* allocate one block of memory for both descriptors
> +	 * and completion handles.
> +	 */
> +	ioat->desc_ring = rte_zmalloc_socket(NULL,
> +			(DESC_SZ + COMPLETION_SZ) * ioat->ring_size,
> +			0, /* alignment, default to 64Byte */
> +			dev->device->numa_node);

Using rte_zmalloc for hardware structures seems suspect. Do you not need 
IOVA-contiguous memory here?

> +	if (ioat->desc_ring == NULL)
> +		return -ENOMEM;
> +	ioat->hdls = (void *)&ioat->desc_ring[ioat->ring_size];
> +
> +	ioat->ring_addr = rte_malloc_virt2iova(ioat->desc_ring);
> +
> +	/* configure descriptor ring - each one points to next */
> +	for (i = 0; i < ioat->ring_size; i++) {
> +		ioat->desc_ring[i].next = ioat->ring_addr +
> +				(((i + 1) % ioat->ring_size) * DESC_SZ);
> +	}

OK, this *definitely* looks suspect :) with rte_zmalloc(), there's no 
guarantee that the entire allocated block resides on the same physical 
page, so you can't assume IOVA addresses will be contiguous either, 
unless you only intend to operate in IOVA as VA mode (which i didn't 
notice).

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3 6/8] raw/ioat: add configure, start and stop functions

[Pattan, Reshma]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> Sent: Thursday, June 27, 2019 11:41 AM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; jerinj@marvell.com; Richardson, Bruce
> <bruce.richardson@intel.com>
> Subject: [dpdk-dev] [PATCH v3 6/8] raw/ioat: add configure, start and stop
> functions

<snip>

> +int ioat_rawdev_test(uint16_t dev_id);
> +
>  static int
>  ioat_rawdev_create(const char *name, struct rte_pci_device *dev)  {
>  	static const struct rte_rawdev_ops ioat_rawdev_ops = {
> +			.dev_configure = ioat_dev_configure,
> +			.dev_start = ioat_dev_start,
> +			.dev_stop = ioat_dev_stop,
>  			.dev_info_get = ioat_dev_info_get,
> +			.dev_selftest = ioat_rawdev_test,

Build fail for  ./devtools/test-build.sh x86_64-native-linux-gcc+next+shared
/drivers/raw/ioat/ioat_rawdev.c: In function 'ioat_rawdev_create':
drivers/raw/ioat/ioat_rawdev.c:163:20: error: initialization of 'int (*)(void)' from incompatible pointer type 'int (*)(uint16_t)' {aka 'int (*)(short unsigned int)'} [-Werror=incompatible-pointer-types]
  163 |    .dev_selftest = ioat_rawdev_test,

Thanks,
Reshma

Re: [dpdk-dev] [PATCH v3 6/8] raw/ioat: add configure, start and stop functions

[Bruce Richardson]

On Thu, Jun 27, 2019 at 05:37:33PM +0100, Pattan, Reshma wrote:
> 
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> > Sent: Thursday, June 27, 2019 11:41 AM
> > To: dev@dpdk.org
> > Cc: thomas@monjalon.net; jerinj@marvell.com; Richardson, Bruce
> > <bruce.richardson@intel.com>
> > Subject: [dpdk-dev] [PATCH v3 6/8] raw/ioat: add configure, start and stop
> > functions
> 
> <snip>
> 
> > +int ioat_rawdev_test(uint16_t dev_id);
> > +
> >  static int
> >  ioat_rawdev_create(const char *name, struct rte_pci_device *dev)  {
> >  	static const struct rte_rawdev_ops ioat_rawdev_ops = {
> > +			.dev_configure = ioat_dev_configure,
> > +			.dev_start = ioat_dev_start,
> > +			.dev_stop = ioat_dev_stop,
> >  			.dev_info_get = ioat_dev_info_get,
> > +			.dev_selftest = ioat_rawdev_test,
> 
> Build fail for  ./devtools/test-build.sh x86_64-native-linux-gcc+next+shared
> /drivers/raw/ioat/ioat_rawdev.c: In function 'ioat_rawdev_create':
> drivers/raw/ioat/ioat_rawdev.c:163:20: error: initialization of 'int (*)(void)' from incompatible pointer type 'int (*)(uint16_t)' {aka 'int (*)(short unsigned int)'} [-Werror=incompatible-pointer-types]
>   163 |    .dev_selftest = ioat_rawdev_test,
> 

Apologies, I forgot to put in the cover letter that this depends upon the
rawdev patchset I previously upstreamed: 

http://patches.dpdk.org/project/dpdk/list/?series=5120

[dpdk-dev] [PATCH v3 7/8] raw/ioat: add statistics functions

[Bruce Richardson]

Add stats functions to track what is happening in the driver, and put
unit tests to check those.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---

V2: test cases moved to self-test routine
---
 doc/guides/rawdevs/ioat_rawdev.rst  | 14 +++++++++
 drivers/raw/ioat/ioat_rawdev.c      | 44 +++++++++++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev_test.c | 39 +++++++++++++++++++++++++
 drivers/raw/ioat/rte_ioat_rawdev.h  |  6 ++++
 4 files changed, 103 insertions(+)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 9ab97e2aa..b908f31e0 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -149,3 +149,17 @@ The following code shows how the device is configured in
 
 Once configured, the device can then be made ready for use by calling the
 ``rte_rawdev_start()`` API.
+
+Querying Device Statistics
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The statistics from the IOAT rawdev device can be got via the xstats
+functions in the ``rte_rawdev`` library, i.e.
+``rte_rawdev_xstats_names_get()``, ``rte_rawdev_xstats_get()`` and
+``rte_rawdev_xstats_by_name_get``. The statistics returned for each device
+instance are:
+
+* ``failed_enqueues``
+* ``successful_enqueues``
+* ``copies_started``
+* ``copies_completed``
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index fd1653da2..0dde0e820 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -4,6 +4,7 @@
 
 #include <rte_cycles.h>
 #include <rte_bus_pci.h>
+#include <rte_string_fns.h>
 #include <rte_rawdev_pmd.h>
 
 #include "rte_ioat_rawdev.h"
@@ -106,6 +107,47 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
 		cfg->ring_size = ioat->ring_size;
 }
 
+static const char * const xstat_names[] = {
+		"failed_enqueues", "successful_enqueues",
+		"copies_started", "copies_completed"
+};
+
+static int
+ioat_xstats_get(const struct rte_rawdev *dev, const unsigned int ids[],
+		uint64_t values[], unsigned int n)
+{
+	const struct rte_ioat_rawdev *ioat = dev->dev_private;
+	unsigned int i;
+
+	for (i = 0; i < n; i++) {
+		switch (ids[i]) {
+		case 0: values[i] = ioat->enqueue_failed; break;
+		case 1: values[i] = ioat->enqueued; break;
+		case 2: values[i] = ioat->started; break;
+		case 3: values[i] = ioat->completed; break;
+		default: values[i] = 0; break;
+		}
+	}
+	return n;
+}
+
+static int
+ioat_xstats_get_names(const struct rte_rawdev *dev,
+		struct rte_rawdev_xstats_name *names,
+		unsigned int size)
+{
+	unsigned int i;
+
+	RTE_SET_USED(dev);
+	if (size < RTE_DIM(xstat_names))
+		return RTE_DIM(xstat_names);
+
+	for (i = 0; i < RTE_DIM(xstat_names); i++)
+		strlcpy(names[i].name, xstat_names[i], sizeof(names[i]));
+
+	return RTE_DIM(xstat_names);
+}
+
 int ioat_rawdev_test(uint16_t dev_id);
 
 static int
@@ -116,6 +158,8 @@ ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 			.dev_start = ioat_dev_start,
 			.dev_stop = ioat_dev_stop,
 			.dev_info_get = ioat_dev_info_get,
+			.xstats_get = ioat_xstats_get,
+			.xstats_get_names = ioat_xstats_get_names,
 			.dev_selftest = ioat_rawdev_test,
 	};
 
diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c
index 5375da26c..5098d71ae 100644
--- a/drivers/raw/ioat/ioat_rawdev_test.c
+++ b/drivers/raw/ioat/ioat_rawdev_test.c
@@ -2,6 +2,7 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <inttypes.h>
 #include "rte_rawdev.h"
 #include "rte_ioat_rawdev.h"
 
@@ -13,6 +14,11 @@ ioat_rawdev_test(uint16_t dev_id)
 #define IOAT_TEST_RINGSIZE 512
 	struct rte_ioat_rawdev_config p = { .ring_size = -1 };
 	struct rte_rawdev_info info = { .dev_private = &p };
+	struct rte_rawdev_xstats_name *snames = NULL;
+	uint64_t *stats = NULL;
+	unsigned int *ids = NULL;
+	unsigned int nb_xstats;
+	unsigned int i;
 
 	rte_rawdev_info_get(dev_id, &info);
 	if (p.ring_size != 0) {
@@ -37,5 +43,38 @@ ioat_rawdev_test(uint16_t dev_id)
 		printf("Error with rte_rawdev_start()\n");
 		return -1;
 	}
+
+	/* allocate memory for xstats names and values */
+	nb_xstats = rte_rawdev_xstats_names_get(dev_id, NULL, 0);
+
+	snames = malloc(sizeof(*snames) * nb_xstats);
+	if (snames == NULL) {
+		printf("Error allocating xstat names memory\n");
+		return -1;
+	}
+	rte_rawdev_xstats_names_get(dev_id, snames, nb_xstats);
+
+	ids = malloc(sizeof(*ids) * nb_xstats);
+	if (ids == NULL) {
+		printf("Error allocating xstat ids memory\n");
+		return -1;
+	}
+	for (i = 0; i < nb_xstats; i++)
+		ids[i] = i;
+
+	stats = malloc(sizeof(*stats) * nb_xstats);
+	if (stats == NULL) {
+		printf("Error allocating xstat memory\n");
+		return -1;
+	}
+
+	rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
+	for (i = 0; i < nb_xstats; i++)
+		printf("%s: %"PRIu64"   ", snames[i].name, stats[i]);
+	printf("\n");
+
+	free(snames);
+	free(stats);
+	free(ids);
 	return 0;
 }
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index c09fd0791..aa534f9cf 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -50,6 +50,12 @@ struct rte_ioat_rawdev {
 	struct rte_ioat_generic_hw_desc *desc_ring;
 	__m128i *hdls; /* completion handles for returning to user */
 
+	/* some statistics for tracking, if added/changed update xstats fns*/
+	uint64_t enqueue_failed __rte_cache_aligned;
+	uint64_t enqueued;
+	uint64_t started;
+	uint64_t completed;
+
 	/* to report completions, the device will write status back here */
 	volatile uint64_t status __rte_cache_aligned;
 };
-- 
2.21.0

Re: [dpdk-dev] [PATCH v3 7/8] raw/ioat: add statistics functions

[Burakov, Anatoly]

On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> Add stats functions to track what is happening in the driver, and put
> unit tests to check those.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---

<snip>

> +	/* allocate memory for xstats names and values */
> +	nb_xstats = rte_rawdev_xstats_names_get(dev_id, NULL, 0);
> +
> +	snames = malloc(sizeof(*snames) * nb_xstats);
> +	if (snames == NULL) {
> +		printf("Error allocating xstat names memory\n");
> +		return -1;
> +	}
> +	rte_rawdev_xstats_names_get(dev_id, snames, nb_xstats);
> +
> +	ids = malloc(sizeof(*ids) * nb_xstats);
> +	if (ids == NULL) {
> +		printf("Error allocating xstat ids memory\n");
> +		return -1;

Leaking snames here.

> +	}
> +	for (i = 0; i < nb_xstats; i++)
> +		ids[i] = i;
> +
> +	stats = malloc(sizeof(*stats) * nb_xstats);
> +	if (stats == NULL) {
> +		printf("Error allocating xstat memory\n");
> +		return -1;

Leaking snames and ids here. Perhaps a goto and a ret value wouldn't hurt :)

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3 7/8] raw/ioat: add statistics functions

[Pattan, Reshma]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> Sent: Thursday, June 27, 2019 11:41 AM
> To: dev@dpdk.org


<snip>

> +static int
> +ioat_xstats_get(const struct rte_rawdev *dev, const unsigned int ids[],
> +		uint64_t values[], unsigned int n)
> +{
> +	const struct rte_ioat_rawdev *ioat = dev->dev_private;
> +	unsigned int i;
> +
> +	for (i = 0; i < n; i++) {
> +		switch (ids[i]) {
> +		case 0: values[i] = ioat->enqueue_failed; break;
> +		case 1: values[i] = ioat->enqueued; break;
> +		case 2: values[i] = ioat->started; break;
> +		case 3: values[i] = ioat->completed; break;
> +		default: values[i] = 0; break;
> +		}
> +	}
> +	return n;

Should this return n or  how many actually filled ?

Thanks,
Reshma

Re: [dpdk-dev] [PATCH v3 7/8] raw/ioat: add statistics functions

[Bruce Richardson]

On Mon, Jul 01, 2019 at 11:11:20AM +0100, Pattan, Reshma wrote:
> 
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> > Sent: Thursday, June 27, 2019 11:41 AM
> > To: dev@dpdk.org
> 
> 
> <snip>
> 
> > +static int
> > +ioat_xstats_get(const struct rte_rawdev *dev, const unsigned int ids[],
> > +		uint64_t values[], unsigned int n)
> > +{
> > +	const struct rte_ioat_rawdev *ioat = dev->dev_private;
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < n; i++) {
> > +		switch (ids[i]) {
> > +		case 0: values[i] = ioat->enqueue_failed; break;
> > +		case 1: values[i] = ioat->enqueued; break;
> > +		case 2: values[i] = ioat->started; break;
> > +		case 3: values[i] = ioat->completed; break;
> > +		default: values[i] = 0; break;
> > +		}
> > +	}
> > +	return n;
> 
> Should this return n or  how many actually filled ?
> 

They should both be "n", since we never break out of the loop early.

/Bruce

[dpdk-dev] [PATCH v3 8/8] raw/ioat: add local API to perform copies

[Bruce Richardson]

Add local APIs to trigger data copies, and retrieve handle values once
those copies are completed. Included are unit tests to validate the data
is copies correctly.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---

V3: updated to use descriptor format in rte_ioat_spec.h
V2: test cases moved to self-test routine
---
 doc/guides/rawdevs/ioat_rawdev.rst  | 100 ++++++++++++++++
 drivers/raw/ioat/Makefile           |   1 +
 drivers/raw/ioat/ioat_rawdev_test.c | 161 +++++++++++++++++++++++++-
 drivers/raw/ioat/meson.build        |   2 +-
 drivers/raw/ioat/rte_ioat_rawdev.h  | 169 +++++++++++++++++++++++++++-
 5 files changed, 428 insertions(+), 5 deletions(-)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index b908f31e0..22bb8a22c 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -150,6 +150,106 @@ The following code shows how the device is configured in
 Once configured, the device can then be made ready for use by calling the
 ``rte_rawdev_start()`` API.
 
+Performing Data Copies
+~~~~~~~~~~~~~~~~~~~~~~~
+
+To perform data copies using IOAT rawdev devices, the functions
+``rte_ioat_enqueue_copy()`` and ``rte_ioat_do_copies()`` should be used.
+Once copies have been completed, the completion will be reported back when
+the application calls ``rte_ioat_completed_copies()``.
+
+The ``rte_ioat_enqueue_copy()`` function enqueues a single copy to the
+device ring for copying at a later point. The parameters to that function
+include the physical addresses of both the source and destination buffers,
+as well as two "handles" to be returned to the user when the copy is
+completed. These handles can be arbitrary values, but two are provided so
+that the library can track handles for both source and destination on
+behalf of the user, e.g. virtual addresses for the buffers, or mbuf
+pointers if packet data is being copied.
+
+While the ``rte_ioat_enqueue_copy()`` function enqueues a copy operation on
+the device ring, the copy will not actually be performed until after the
+application calls the ``rte_ioat_do_copies()`` function. This function
+informs the device hardware of the elements enqueued on the ring, and the
+device will begin to process them. It is expected that, for efficiency
+reasons, a burst of operations will be enqueued to the device via multiple
+enqueue calls between calls to the ``rte_ioat_do_copies()`` function.
+
+The following code from ``test_ioat_rawdev.c`` demonstrates how to enqueue
+a burst of copies to the device and start the hardware processing of them:
+
+.. code-block:: C
+
+        struct rte_mbuf *srcs[32], *dsts[32];
+        unsigned int j;
+
+        for (i = 0; i < RTE_DIM(srcs); i++) {
+                char *src_data;
+
+                srcs[i] = rte_pktmbuf_alloc(pool);
+                dsts[i] = rte_pktmbuf_alloc(pool);
+                srcs[i]->data_len = srcs[i]->pkt_len = length;
+                dsts[i]->data_len = dsts[i]->pkt_len = length;
+                src_data = rte_pktmbuf_mtod(srcs[i], char *);
+
+                for (j = 0; j < length; j++)
+                        src_data[j] = rand() & 0xFF;
+
+                if (rte_ioat_enqueue_copy(dev_id,
+                                srcs[i]->buf_iova + srcs[i]->data_off,
+                                dsts[i]->buf_iova + dsts[i]->data_off,
+                                length,
+                                (uintptr_t)srcs[i],
+                                (uintptr_t)dsts[i],
+                                0 /* nofence */) != 1) {
+                        printf("Error with rte_ioat_enqueue_copy for buffer %u\n",
+                                        i);
+                        return -1;
+                }
+        }
+        rte_ioat_do_copies(dev_id);
+
+To retrieve information about completed copies, the API
+``rte_ioat_completed_copies()`` should be used. This API will return to the
+application a set of completion handles passed in when the relevant copies
+were enqueued.
+
+The following code from ``test_ioat_rawdev.c`` shows the test code
+retrieving information about the completed copies and validating the data
+is correct before freeing the data buffers using the returned handles:
+
+.. code-block:: C
+
+        if (rte_ioat_completed_copies(dev_id, 64, (void *)completed_src,
+                        (void *)completed_dst) != RTE_DIM(srcs)) {
+                printf("Error with rte_ioat_completed_copies\n");
+                return -1;
+        }
+        for (i = 0; i < RTE_DIM(srcs); i++) {
+                char *src_data, *dst_data;
+
+                if (completed_src[i] != srcs[i]) {
+                        printf("Error with source pointer %u\n", i);
+                        return -1;
+                }
+                if (completed_dst[i] != dsts[i]) {
+                        printf("Error with dest pointer %u\n", i);
+                        return -1;
+                }
+
+                src_data = rte_pktmbuf_mtod(srcs[i], char *);
+                dst_data = rte_pktmbuf_mtod(dsts[i], char *);
+                for (j = 0; j < length; j++)
+                        if (src_data[j] != dst_data[j]) {
+                                printf("Error with copy of packet %u, byte %u\n",
+                                                i, j);
+                                return -1;
+                        }
+                rte_pktmbuf_free(srcs[i]);
+                rte_pktmbuf_free(dsts[i]);
+        }
+
+
 Querying Device Statistics
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
index b1af9c666..32f079845 100644
--- a/drivers/raw/ioat/Makefile
+++ b/drivers/raw/ioat/Makefile
@@ -12,6 +12,7 @@ CFLAGS += $(WERROR_FLAGS)
 
 LDLIBS += -lrte_eal -lrte_rawdev
 LDLIBS += -lrte_pci -lrte_bus_pci
+LDLIBS += -lrte_mbuf -lrte_mempool
 
 # library version
 LIBABIVER := 1
diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c
index 5098d71ae..8e57cfd7c 100644
--- a/drivers/raw/ioat/ioat_rawdev_test.c
+++ b/drivers/raw/ioat/ioat_rawdev_test.c
@@ -2,12 +2,139 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <unistd.h>
 #include <inttypes.h>
+#include <rte_mbuf.h>
 #include "rte_rawdev.h"
 #include "rte_ioat_rawdev.h"
 
 int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */
 
+static struct rte_mempool *pool;
+
+static int
+test_enqueue_copies(int dev_id)
+{
+	const unsigned int length = 1024;
+	unsigned int i;
+
+	do {
+		struct rte_mbuf *src, *dst;
+		char *src_data, *dst_data;
+		struct rte_mbuf *completed[2] = {0};
+
+		/* test doing a single copy */
+		src = rte_pktmbuf_alloc(pool);
+		dst = rte_pktmbuf_alloc(pool);
+		src->data_len = src->pkt_len = length;
+		dst->data_len = dst->pkt_len = length;
+		src_data = rte_pktmbuf_mtod(src, char *);
+		dst_data = rte_pktmbuf_mtod(dst, char *);
+
+		for (i = 0; i < length; i++)
+			src_data[i] = rand() & 0xFF;
+
+		if (rte_ioat_enqueue_copy(dev_id,
+				src->buf_iova + src->data_off,
+				dst->buf_iova + dst->data_off,
+				length,
+				(uintptr_t)src,
+				(uintptr_t)dst,
+				0 /* no fence */) != 1) {
+			printf("Error with rte_ioat_enqueue_copy\n");
+			return -1;
+		}
+		rte_ioat_do_copies(dev_id);
+		usleep(10);
+
+		if (rte_ioat_completed_copies(dev_id, 1, (void *)&completed[0],
+				(void *)&completed[1]) != 1) {
+			printf("Error with rte_ioat_completed_copies\n");
+			return -1;
+		}
+		if (completed[0] != src || completed[1] != dst) {
+			printf("Error with completions: got (%p, %p), not (%p,%p)\n",
+					completed[0], completed[1], src, dst);
+			return -1;
+		}
+
+		for (i = 0; i < length; i++)
+			if (dst_data[i] != src_data[i]) {
+				printf("Data mismatch at char %u\n", i);
+				return -1;
+			}
+		rte_pktmbuf_free(src);
+		rte_pktmbuf_free(dst);
+	} while (0);
+
+	/* test doing multiple copies */
+	do {
+		struct rte_mbuf *srcs[32], *dsts[32];
+		struct rte_mbuf *completed_src[64];
+		struct rte_mbuf *completed_dst[64];
+		unsigned int j;
+
+		for (i = 0; i < RTE_DIM(srcs); i++) {
+			char *src_data;
+
+			srcs[i] = rte_pktmbuf_alloc(pool);
+			dsts[i] = rte_pktmbuf_alloc(pool);
+			srcs[i]->data_len = srcs[i]->pkt_len = length;
+			dsts[i]->data_len = dsts[i]->pkt_len = length;
+			src_data = rte_pktmbuf_mtod(srcs[i], char *);
+
+			for (j = 0; j < length; j++)
+				src_data[j] = rand() & 0xFF;
+
+			if (rte_ioat_enqueue_copy(dev_id,
+					srcs[i]->buf_iova + srcs[i]->data_off,
+					dsts[i]->buf_iova + dsts[i]->data_off,
+					length,
+					(uintptr_t)srcs[i],
+					(uintptr_t)dsts[i],
+					0 /* nofence */) != 1) {
+				printf("Error with rte_ioat_enqueue_copy for buffer %u\n",
+						i);
+				return -1;
+			}
+		}
+		rte_ioat_do_copies(dev_id);
+		usleep(100);
+
+		if (rte_ioat_completed_copies(dev_id, 64, (void *)completed_src,
+				(void *)completed_dst) != RTE_DIM(srcs)) {
+			printf("Error with rte_ioat_completed_copies\n");
+			return -1;
+		}
+		for (i = 0; i < RTE_DIM(srcs); i++) {
+			char *src_data, *dst_data;
+
+			if (completed_src[i] != srcs[i]) {
+				printf("Error with source pointer %u\n", i);
+				return -1;
+			}
+			if (completed_dst[i] != dsts[i]) {
+				printf("Error with dest pointer %u\n", i);
+				return -1;
+			}
+
+			src_data = rte_pktmbuf_mtod(srcs[i], char *);
+			dst_data = rte_pktmbuf_mtod(dsts[i], char *);
+			for (j = 0; j < length; j++)
+				if (src_data[j] != dst_data[j]) {
+					printf("Error with copy of packet %u, byte %u\n",
+							i, j);
+					return -1;
+				}
+			rte_pktmbuf_free(srcs[i]);
+			rte_pktmbuf_free(dsts[i]);
+		}
+
+	} while (0);
+
+	return 0;
+}
+
 int
 ioat_rawdev_test(uint16_t dev_id)
 {
@@ -44,6 +171,17 @@ ioat_rawdev_test(uint16_t dev_id)
 		return -1;
 	}
 
+	pool = rte_pktmbuf_pool_create("TEST_IOAT_POOL",
+			256, /* n == num elements */
+			32,  /* cache size */
+			0,   /* priv size */
+			2048, /* data room size */
+			info.socket_id);
+	if (pool == NULL) {
+		printf("Error with mempool creation\n");
+		return -1;
+	}
+
 	/* allocate memory for xstats names and values */
 	nb_xstats = rte_rawdev_xstats_names_get(dev_id, NULL, 0);
 
@@ -68,13 +206,30 @@ ioat_rawdev_test(uint16_t dev_id)
 		return -1;
 	}
 
-	rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
-	for (i = 0; i < nb_xstats; i++)
-		printf("%s: %"PRIu64"   ", snames[i].name, stats[i]);
+	/* run the test cases */
+	for (i = 0; i < 100; i++) {
+		unsigned int j;
+
+		if (test_enqueue_copies(dev_id) != 0)
+			goto err;
+
+		rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
+		for (j = 0; j < nb_xstats; j++)
+			printf("%s: %"PRIu64"   ", snames[j].name, stats[j]);
+		printf("\r");
+	}
 	printf("\n");
 
+	rte_mempool_free(pool);
 	free(snames);
 	free(stats);
 	free(ids);
 	return 0;
+
+err:
+	rte_mempool_free(pool);
+	free(snames);
+	free(stats);
+	free(ids);
+	return -1;
 }
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
index 40fff6654..247ff88bf 100644
--- a/drivers/raw/ioat/meson.build
+++ b/drivers/raw/ioat/meson.build
@@ -4,7 +4,7 @@
 build = dpdk_conf.has('RTE_ARCH_X86')
 sources = files('ioat_rawdev.c',
 		'ioat_rawdev_test.c')
-deps += ['rawdev', 'bus_pci']
+deps += ['rawdev', 'bus_pci', 'mbuf']
 
 install_headers('rte_ioat_rawdev.h',
 		'rte_ioat_spec.h')
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index aa534f9cf..5aeecb26a 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -15,7 +15,9 @@
  */
 
 #include <x86intrin.h>
+#include <rte_atomic.h>
 #include <rte_memory.h>
+#include <rte_prefetch.h>
 #include <rte_ioat_spec.h>
 
 /** Name of the device driver */
@@ -50,6 +52,10 @@ struct rte_ioat_rawdev {
 	struct rte_ioat_generic_hw_desc *desc_ring;
 	__m128i *hdls; /* completion handles for returning to user */
 
+
+	unsigned short next_read;
+	unsigned short next_write;
+
 	/* some statistics for tracking, if added/changed update xstats fns*/
 	uint64_t enqueue_failed __rte_cache_aligned;
 	uint64_t enqueued;
@@ -60,4 +66,165 @@ struct rte_ioat_rawdev {
 	volatile uint64_t status __rte_cache_aligned;
 };
 
-#endif
+/**
+ * Enqueue a copy operation onto the ioat device
+ *
+ * This queues up a copy operation to be performed by hardware, but does not
+ * trigger hardware to begin that operation.
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ * @param src
+ *   The physical address of the source buffer
+ * @param dst
+ *   The physical address of the destination buffer
+ * @param length
+ *   The length of the data to be copied
+ * @param src_hdl
+ *   An opaque handle for the source data, to be returned when this operation
+ *   has been completed and the user polls for the completion details
+ * @param dst_hdl
+ *   An opaque handle for the destination data, to be returned when this
+ *   operation has been completed and the user polls for the completion details
+ * @param fence
+ *   A flag parameter indicating that hardware should not begin to perform any
+ *   subsequently enqueued copy operations until after this operation has
+ *   completed
+ * @return
+ *   Number of operations enqueued, either 0 or 1
+ */
+static inline int
+rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst,
+		unsigned int length, uintptr_t src_hdl, uintptr_t dst_hdl,
+		int fence)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	unsigned short read = ioat->next_read;
+	unsigned short write = ioat->next_write;
+	unsigned short mask = ioat->ring_size - 1;
+	unsigned short space = mask + read - write;
+	struct rte_ioat_generic_hw_desc *desc;
+
+	if (space == 0) {
+		ioat->enqueue_failed++;
+		return 0;
+	}
+
+	ioat->next_write = write + 1;
+	write &= mask;
+
+	desc = &ioat->desc_ring[write];
+	desc->size = length;
+	/* set descriptor write-back every 16th descriptor */
+	desc->u.control_raw = (uint32_t)((!!fence << 4) | (!(write & 0xF)) << 3);
+	desc->src_addr = src;
+	desc->dest_addr = dst;
+
+	ioat->hdls[write] = _mm_set_epi64((__m64)((uint64_t)dst_hdl),
+			(__m64)((uint64_t)src_hdl));
+	rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]);
+
+	ioat->enqueued++;
+	return 1;
+}
+
+/**
+ * Trigger hardware to begin performing enqueued copy operations
+ *
+ * This API is used to write the "doorbell" to the hardware to trigger it
+ * to begin the copy operations previously enqueued by rte_ioat_enqueue_copy()
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ */
+static inline void
+rte_ioat_do_copies(int dev_id)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	ioat->desc_ring[(ioat->next_write - 1) & (ioat->ring_size - 1)].u
+			.control.completion_update = 1;
+	rte_compiler_barrier();
+	ioat->regs->dmacount = ioat->next_write;
+	ioat->started = ioat->enqueued;
+}
+
+/**
+ * @internal
+ * Returns the index of the last completed operation.
+ */
+static inline int
+rte_ioat_get_last_completed(struct rte_ioat_rawdev *ioat, int *error)
+{
+	uint64_t status = ioat->status;
+
+	/* lower 3 bits indicate "transfer status" : active, idle, halted.
+	 * We can ignore bit 0.
+	 */
+	*error = status & (RTE_IOAT_CHANSTS_SUSPENDED | RTE_IOAT_CHANSTS_ARMED);
+	return (status - ioat->ring_addr) >> 6;
+}
+
+/**
+ * Returns details of copy operations that have been completed
+ *
+ * Returns to the caller the user-provided "handles" for the copy operations
+ * which have been completed by the hardware, and not already returned by
+ * a previous call to this API.
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ * @param max_copies
+ *   The number of entries which can fit in the src_hdls and dst_hdls
+ *   arrays, i.e. max number of completed operations to report
+ * @param src_hdls
+ *   Array to hold the source handle parameters of the completed copies
+ * @param dst_hdls
+ *   Array to hold the destination handle parameters of the completed copies
+ * @return
+ *   -1 on error, with rte_errno set appropriately.
+ *   Otherwise number of completed operations i.e. number of entries written
+ *   to the src_hdls and dst_hdls array parameters.
+ */
+static inline int
+rte_ioat_completed_copies(int dev_id, uint8_t max_copies,
+		uintptr_t *src_hdls, uintptr_t *dst_hdls)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	unsigned short mask = (ioat->ring_size - 1);
+	unsigned short read = ioat->next_read;
+	unsigned short end_read, count;
+	int error;
+	int i = 0;
+
+	end_read = (rte_ioat_get_last_completed(ioat, &error) + 1) & mask;
+	count = (end_read - (read & mask)) & mask;
+
+	if (error) {
+		rte_errno = EIO;
+		return -1;
+	}
+
+	if (count > max_copies)
+		count = max_copies;
+
+	for (; i < count - 1; i += 2, read += 2) {
+		__m128i hdls0 = _mm_load_si128(&ioat->hdls[read & mask]);
+		__m128i hdls1 = _mm_load_si128(&ioat->hdls[(read + 1) & mask]);
+
+		_mm_storeu_si128((void *)&src_hdls[i],
+				_mm_unpacklo_epi64(hdls0, hdls1));
+		_mm_storeu_si128((void *)&dst_hdls[i],
+				_mm_unpackhi_epi64(hdls0, hdls1));
+	}
+	for (; i < count; i++, read++) {
+		uintptr_t *hdls = (void *)&ioat->hdls[read & mask];
+		src_hdls[i] = hdls[0];
+		dst_hdls[i] = hdls[1];
+	}
+
+	ioat->next_read = read;
+	ioat->completed += count;
+	return count;
+}
+
+#endif /* _RTE_IOAT_RAWDEV_H_ */
-- 
2.21.0

Re: [dpdk-dev] [PATCH v3 8/8] raw/ioat: add local API to perform copies

[Burakov, Anatoly]

On 27-Jun-19 11:40 AM, Bruce Richardson wrote:
> Add local APIs to trigger data copies, and retrieve handle values once
> those copies are completed. Included are unit tests to validate the data
> is copies correctly.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> 
> ---
> 
> V3: updated to use descriptor format in rte_ioat_spec.h
> V2: test cases moved to self-test routine
> ---
>   doc/guides/rawdevs/ioat_rawdev.rst  | 100 ++++++++++++++++
>   drivers/raw/ioat/Makefile           |   1 +
>   drivers/raw/ioat/ioat_rawdev_test.c | 161 +++++++++++++++++++++++++-
>   drivers/raw/ioat/meson.build        |   2 +-
>   drivers/raw/ioat/rte_ioat_rawdev.h  | 169 +++++++++++++++++++++++++++-
>   5 files changed, 428 insertions(+), 5 deletions(-)
> 
> diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
> index b908f31e0..22bb8a22c 100644
> --- a/doc/guides/rawdevs/ioat_rawdev.rst
> +++ b/doc/guides/rawdevs/ioat_rawdev.rst
> @@ -150,6 +150,106 @@ The following code shows how the device is configured in
>   Once configured, the device can then be made ready for use by calling the
>   ``rte_rawdev_start()`` API.
>   
> +Performing Data Copies
> +~~~~~~~~~~~~~~~~~~~~~~~
> +
> +To perform data copies using IOAT rawdev devices, the functions
> +``rte_ioat_enqueue_copy()`` and ``rte_ioat_do_copies()`` should be used.
> +Once copies have been completed, the completion will be reported back when
> +the application calls ``rte_ioat_completed_copies()``.
> +
> +The ``rte_ioat_enqueue_copy()`` function enqueues a single copy to the
> +device ring for copying at a later point. The parameters to that function
> +include the physical addresses of both the source and destination buffers,

Physical or IOVA?

> +as well as two "handles" to be returned to the user when the copy is
> +completed. These handles can be arbitrary values, but two are provided so
> +that the library can track handles for both source and destination on
> +behalf of the user, e.g. virtual addresses for the buffers, or mbuf
> +pointers if packet data is being copied.
> +
> +While the ``rte_ioat_enqueue_copy()`` function enqueues a copy operation on
> +the device ring, the copy will not actually be performed until after the

<snip>

> +	rte_mempool_free(pool);
>   	free(snames);
>   	free(stats);
>   	free(ids);
>   	return 0;
> +
> +err:
> +	rte_mempool_free(pool);
> +	free(snames);
> +	free(stats);
> +	free(ids);
> +	return -1;

The goto is added a commit too late :D

Otherwise, LGTM

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3 0/8] raw/ioat: driver for Intel QuickData Technology

[Van Haaren, Harry]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> Sent: Thursday, June 27, 2019 11:41 AM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; jerinj@marvell.com; Richardson, Bruce
> <bruce.richardson@intel.com>
> Subject: [dpdk-dev] [PATCH v3 0/8] raw/ioat: driver for Intel QuickData
> Technology
> 
> This patch series adds support for the Intel QuickData Technology
> device, part of the Intel I/O Acceleration Technology (Intel I/OAT). It
> is a raw device for allowing hardware DMA i.e. data copies in hardware.
> 
> Performing the copies in hardware can provide performance improvements
> for applications where the average copy size is reasonably large, e.g.
> 1k packets. For smaller packets, e.g. 64-256 bytes, offloading the copy
> may reduce performance due to the overhead of using hardware.

Applied (release notes conflict - but that's par for the course close to integration)
Build each patch with meson (git rebase -x is awesome :)

Ran unit tests:
$ ioat_rawdev_probe(): Init 0000:00:04.0 on NUMA node 0
$ failed_enqueues: 0   successful_enqueues: 3300   copies_started: 3300   copies_completed: 3300

All seems fine to me;

Series-Tested-by: Harry van Haaren <harry.van.haaren@intel.com>

[dpdk-dev] [PATCH v4 0/9] raw/ioat: driver for Intel QuickData Technology

[Bruce Richardson]

This patch series adds support for the Intel QuickData Technology
device, part of the Intel I/O Acceleration Technology (Intel I/OAT). It
is a raw device for allowing hardware DMA i.e. data copies in hardware.

Performing the copies in hardware can provide performance improvements
for applications where the average copy size is reasonably large, e.g.
1k packets. For smaller packets, e.g. 64-256 bytes, offloading the copy
may reduce performance due to the overhead of using hardware.

V4:
 * changed memory management to use contiguous memzones instead of
   malloc memory
 * added missing device ids for BDX platforms
 * other misc cleanup following review (see individual patches logs)

V3:
 * removed DPDK-specific structure for the descriptor format and reused
   the structure in the imported file rte_ioat_spec.h

V2:
* moved tests to rawdev selftest function
* some checkpatch and other small cleanups
* added extra documentation details on supported hardware
* aligned the changes to dpdk-devbind with the changes in the NTB set
  for consistency

Bruce Richardson (9):
  rawdev: allow devices to skip extra memory allocation
  raw/ioat: add initial support for ioat rawdev driver
  usertools/dpdk-devbind.py: add support for IOAT devices
  raw/ioat: add register definition file
  raw/ioat: create device on probe and destroy on release
  raw/ioat: add device info function
  raw/ioat: add configure, start and stop functions
  raw/ioat: add statistics functions
  raw/ioat: add local API to perform copies

 MAINTAINERS                                 |   5 +
 app/test/test_rawdev.c                      |  20 ++
 config/common_armv8a_linux                  |   1 +
 config/common_base                          |   5 +
 config/defconfig_arm-armv7a-linuxapp-gcc    |   1 +
 config/defconfig_ppc_64-power8-linuxapp-gcc |   1 +
 doc/guides/rawdevs/index.rst                |   1 +
 doc/guides/rawdevs/ioat_rawdev.rst          | 265 +++++++++++++++
 doc/guides/rel_notes/release_19_08.rst      |  11 +
 drivers/raw/Makefile                        |   1 +
 drivers/raw/ioat/Makefile                   |  31 ++
 drivers/raw/ioat/ioat_rawdev.c              | 356 ++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev_test.c         | 235 +++++++++++++
 drivers/raw/ioat/meson.build                |  10 +
 drivers/raw/ioat/rte_ioat_rawdev.h          | 234 +++++++++++++
 drivers/raw/ioat/rte_ioat_spec.h            | 301 +++++++++++++++++
 drivers/raw/ioat/rte_pmd_ioat_version.map   |   4 +
 drivers/raw/meson.build                     |   4 +-
 lib/librte_rawdev/rte_rawdev.c              |  11 +-
 mk/rte.app.mk                               |   1 +
 usertools/dpdk-devbind.py                   |  10 +
 21 files changed, 1502 insertions(+), 6 deletions(-)
 create mode 100644 doc/guides/rawdevs/ioat_rawdev.rst
 create mode 100644 drivers/raw/ioat/Makefile
 create mode 100644 drivers/raw/ioat/ioat_rawdev.c
 create mode 100644 drivers/raw/ioat/ioat_rawdev_test.c
 create mode 100644 drivers/raw/ioat/meson.build
 create mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h
 create mode 100644 drivers/raw/ioat/rte_ioat_spec.h
 create mode 100644 drivers/raw/ioat/rte_pmd_ioat_version.map

-- 
2.21.0

[dpdk-dev] [PATCH v4 1/9] rawdev: allow devices to skip extra memory allocation

[Bruce Richardson]

Some device drivers want to allocate their own private memory, and should
be allowed to do so. Therefore skip memory allocation and associated error
checks if zero-length private memory is requested.

While adjusting the code for new indent level, fix incorrect error
message.

Cc: Shreyansh Jain <shreyansh.jain@nxp.com>
Cc: Hemant Agrawal <hemant.agrawal@nxp.com>
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 lib/librte_rawdev/rte_rawdev.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/lib/librte_rawdev/rte_rawdev.c b/lib/librte_rawdev/rte_rawdev.c
index 15de2d413..b6f1e1c77 100644
--- a/lib/librte_rawdev/rte_rawdev.c
+++ b/lib/librte_rawdev/rte_rawdev.c
@@ -496,16 +496,17 @@ rte_rawdev_pmd_allocate(const char *name, size_t dev_priv_size, int socket_id)
 
 	rawdev = &rte_rawdevs[dev_id];
 
-	rawdev->dev_private = rte_zmalloc_socket("rawdev private",
+	if (dev_priv_size > 0) {
+		rawdev->dev_private = rte_zmalloc_socket("rawdev private",
 				     dev_priv_size,
 				     RTE_CACHE_LINE_SIZE,
 				     socket_id);
-	if (!rawdev->dev_private) {
-		RTE_RDEV_ERR("Unable to allocate memory to Skeleton dev");
-		return NULL;
+		if (!rawdev->dev_private) {
+			RTE_RDEV_ERR("Unable to allocate memory for rawdev");
+			return NULL;
+		}
 	}
 
-
 	rawdev->dev_id = dev_id;
 	rawdev->socket_id = socket_id;
 	rawdev->started = 0;
-- 
2.21.0

Re: [dpdk-dev] [PATCH v4 1/9] rawdev: allow devices to skip extra memory allocation

[Hemant Agrawal]

Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com

Re: [dpdk-dev] [PATCH v4 1/9] rawdev: allow devices to skip extra memory allocation

[Shreyansh Jain]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Monday, July 1, 2019 9:26 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; jerinj@marvell.com; jiayu.hu@intel.com; Bruce
> Richardson <bruce.richardson@intel.com>; Shreyansh Jain
> <shreyansh.jain@nxp.com>; Hemant Agrawal <hemant.agrawal@nxp.com>
> Subject: [PATCH v4 1/9] rawdev: allow devices to skip extra memory
> allocation
> 
> Some device drivers want to allocate their own private memory, and
> should
> be allowed to do so. Therefore skip memory allocation and associated
> error
> checks if zero-length private memory is requested.

Agree with this - rawdev was intended for flexibility and this (allowing them their own memory) is definitely better way ahead. Thanks for proposing.
But, I think the kind of caveat should also be added to the header declaring this function:

Probably something like this:

--->8--- lib/librte_rawdev/rte_rawdev_pmd.h ---
/**
 * Allocates a new rawdev slot for an raw device and returns the pointer 
 * to that slot for the driver to use.        
 *
 * @param name 
 *   Unique identifier name for each device 
 * @param dev_private_size 
 *   Private data allocated within rte_rawdev object.
 *   <b>Set to 0 to disable internal allocation and allow for self-allocation</b>
 * @param socket_id 
 *   Socket to allocate resources on. 
 * @return 
 *   - Slot in the rte_dev_devices array for a new device; 
 */
struct rte_rawdev *
rte_rawdev_pmd_allocate(const char *name, size_t dev_private_size, 
                        int socket_id);
--->8---

> 
> While adjusting the code for new indent level, fix incorrect error
> message.
> 
> Cc: Shreyansh Jain <shreyansh.jain@nxp.com>
> Cc: Hemant Agrawal <hemant.agrawal@nxp.com>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---

If you can update the header, please use my ACK in next version.

Acked-by: Shreyansh Jain <shreyansh.jain@nxp.com>

[...]

Re: [dpdk-dev] [PATCH v4 1/9] rawdev: allow devices to skip extra memory allocation

[Bruce Richardson]

On Tue, Jul 02, 2019 at 11:43:58AM +0000, Shreyansh Jain wrote:
> > -----Original Message-----
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Monday, July 1, 2019 9:26 PM
> > To: dev@dpdk.org
> > Cc: thomas@monjalon.net; jerinj@marvell.com; jiayu.hu@intel.com; Bruce
> > Richardson <bruce.richardson@intel.com>; Shreyansh Jain
> > <shreyansh.jain@nxp.com>; Hemant Agrawal <hemant.agrawal@nxp.com>
> > Subject: [PATCH v4 1/9] rawdev: allow devices to skip extra memory
> > allocation
> > 
> > Some device drivers want to allocate their own private memory, and
> > should
> > be allowed to do so. Therefore skip memory allocation and associated
> > error
> > checks if zero-length private memory is requested.
> 
> Agree with this - rawdev was intended for flexibility and this (allowing them their own memory) is definitely better way ahead. Thanks for proposing.
> But, I think the kind of caveat should also be added to the header declaring this function:
> 
> Probably something like this:
> 
> --->8--- lib/librte_rawdev/rte_rawdev_pmd.h ---
> /**
>  * Allocates a new rawdev slot for an raw device and returns the pointer 
>  * to that slot for the driver to use.        
>  *
>  * @param name 
>  *   Unique identifier name for each device 
>  * @param dev_private_size 
>  *   Private data allocated within rte_rawdev object.
>  *   <b>Set to 0 to disable internal allocation and allow for self-allocation</b>
>  * @param socket_id 
>  *   Socket to allocate resources on. 
>  * @return 
>  *   - Slot in the rte_dev_devices array for a new device; 
>  */
> struct rte_rawdev *
> rte_rawdev_pmd_allocate(const char *name, size_t dev_private_size, 
>                         int socket_id);
> --->8---
> 
> > 
> > While adjusting the code for new indent level, fix incorrect error
> > message.
> > 
> > Cc: Shreyansh Jain <shreyansh.jain@nxp.com>
> > Cc: Hemant Agrawal <hemant.agrawal@nxp.com>
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
> 
> If you can update the header, please use my ACK in next version.
> 
> Acked-by: Shreyansh Jain <shreyansh.jain@nxp.com>
> 
Thanks Shreyansh, will do.

/Bruce

[dpdk-dev] [PATCH v4 2/9] raw/ioat: add initial support for ioat rawdev driver

[Bruce Richardson]

Add stubs for ioat rawdev driver support in DPDK, specifically:

  * makefile and meson build hooks
  * initial public header file
  * rawdev main C file, with probe and release functions
  * release note update announcing the driver
  * initial documentation for the new section in the rawdev doc
  * unit test stubs for device unit tests

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>

---
V4: add in missing device IDs
V3: remove unneeded whitespace change
V2: don't create a new file for unit testing, add to existing rawdev test
    file, and place test cases in the driver selftest routine (added later
    in set)
    Add new section in document about identifying hardware using lspci
---
 MAINTAINERS                                 |   5 +
 app/test/test_rawdev.c                      |   8 ++
 config/common_armv8a_linux                  |   1 +
 config/common_base                          |   5 +
 config/defconfig_arm-armv7a-linuxapp-gcc    |   1 +
 config/defconfig_ppc_64-power8-linuxapp-gcc |   1 +
 doc/guides/rawdevs/index.rst                |   1 +
 doc/guides/rawdevs/ioat_rawdev.rst          |  63 +++++++++++
 doc/guides/rel_notes/release_19_08.rst      |  11 ++
 drivers/raw/Makefile                        |   1 +
 drivers/raw/ioat/Makefile                   |  28 +++++
 drivers/raw/ioat/ioat_rawdev.c              | 113 ++++++++++++++++++++
 drivers/raw/ioat/meson.build                |   8 ++
 drivers/raw/ioat/rte_ioat_rawdev.h          |  24 +++++
 drivers/raw/ioat/rte_pmd_ioat_version.map   |   4 +
 drivers/raw/meson.build                     |   4 +-
 mk/rte.app.mk                               |   1 +
 17 files changed, 278 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/rawdevs/ioat_rawdev.rst
 create mode 100644 drivers/raw/ioat/Makefile
 create mode 100644 drivers/raw/ioat/ioat_rawdev.c
 create mode 100644 drivers/raw/ioat/meson.build
 create mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h
 create mode 100644 drivers/raw/ioat/rte_pmd_ioat_version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index bbec1982c..2cfafe000 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1074,6 +1074,11 @@ M: Tianfei zhang <tianfei.zhang@intel.com>
 F: drivers/raw/ifpga_rawdev/
 F: doc/guides/rawdevs/ifpga_rawdev.rst
 
+IOAT Rawdev
+M: Bruce Richardson <bruce.richardson@intel.com>
+F: drivers/raw/ioat/
+F: doc/guides/rawdevs/ioat_rawdev.rst
+
 NXP DPAA2 QDMA
 M: Nipun Gupta <nipun.gupta@nxp.com>
 F: drivers/raw/dpaa2_qdma/
diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
index 043a38a13..88549fb61 100644
--- a/app/test/test_rawdev.c
+++ b/app/test/test_rawdev.c
@@ -25,3 +25,11 @@ test_rawdev_selftest_skeleton(void)
 }
 
 REGISTER_TEST_COMMAND(rawdev_autotest, test_rawdev_selftest_skeleton);
+
+static int
+test_rawdev_selftest_ioat(void)
+{
+	return TEST_SKIPPED;
+}
+
+REGISTER_TEST_COMMAND(ioat_rawdev_autotest, test_rawdev_selftest_ioat);
diff --git a/config/common_armv8a_linux b/config/common_armv8a_linux
index 72091de1c..481712ebc 100644
--- a/config/common_armv8a_linux
+++ b/config/common_armv8a_linux
@@ -34,5 +34,6 @@ CONFIG_RTE_ARCH_ARM64_MEMCPY=n
 CONFIG_RTE_LIBRTE_FM10K_PMD=n
 CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
 
 CONFIG_RTE_SCHED_VECTOR=n
diff --git a/config/common_base b/config/common_base
index fa1ae249a..e6b830923 100644
--- a/config/common_base
+++ b/config/common_base
@@ -747,6 +747,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n
 #
 CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
 
+#
+# Compile PMD for Intel IOAT raw device
+#
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=y
+
 #
 # Compile librte_ring
 #
diff --git a/config/defconfig_arm-armv7a-linuxapp-gcc b/config/defconfig_arm-armv7a-linuxapp-gcc
index c9509b274..ee158ef9d 100644
--- a/config/defconfig_arm-armv7a-linuxapp-gcc
+++ b/config/defconfig_arm-armv7a-linuxapp-gcc
@@ -54,3 +54,4 @@ CONFIG_RTE_LIBRTE_QEDE_PMD=n
 CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
 CONFIG_RTE_LIBRTE_NFP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
diff --git a/config/defconfig_ppc_64-power8-linuxapp-gcc b/config/defconfig_ppc_64-power8-linuxapp-gcc
index 7e248b755..9f3670ec0 100644
--- a/config/defconfig_ppc_64-power8-linuxapp-gcc
+++ b/config/defconfig_ppc_64-power8-linuxapp-gcc
@@ -56,3 +56,4 @@ CONFIG_RTE_LIBRTE_ENIC_PMD=n
 CONFIG_RTE_LIBRTE_FM10K_PMD=n
 CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
diff --git a/doc/guides/rawdevs/index.rst b/doc/guides/rawdevs/index.rst
index 7c3bd9586..0a21989e4 100644
--- a/doc/guides/rawdevs/index.rst
+++ b/doc/guides/rawdevs/index.rst
@@ -14,3 +14,4 @@ application through rawdev API.
     dpaa2_cmdif
     dpaa2_qdma
     ifpga_rawdev
+    ioat_rawdev
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
new file mode 100644
index 000000000..0c612e73a
--- /dev/null
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -0,0 +1,63 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+.. include:: <isonum.txt>
+
+IOAT Rawdev Driver for Intel\ |reg| QuickData Technology
+======================================================================
+
+The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg|
+QuickData Technology, part of Intel\ |reg| I/O Acceleration Technology
+`(Intel I/OAT)
+<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_.
+This PMD, when used on supported hardware, allows data copies, for example,
+cloning packet data, to be accelerated by that hardware rather than having to
+be done by software, freeing up CPU cycles for other tasks.
+
+Hardware Requirements
+----------------------
+
+On Linux, the presence of an Intel\ |reg| QuickData Technology hardware can
+be detected by checking the output of the ``lspci`` command, where the
+hardware will be often listed as "Crystal Beach DMA" or "CBDMA". For
+example, on a system with Intel\ |reg| Xeon\ |reg| CPU E5-2699 v4 @2.20GHz,
+lspci shows:
+
+.. code-block:: console
+
+  # lspci | grep DMA
+  00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01)
+  00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01)
+  00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01)
+  00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01)
+  00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01)
+  00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01)
+  00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01)
+  00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01)
+
+On a system with Intel\ |reg| Xeon\ |reg| Gold 6154 CPU @ 3.00GHz, lspci
+shows:
+
+.. code-block:: console
+
+  # lspci | grep DMA
+  00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+
+
+Compilation
+------------
+
+For builds done with ``make``, the driver compilation is enabled by the
+``CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV`` build configuration option. This is
+enabled by default in builds for x86 platforms, and disabled in other
+configurations.
+
+For builds using ``meson`` and ``ninja``, the driver will be built when the
+target platform is x86-based.
diff --git a/doc/guides/rel_notes/release_19_08.rst b/doc/guides/rel_notes/release_19_08.rst
index ad0ac4333..aa77348db 100644
--- a/doc/guides/rel_notes/release_19_08.rst
+++ b/doc/guides/rel_notes/release_19_08.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright 2019 The DPDK contributors
 
+.. include:: <isonum.txt>
+
 DPDK Release 19.08
 ==================
 
@@ -118,6 +120,15 @@ New Features
   Added telemetry mode to l3fwd-power application to report
   application level busyness, empty and full polls of rte_eth_rx_burst().
 
+* **Added Intel QuickData Technology PMD**
+
+  The PMD for Intel\ |reg|  QuickData Technology, part of
+  Intel\ |reg|  I/O Acceleration Technology `(Intel I/OAT)
+  <https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_,
+  allows data copies to be done by hardware instead
+  of via software, reducing cycles spent copying large blocks of data in
+  applications.
+
 
 Removed Items
 -------------
diff --git a/drivers/raw/Makefile b/drivers/raw/Makefile
index 8e29b4a56..c1b85c8c7 100644
--- a/drivers/raw/Makefile
+++ b/drivers/raw/Makefile
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev
+DIRS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat
 
 include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
new file mode 100644
index 000000000..7726e310a
--- /dev/null
+++ b/drivers/raw/ioat/Makefile
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_pmd_ioat_rawdev.a
+
+# build flags
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+
+LDLIBS += -lrte_eal -lrte_rawdev
+LDLIBS += -lrte_pci -lrte_bus_pci
+
+# library version
+LIBABIVER := 1
+
+# versioning export map
+EXPORT_MAP := rte_pmd_ioat_version.map
+
+# library source files
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
+
+# export include files
+SYMLINK-y-include += rte_ioat_rawdev.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
new file mode 100644
index 000000000..07f6d8dfa
--- /dev/null
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -0,0 +1,113 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include <rte_bus_pci.h>
+#include <rte_rawdev_pmd.h>
+
+#include "rte_ioat_rawdev.h"
+
+/* Dynamic log type identifier */
+int ioat_pmd_logtype;
+
+static struct rte_pci_driver ioat_pmd_drv;
+
+#define IOAT_VENDOR_ID		0x8086
+#define IOAT_DEVICE_ID_SKX	0x2021
+#define IOAT_DEVICE_ID_BDX0	0x6f20
+#define IOAT_DEVICE_ID_BDX1	0x6f21
+#define IOAT_DEVICE_ID_BDX2	0x6f22
+#define IOAT_DEVICE_ID_BDX3	0x6f23
+#define IOAT_DEVICE_ID_BDX4	0x6f24
+#define IOAT_DEVICE_ID_BDX5	0x6f25
+#define IOAT_DEVICE_ID_BDX6	0x6f26
+#define IOAT_DEVICE_ID_BDX7	0x6f27
+#define IOAT_DEVICE_ID_BDXE	0x6f2E
+#define IOAT_DEVICE_ID_BDXF	0x6f2F
+
+#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \
+	ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args)
+
+#define IOAT_PMD_DEBUG(fmt, args...)  IOAT_PMD_LOG(DEBUG, fmt, ## args)
+#define IOAT_PMD_INFO(fmt, args...)   IOAT_PMD_LOG(INFO, fmt, ## args)
+#define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
+#define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
+
+static int
+ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
+{
+	RTE_SET_USED(name);
+	RTE_SET_USED(dev);
+	return 0;
+}
+
+static int
+ioat_rawdev_destroy(const char *name)
+{
+	RTE_SET_USED(name);
+	return 0;
+}
+
+static int
+ioat_rawdev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev)
+{
+	char name[32];
+	int ret = 0;
+
+
+	rte_pci_device_name(&dev->addr, name, sizeof(name));
+	IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node);
+
+	dev->device.driver = &drv->driver;
+	ret = ioat_rawdev_create(name, dev);
+	return ret;
+}
+
+static int
+ioat_rawdev_remove(struct rte_pci_device *dev)
+{
+	char name[32];
+	int ret;
+
+	rte_pci_device_name(&dev->addr, name, sizeof(name));
+
+	IOAT_PMD_INFO("Closing %s on NUMA node %d",
+			name, dev->device.numa_node);
+
+	ret = ioat_rawdev_destroy(name);
+	return ret;
+}
+
+static const struct rte_pci_id pci_id_ioat_map[] = {
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_SKX) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX0) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX1) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX2) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX3) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX4) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX5) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX6) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX7) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXE) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXF) },
+	{ .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver ioat_pmd_drv = {
+	.id_table = pci_id_ioat_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+		     RTE_PCI_DRV_IOVA_AS_VA,
+	.probe = ioat_rawdev_probe,
+	.remove = ioat_rawdev_remove,
+};
+
+RTE_PMD_REGISTER_PCI(IOAT_PMD_RAWDEV_NAME, ioat_pmd_drv);
+RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_RAWDEV_NAME, pci_id_ioat_map);
+RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_RAWDEV_NAME, "* igb_uio | uio_pci_generic");
+
+RTE_INIT(ioat_pmd_init_log)
+{
+	ioat_pmd_logtype = rte_log_register(IOAT_PMD_LOG_NAME);
+	if (ioat_pmd_logtype >= 0)
+		rte_log_set_level(ioat_pmd_logtype, RTE_LOG_INFO);
+}
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
new file mode 100644
index 000000000..ba7620a68
--- /dev/null
+++ b/drivers/raw/ioat/meson.build
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2019 Intel Corporation
+
+build = dpdk_conf.has('RTE_ARCH_X86')
+sources = files('ioat_rawdev.c')
+deps += ['rawdev', 'bus_pci']
+
+install_headers('rte_ioat_rawdev.h')
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
new file mode 100644
index 000000000..e77406403
--- /dev/null
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_IOAT_RAWDEV_H_
+#define _RTE_IOAT_RAWDEV_H_
+
+/**
+ * @file rte_ioat_rawdev.h
+ *
+ * Definitions for using the ioat rawdev device driver
+ *
+ * @warning
+ * @b EXPERIMENTAL: these structures and APIs may change without prior notice
+ */
+
+/** Name of the device driver */
+#define IOAT_PMD_RAWDEV_NAME rawdev_ioat
+/** String reported as the device driver name by rte_rawdev_info_get() */
+#define IOAT_PMD_RAWDEV_NAME_STR "rawdev_ioat"
+/** Name used to adjust the log level for this driver */
+#define IOAT_PMD_LOG_NAME "rawdev.ioat"
+
+#endif
diff --git a/drivers/raw/ioat/rte_pmd_ioat_version.map b/drivers/raw/ioat/rte_pmd_ioat_version.map
new file mode 100644
index 000000000..9a61188cd
--- /dev/null
+++ b/drivers/raw/ioat/rte_pmd_ioat_version.map
@@ -0,0 +1,4 @@
+DPDK_19.08 {
+
+	local: *;
+};
diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build
index a61cdccef..2af8a70d4 100644
--- a/drivers/raw/meson.build
+++ b/drivers/raw/meson.build
@@ -1,7 +1,9 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2018 NXP
 
-drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
+drivers = ['dpaa2_cmdif', 'dpaa2_qdma',
+	'ifpga_rawdev', 'ioat',
+	'skeleton_rawdev']
 std_deps = ['rawdev']
 config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
 driver_name_fmt = 'rte_pmd_@0@'
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 81be289a8..2a534796f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -306,6 +306,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV)   += -lrte_pmd_ifpga_rawdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD)       += -lrte_pmd_ipn3ke
 endif # CONFIG_RTE_LIBRTE_IFPGA_BUS
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV)   += -lrte_pmd_ioat_rawdev
 endif # CONFIG_RTE_LIBRTE_RAWDEV
 
 endif # !CONFIG_RTE_BUILD_SHARED_LIBS
-- 
2.21.0

[dpdk-dev] [PATCH v4 3/9] usertools/dpdk-devbind.py: add support for IOAT devices

[Bruce Richardson]

In order to allow binding/unbinding of devices for use by the
ioat_rawdev, we need to update the devbind script to add a new class
of device, and add device ids for the specific HW instances.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>

---

V2: rather than adding a DMA section, add to "misc (rawdev)" section where
    other device types, e.g. ntb can also do.

NOTE: this set largely overlaps with the equivalent changes made by the
patchset adding NTB support [1]. Since it's unclear which set will be added
first, this set is based off the latest head.

[1] http://patches.dpdk.org/patch/55127/
---
 doc/guides/rawdevs/ioat_rawdev.rst | 11 +++++++++++
 usertools/dpdk-devbind.py          | 10 ++++++++++
 2 files changed, 21 insertions(+)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 0c612e73a..1a4b0e03e 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -61,3 +61,14 @@ configurations.
 
 For builds using ``meson`` and ``ninja``, the driver will be built when the
 target platform is x86-based.
+
+Device Setup
+-------------
+
+The Intel\ |reg| QuickData Technology HW devices will need to be bound to a
+user-space IO driver for use. The script ``dpdk-devbind.py`` script
+included with DPDK can be used to view the state of the devices and to bind
+them to a suitable DPDK-supported kernel driver. When querying the status
+of the devices, they will appear under the category of "Misc (rawdev)
+devices", i.e. the command ``dpdk-devbind.py --status-dev misc`` can be
+used to see the state of those devices alone.
diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py
index 9e79f0d28..5c1cd3548 100755
--- a/usertools/dpdk-devbind.py
+++ b/usertools/dpdk-devbind.py
@@ -36,11 +36,17 @@
 octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc',
               'SVendor': None, 'SDevice': None}
 
+intel_ioat_bdw = {'Class': '08', 'Vendor': '8086', 'Device': '6f20,6f21,6f22,6f23,6f24,6f25,6f26,6f27,6f2e,6f2f',
+              'SVendor': None, 'SDevice': None}
+intel_ioat_skx = {'Class': '08', 'Vendor': '8086', 'Device': '2021',
+              'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 crypto_devices = [encryption_class, intel_processor_class]
 eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso]
 mempool_devices = [cavium_fpa, octeontx2_npa]
 compress_devices = [cavium_zip]
+misc_devices = [intel_ioat_bdw, intel_ioat_skx]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
@@ -595,6 +601,8 @@ def show_status():
     if status_dev == "compress" or status_dev == "all":
         show_device_status(compress_devices , "Compress")
 
+    if status_dev == "misc" or status_dev == "all":
+        show_device_status(misc_devices, "Misc (rawdev)")
 
 def parse_args():
     '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +678,7 @@ def do_arg_actions():
             get_device_details(eventdev_devices)
             get_device_details(mempool_devices)
             get_device_details(compress_devices)
+            get_device_details(misc_devices)
         show_status()
 
 
@@ -690,6 +699,7 @@ def main():
     get_device_details(eventdev_devices)
     get_device_details(mempool_devices)
     get_device_details(compress_devices)
+    get_device_details(misc_devices)
     do_arg_actions()
 
 if __name__ == "__main__":
-- 
2.21.0

[dpdk-dev] [PATCH v4 4/9] raw/ioat: add register definition file

[Bruce Richardson]

Add in the list of registers for the device. File is taken from the SPDK
project:

  https://github.com/spdk/spdk/blob/master/include/spdk/ioat_spec.h

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
---
 drivers/raw/ioat/Makefile        |   1 +
 drivers/raw/ioat/meson.build     |   3 +-
 drivers/raw/ioat/rte_ioat_spec.h | 301 +++++++++++++++++++++++++++++++
 3 files changed, 304 insertions(+), 1 deletion(-)
 create mode 100644 drivers/raw/ioat/rte_ioat_spec.h

diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
index 7726e310a..1e10938f3 100644
--- a/drivers/raw/ioat/Makefile
+++ b/drivers/raw/ioat/Makefile
@@ -24,5 +24,6 @@ SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
 
 # export include files
 SYMLINK-y-include += rte_ioat_rawdev.h
+SYMLINK-y-include += rte_ioat_spec.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
index ba7620a68..ca23e23fc 100644
--- a/drivers/raw/ioat/meson.build
+++ b/drivers/raw/ioat/meson.build
@@ -5,4 +5,5 @@ build = dpdk_conf.has('RTE_ARCH_X86')
 sources = files('ioat_rawdev.c')
 deps += ['rawdev', 'bus_pci']
 
-install_headers('rte_ioat_rawdev.h')
+install_headers('rte_ioat_rawdev.h',
+		'rte_ioat_spec.h')
diff --git a/drivers/raw/ioat/rte_ioat_spec.h b/drivers/raw/ioat/rte_ioat_spec.h
new file mode 100644
index 000000000..305e36ded
--- /dev/null
+++ b/drivers/raw/ioat/rte_ioat_spec.h
@@ -0,0 +1,301 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) Intel Corporation
+ */
+
+/**
+ * \file
+ * I/OAT specification definitions
+ *
+ * Taken from ioat_spec.h from SPDK project, with prefix renames and
+ * other minor changes.
+ */
+
+#ifndef RTE_IOAT_SPEC_H
+#define RTE_IOAT_SPEC_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+#define RTE_IOAT_PCI_CHANERR_INT_OFFSET	0x180
+
+#define RTE_IOAT_INTRCTRL_MASTER_INT_EN	0x01
+
+#define RTE_IOAT_VER_3_0                0x30
+#define RTE_IOAT_VER_3_3                0x33
+
+/* DMA Channel Registers */
+#define RTE_IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK	0xF000
+#define RTE_IOAT_CHANCTRL_COMPL_DCA_EN		0x0200
+#define RTE_IOAT_CHANCTRL_CHANNEL_IN_USE		0x0100
+#define RTE_IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL	0x0020
+#define RTE_IOAT_CHANCTRL_ERR_INT_EN		0x0010
+#define RTE_IOAT_CHANCTRL_ANY_ERR_ABORT_EN		0x0008
+#define RTE_IOAT_CHANCTRL_ERR_COMPLETION_EN		0x0004
+#define RTE_IOAT_CHANCTRL_INT_REARM			0x0001
+
+/* DMA Channel Capabilities */
+#define	RTE_IOAT_DMACAP_PB		(1 << 0)
+#define	RTE_IOAT_DMACAP_DCA		(1 << 4)
+#define	RTE_IOAT_DMACAP_BFILL		(1 << 6)
+#define	RTE_IOAT_DMACAP_XOR		(1 << 8)
+#define	RTE_IOAT_DMACAP_PQ		(1 << 9)
+#define	RTE_IOAT_DMACAP_DMA_DIF	(1 << 10)
+
+struct rte_ioat_registers {
+	uint8_t		chancnt;
+	uint8_t		xfercap;
+	uint8_t		genctrl;
+	uint8_t		intrctrl;
+	uint32_t	attnstatus;
+	uint8_t		cbver;		/* 0x08 */
+	uint8_t		reserved4[0x3]; /* 0x09 */
+	uint16_t	intrdelay;	/* 0x0C */
+	uint16_t	cs_status;	/* 0x0E */
+	uint32_t	dmacapability;	/* 0x10 */
+	uint8_t		reserved5[0x6C]; /* 0x14 */
+	uint16_t	chanctrl;	/* 0x80 */
+	uint8_t		reserved6[0x2];	/* 0x82 */
+	uint8_t		chancmd;	/* 0x84 */
+	uint8_t		reserved3[1];	/* 0x85 */
+	uint16_t	dmacount;	/* 0x86 */
+	uint64_t	chansts;	/* 0x88 */
+	uint64_t	chainaddr;	/* 0x90 */
+	uint64_t	chancmp;	/* 0x98 */
+	uint8_t		reserved2[0x8];	/* 0xA0 */
+	uint32_t	chanerr;	/* 0xA8 */
+	uint32_t	chanerrmask;	/* 0xAC */
+} __attribute__((packed));
+
+#define RTE_IOAT_CHANCMD_RESET			0x20
+#define RTE_IOAT_CHANCMD_SUSPEND		0x04
+
+#define RTE_IOAT_CHANSTS_STATUS		0x7ULL
+#define RTE_IOAT_CHANSTS_ACTIVE		0x0
+#define RTE_IOAT_CHANSTS_IDLE			0x1
+#define RTE_IOAT_CHANSTS_SUSPENDED		0x2
+#define RTE_IOAT_CHANSTS_HALTED		0x3
+#define RTE_IOAT_CHANSTS_ARMED			0x4
+
+#define RTE_IOAT_CHANSTS_UNAFFILIATED_ERROR	0x8ULL
+#define RTE_IOAT_CHANSTS_SOFT_ERROR		0x10ULL
+
+#define RTE_IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK	(~0x3FULL)
+
+#define RTE_IOAT_CHANCMP_ALIGN			8	/* CHANCMP address must be 64-bit aligned */
+
+struct rte_ioat_generic_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t reserved2: 1;
+			uint32_t src_page_break: 1;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t op_specific[4];
+};
+
+struct rte_ioat_dma_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t null: 1;
+			uint32_t src_page_break: 1;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+#define RTE_IOAT_OP_COPY 0x00
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t reserved;
+	uint64_t reserved2;
+	uint64_t user1;
+	uint64_t user2;
+};
+
+struct rte_ioat_fill_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t reserved: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t reserved2: 2;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t reserved3: 15;
+#define RTE_IOAT_OP_FILL 0x01
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_data;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t reserved;
+	uint64_t next_dest_addr;
+	uint64_t user1;
+	uint64_t user2;
+};
+
+struct rte_ioat_xor_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_count: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+#define RTE_IOAT_OP_XOR 0x87
+#define RTE_IOAT_OP_XOR_VAL 0x88
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t src_addr3;
+	uint64_t src_addr4;
+	uint64_t src_addr5;
+};
+
+struct rte_ioat_xor_ext_hw_desc {
+	uint64_t src_addr6;
+	uint64_t src_addr7;
+	uint64_t src_addr8;
+	uint64_t next;
+	uint64_t reserved[4];
+};
+
+struct rte_ioat_pq_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_count: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t p_disable: 1;
+			uint32_t q_disable: 1;
+			uint32_t reserved: 11;
+#define RTE_IOAT_OP_PQ 0x89
+#define RTE_IOAT_OP_PQ_VAL 0x8a
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t p_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t src_addr3;
+	uint8_t  coef[8];
+	uint64_t q_addr;
+};
+
+struct rte_ioat_pq_ext_hw_desc {
+	uint64_t src_addr4;
+	uint64_t src_addr5;
+	uint64_t src_addr6;
+	uint64_t next;
+	uint64_t src_addr7;
+	uint64_t src_addr8;
+	uint64_t reserved[2];
+};
+
+struct rte_ioat_pq_update_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_cnt: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t p_disable: 1;
+			uint32_t q_disable: 1;
+			uint32_t reserved: 3;
+			uint32_t coef: 8;
+#define RTE_IOAT_OP_PQ_UP 0x8b
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t p_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t p_src;
+	uint64_t q_src;
+	uint64_t q_addr;
+};
+
+struct rte_ioat_raw_hw_desc {
+	uint64_t field[8];
+};
+
+union rte_ioat_hw_desc {
+	struct rte_ioat_raw_hw_desc raw;
+	struct rte_ioat_generic_hw_desc generic;
+	struct rte_ioat_dma_hw_desc dma;
+	struct rte_ioat_fill_hw_desc fill;
+	struct rte_ioat_xor_hw_desc xor_desc;
+	struct rte_ioat_xor_ext_hw_desc xor_ext;
+	struct rte_ioat_pq_hw_desc pq;
+	struct rte_ioat_pq_ext_hw_desc pq_ext;
+	struct rte_ioat_pq_update_hw_desc pq_update;
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_IOAT_SPEC_H */
-- 
2.21.0

[dpdk-dev] [PATCH v4 5/9] raw/ioat: create device on probe and destroy on release

[Bruce Richardson]

Add the create/destroy driver functions so that we can actually allocate
a rawdev and destroy it when done. No rawdev API functions are actually
implemented at this point.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
---
V4:
* changed error code for pmd allocation to ENOMEM instead of EINVAL
* Use contiguous memory for device structure allocation
---
 doc/guides/rawdevs/ioat_rawdev.rst |  11 +++
 drivers/raw/ioat/ioat_rawdev.c     | 112 ++++++++++++++++++++++++++++-
 drivers/raw/ioat/rte_ioat_rawdev.h |  24 +++++++
 3 files changed, 144 insertions(+), 3 deletions(-)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 1a4b0e03e..4b7fe8a8f 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -72,3 +72,14 @@ them to a suitable DPDK-supported kernel driver. When querying the status
 of the devices, they will appear under the category of "Misc (rawdev)
 devices", i.e. the command ``dpdk-devbind.py --status-dev misc`` can be
 used to see the state of those devices alone.
+
+Device Probing and Initialization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once bound to a suitable kernel device driver, the HW devices will be found
+as part of the PCI scan done at application initialization time. No vdev
+parameters need to be passed to create or initialize the device.
+
+Once probed successfully, the device will appear as a ``rawdev``, that is a
+"raw device type" inside DPDK, and can be accessed using APIs from the
+``rte_rawdev`` library.
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index 07f6d8dfa..86b5b2a77 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -2,6 +2,7 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <rte_cycles.h>
 #include <rte_bus_pci.h>
 #include <rte_rawdev_pmd.h>
 
@@ -36,15 +37,120 @@ static struct rte_pci_driver ioat_pmd_drv;
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
-	RTE_SET_USED(name);
-	RTE_SET_USED(dev);
+	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+	};
+
+	struct rte_rawdev *rawdev = NULL;
+	struct rte_ioat_rawdev *ioat = NULL;
+	const struct rte_memzone *mz = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	int ret = 0;
+	int retry = 0;
+
+	if (!name) {
+		IOAT_PMD_ERR("Invalid name of the device!");
+		ret = -EINVAL;
+		goto cleanup;
+	}
+
+	/* Allocate device structure */
+	rawdev = rte_rawdev_pmd_allocate(name, sizeof(struct rte_ioat_rawdev),
+					 dev->device.numa_node);
+	if (rawdev == NULL) {
+		IOAT_PMD_ERR("Unable to allocate raw device");
+		ret = -ENOMEM;
+		goto cleanup;
+	}
+
+	snprintf(mz_name, sizeof(mz_name), "rawdev%u_private", rawdev->dev_id);
+	mz = rte_memzone_reserve(mz_name, sizeof(struct rte_ioat_rawdev),
+			dev->device.numa_node, RTE_MEMZONE_IOVA_CONTIG);
+	if (mz == NULL) {
+		IOAT_PMD_ERR("Unable to reserve memzone for private data\n");
+		ret = -ENOMEM;
+		goto cleanup;
+	}
+
+	rawdev->dev_private = mz->addr;
+	rawdev->dev_ops = &ioat_rawdev_ops;
+	rawdev->device = &dev->device;
+	rawdev->driver_name = dev->device.driver->name;
+
+	ioat = rawdev->dev_private;
+	ioat->rawdev = rawdev;
+	ioat->mz = mz;
+	ioat->regs = dev->mem_resource[0].addr;
+	ioat->ring_size = 0;
+	ioat->desc_ring = NULL;
+	ioat->status_addr = ioat->mz->iova +
+			offsetof(struct rte_ioat_rawdev, status);
+
+	/* do device initialization - reset and set error behaviour */
+	if (ioat->regs->chancnt != 1)
+		IOAT_PMD_ERR("%s: Channel count == %d\n", __func__,
+				ioat->regs->chancnt);
+
+	if (ioat->regs->chanctrl & 0x100) { /* locked by someone else */
+		IOAT_PMD_WARN("%s: Channel appears locked\n", __func__);
+		ioat->regs->chanctrl = 0;
+	}
+
+	ioat->regs->chancmd = RTE_IOAT_CHANCMD_SUSPEND;
+	rte_delay_ms(1);
+	ioat->regs->chancmd = RTE_IOAT_CHANCMD_RESET;
+	rte_delay_ms(1);
+	while (ioat->regs->chancmd & RTE_IOAT_CHANCMD_RESET) {
+		ioat->regs->chainaddr = 0;
+		rte_delay_ms(1);
+		if (++retry >= 200) {
+			IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=0x%"PRIx8", CHANSTS=0x%"PRIx64", CHANERR=0x%"PRIx32"\n",
+					__func__,
+					ioat->regs->chancmd,
+					ioat->regs->chansts,
+					ioat->regs->chanerr);
+			ret = -EIO;
+		}
+	}
+	ioat->regs->chanctrl = RTE_IOAT_CHANCTRL_ANY_ERR_ABORT_EN |
+			RTE_IOAT_CHANCTRL_ERR_COMPLETION_EN;
+
 	return 0;
+
+cleanup:
+	if (rawdev)
+		rte_rawdev_pmd_release(rawdev);
+
+	return ret;
 }
 
 static int
 ioat_rawdev_destroy(const char *name)
 {
-	RTE_SET_USED(name);
+	int ret;
+	struct rte_rawdev *rdev;
+
+	if (!name) {
+		IOAT_PMD_ERR("Invalid device name");
+		return -EINVAL;
+	}
+
+	rdev = rte_rawdev_pmd_get_named_dev(name);
+	if (!rdev) {
+		IOAT_PMD_ERR("Invalid device name (%s)", name);
+		return -EINVAL;
+	}
+
+	if (rdev->dev_private != NULL) {
+		struct rte_ioat_rawdev *ioat = rdev->dev_private;
+		rdev->dev_private = NULL;
+		rte_memzone_free(ioat->mz);
+	}
+
+	/* rte_rawdev_close is called by pmd_release */
+	ret = rte_rawdev_pmd_release(rdev);
+	if (ret)
+		IOAT_PMD_DEBUG("Device cleanup failed");
+
 	return 0;
 }
 
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index e77406403..f83d26ef0 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -14,6 +14,10 @@
  * @b EXPERIMENTAL: these structures and APIs may change without prior notice
  */
 
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_ioat_spec.h>
+
 /** Name of the device driver */
 #define IOAT_PMD_RAWDEV_NAME rawdev_ioat
 /** String reported as the device driver name by rte_rawdev_info_get() */
@@ -21,4 +25,24 @@
 /** Name used to adjust the log level for this driver */
 #define IOAT_PMD_LOG_NAME "rawdev.ioat"
 
+/**
+ * @internal
+ * Structure representing a device instance
+ */
+struct rte_ioat_rawdev {
+	struct rte_rawdev *rawdev;
+	const struct rte_memzone *mz;
+	const struct rte_memzone *desc_mz;
+
+	volatile struct rte_ioat_registers *regs;
+	phys_addr_t status_addr;
+	phys_addr_t ring_addr;
+
+	unsigned short ring_size;
+	struct rte_ioat_generic_hw_desc *desc_ring;
+
+	/* to report completions, the device will write status back here */
+	volatile uint64_t status __rte_cache_aligned;
+};
+
 #endif
-- 
2.21.0

Re: [dpdk-dev] [PATCH v4 5/9] raw/ioat: create device on probe and destroy on release

[Burakov, Anatoly]

On 01-Jul-19 4:55 PM, Bruce Richardson wrote:
> Add the create/destroy driver functions so that we can actually allocate
> a rawdev and destroy it when done. No rawdev API functions are actually
> implemented at this point.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> ---
> V4:
> * changed error code for pmd allocation to ENOMEM instead of EINVAL
> * Use contiguous memory for device structure allocation
> ---

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

[dpdk-dev] [PATCH v4 6/9] raw/ioat: add device info function

[Bruce Richardson]

Add in the "info_get" function to the driver, to allow us to query the
device. This allows us to have the unit test pick up the presence of
supported hardware or not.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
---
V4: Use TEST_SUCCESS in place of 0, when returning from test case.
    Minor documentation updates.

V2: Test case is placed in driver self-test routine
---
 app/test/test_rawdev.c             | 11 ++++++++++
 doc/guides/rawdevs/ioat_rawdev.rst | 34 ++++++++++++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev.c     | 11 ++++++++++
 drivers/raw/ioat/rte_ioat_rawdev.h | 11 ++++++++++
 4 files changed, 67 insertions(+)

diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
index 88549fb61..623117af9 100644
--- a/app/test/test_rawdev.c
+++ b/app/test/test_rawdev.c
@@ -29,6 +29,17 @@ REGISTER_TEST_COMMAND(rawdev_autotest, test_rawdev_selftest_skeleton);
 static int
 test_rawdev_selftest_ioat(void)
 {
+	const int count = rte_rawdev_count();
+	int i;
+
+	for (i = 0; i < count; i++) {
+		struct rte_rawdev_info info = { .dev_private = NULL };
+		if (rte_rawdev_info_get(i, &info) == 0 &&
+				strstr(info.driver_name, "ioat") != NULL)
+			return TEST_SUCCESS;
+	}
+
+	printf("No IOAT rawdev found, skipping tests\n");
 	return TEST_SKIPPED;
 }
 
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 4b7fe8a8f..efed9c64c 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -83,3 +83,37 @@ parameters need to be passed to create or initialize the device.
 Once probed successfully, the device will appear as a ``rawdev``, that is a
 "raw device type" inside DPDK, and can be accessed using APIs from the
 ``rte_rawdev`` library.
+
+Using IOAT Rawdev Devices
+--------------------------
+
+To use the devices from an application, the rawdev API can be used, along
+with definitions taken from the device-specific header file
+``rte_ioat_rawdev.h``. This header is needed to get the definition of
+structure parameters used by some of the rawdev APIs for IOAT rawdev
+devices, as well as providing key functions for using the device for memory
+copies.
+
+Getting Device Information
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Basic information about each rawdev device can be queried using the
+``rte_rawdev_info_get()`` API. For most applications, this API will be
+needed to verify that the rawdev in question is of the expected type. For
+example, the following code snippet can be used to identify an IOAT
+rawdev device for use by an application:
+
+.. code-block:: C
+
+        for (i = 0; i < count && !found; i++) {
+                struct rte_rawdev_info info = { .dev_private = NULL };
+                found = (rte_rawdev_info_get(i, &info) == 0 &&
+                                strcmp(info.driver_name,
+                                                IOAT_PMD_RAWDEV_NAME_STR) == 0);
+        }
+
+When calling the ``rte_rawdev_info_get()`` API for an IOAT rawdev device,
+the ``dev_private`` field in the ``rte_rawdev_info`` struct should either
+be NULL, or else be set to point to a structure of type
+``rte_ioat_rawdev_config``, in which case the size of the configured device
+input ring will be returned in that structure.
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index 86b5b2a77..2bfe2544d 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -34,10 +34,21 @@ static struct rte_pci_driver ioat_pmd_drv;
 #define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
 #define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
 
+static void
+ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
+{
+	struct rte_ioat_rawdev_config *cfg = dev_info;
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+
+	if (cfg != NULL)
+		cfg->ring_size = ioat->ring_size;
+}
+
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
 	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+			.dev_info_get = ioat_dev_info_get,
 	};
 
 	struct rte_rawdev *rawdev = NULL;
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index f83d26ef0..1d0dbdd2e 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -25,6 +25,17 @@
 /** Name used to adjust the log level for this driver */
 #define IOAT_PMD_LOG_NAME "rawdev.ioat"
 
+/**
+ * Configuration structure for an ioat rawdev instance
+ *
+ * This structure is to be passed as the ".dev_private" parameter when
+ * calling the rte_rawdev_get_info() and rte_rawdev_configure() APIs on
+ * an ioat rawdev instance.
+ */
+struct rte_ioat_rawdev_config {
+	unsigned short ring_size;
+};
+
 /**
  * @internal
  * Structure representing a device instance
-- 
2.21.0

Re: [dpdk-dev] [PATCH v4 6/9] raw/ioat: add device info function

[Hu, Jiayu]

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Monday, July 1, 2019 11:56 PM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; jerinj@marvell.com; Hu, Jiayu
> <jiayu.hu@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> Van Haaren, Harry <harry.van.haaren@intel.com>
> Subject: [PATCH v4 6/9] raw/ioat: add device info function
> 
> Add in the "info_get" function to the driver, to allow us to query the
> device. This allows us to have the unit test pick up the presence of
> supported hardware or not.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> ---
> V4: Use TEST_SUCCESS in place of 0, when returning from test case.
>     Minor documentation updates.
> 
> V2: Test case is placed in driver self-test routine
> ---
>  app/test/test_rawdev.c             | 11 ++++++++++
>  doc/guides/rawdevs/ioat_rawdev.rst | 34
> ++++++++++++++++++++++++++++++
>  drivers/raw/ioat/ioat_rawdev.c     | 11 ++++++++++
>  drivers/raw/ioat/rte_ioat_rawdev.h | 11 ++++++++++
>  4 files changed, 67 insertions(+)
> 
> diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
> index 86b5b2a77..2bfe2544d 100644
> --- a/drivers/raw/ioat/ioat_rawdev.c
> +++ b/drivers/raw/ioat/ioat_rawdev.c
> @@ -34,10 +34,21 @@ static struct rte_pci_driver ioat_pmd_drv;
>  #define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
>  #define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ##
> args)
> 
> +static void
> +ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
> +{
> +	struct rte_ioat_rawdev_config *cfg = dev_info;
> +	struct rte_ioat_rawdev *ioat = dev->dev_private;
> +
> +	if (cfg != NULL)
> +		cfg->ring_size = ioat->ring_size;
> +}

The CBDMA user may want to know the maximum copy length that the device
supports and the NUMA location of the device. So can ioat_dev_info_get()
return more device information?

Thanks,
Jiayu

Re: [dpdk-dev] [PATCH v4 6/9] raw/ioat: add device info function

[Bruce Richardson]

On Tue, Jul 02, 2019 at 03:33:25AM +0100, Hu, Jiayu wrote:
> 
> > -----Original Message-----
> > From: Richardson, Bruce
> > Sent: Monday, July 1, 2019 11:56 PM
> > To: dev@dpdk.org
> > Cc: thomas@monjalon.net; jerinj@marvell.com; Hu, Jiayu
> > <jiayu.hu@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> > Van Haaren, Harry <harry.van.haaren@intel.com>
> > Subject: [PATCH v4 6/9] raw/ioat: add device info function
> > 
> > Add in the "info_get" function to the driver, to allow us to query the
> > device. This allows us to have the unit test pick up the presence of
> > supported hardware or not.
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> > ---
> > V4: Use TEST_SUCCESS in place of 0, when returning from test case.
> >     Minor documentation updates.
> > 
> > V2: Test case is placed in driver self-test routine
> > ---
> >  app/test/test_rawdev.c             | 11 ++++++++++
> >  doc/guides/rawdevs/ioat_rawdev.rst | 34
> > ++++++++++++++++++++++++++++++
> >  drivers/raw/ioat/ioat_rawdev.c     | 11 ++++++++++
> >  drivers/raw/ioat/rte_ioat_rawdev.h | 11 ++++++++++
> >  4 files changed, 67 insertions(+)
> > 
> > diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
> > index 86b5b2a77..2bfe2544d 100644
> > --- a/drivers/raw/ioat/ioat_rawdev.c
> > +++ b/drivers/raw/ioat/ioat_rawdev.c
> > @@ -34,10 +34,21 @@ static struct rte_pci_driver ioat_pmd_drv;
> >  #define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
> >  #define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ##
> > args)
> > 
> > +static void
> > +ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
> > +{
> > +	struct rte_ioat_rawdev_config *cfg = dev_info;
> > +	struct rte_ioat_rawdev *ioat = dev->dev_private;
> > +
> > +	if (cfg != NULL)
> > +		cfg->ring_size = ioat->ring_size;
> > +}
> 
> The CBDMA user may want to know the maximum copy length that the device
> supports and the NUMA location of the device. So can ioat_dev_info_get()
> return more device information?
> 
The NUMA node id is already part of the rawdev info structure returned to
the user.

As for maximum copy length, yes, that could be added.  However, since it's
actually a characteristic of the hardware, it could be just a define inside
the header file rather than needing to be returned per instance. Unlike
with ethdevs and cryptodevs, rawdevs always have a device-specific
dependency so using defines from the header should work fine, and not
everything needs to come via dev_info.

Regards,
/Bruce

Re: [dpdk-dev] [PATCH v4 6/9] raw/ioat: add device info function

[Burakov, Anatoly]

On 01-Jul-19 4:55 PM, Bruce Richardson wrote:
> Add in the "info_get" function to the driver, to allow us to query the
> device. This allows us to have the unit test pick up the presence of
> supported hardware or not.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> ---
> V4: Use TEST_SUCCESS in place of 0, when returning from test case.
>      Minor documentation updates.
> 
> V2: Test case is placed in driver self-test routine
> ---

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

[dpdk-dev] [PATCH v4 7/9] raw/ioat: add configure, start and stop functions

[Bruce Richardson]

Allow initializing a driver instance. Include selftest to validate these
functions.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
---

V4: Guarantee correct SUCCESS/FAILURE return values from test function
    Use memzone rather than malloc for ring allocation so we can guarantee
    contiguous memory.
V3: don't add a new descriptor format struct, reuse from rte_ioat_spec.h
V2: test cases placed in self-test routine
---
 app/test/test_rawdev.c              |  3 +-
 doc/guides/rawdevs/ioat_rawdev.rst  | 32 +++++++++++
 drivers/raw/ioat/Makefile           |  1 +
 drivers/raw/ioat/ioat_rawdev.c      | 82 +++++++++++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev_test.c | 41 +++++++++++++++
 drivers/raw/ioat/meson.build        |  3 +-
 drivers/raw/ioat/rte_ioat_rawdev.h  |  2 +
 7 files changed, 162 insertions(+), 2 deletions(-)
 create mode 100644 drivers/raw/ioat/ioat_rawdev_test.c

diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
index 623117af9..524a9d5f3 100644
--- a/app/test/test_rawdev.c
+++ b/app/test/test_rawdev.c
@@ -36,7 +36,8 @@ test_rawdev_selftest_ioat(void)
 		struct rte_rawdev_info info = { .dev_private = NULL };
 		if (rte_rawdev_info_get(i, &info) == 0 &&
 				strstr(info.driver_name, "ioat") != NULL)
-			return TEST_SUCCESS;
+			return rte_rawdev_selftest(i) == 0 ?
+					TEST_SUCCESS : TEST_FAILED;
 	}
 
 	printf("No IOAT rawdev found, skipping tests\n");
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index efed9c64c..a0594d2cb 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -117,3 +117,35 @@ the ``dev_private`` field in the ``rte_rawdev_info`` struct should either
 be NULL, or else be set to point to a structure of type
 ``rte_ioat_rawdev_config``, in which case the size of the configured device
 input ring will be returned in that structure.
+
+Device Configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+Configuring an IOAT rawdev device is done using the
+``rte_rawdev_configure()`` API, which takes the same structure parameters
+as the, previously referenced, ``rte_rawdev_info_get()`` API. The main
+difference is that, because the parameter is used as input rather than
+output, the ``dev_private`` structure element cannot be NULL, and must
+point to a valid ``rte_ioat_rawdev_config`` structure, containing the ring
+size to be used by the device. The ring size must be a power of two,
+between 64 and 4096.
+
+The following code shows how the device is configured in
+``test_ioat_rawdev.c``:
+
+.. code-block:: C
+
+   #define IOAT_TEST_RINGSIZE 512
+        struct rte_ioat_rawdev_config p = { .ring_size = -1 };
+        struct rte_rawdev_info info = { .dev_private = &p };
+
+        /* ... */
+
+        p.ring_size = IOAT_TEST_RINGSIZE;
+        if (rte_rawdev_configure(dev_id, &info) != 0) {
+                printf("Error with rte_rawdev_configure()\n");
+                return -1;
+        }
+
+Once configured, the device can then be made ready for use by calling the
+``rte_rawdev_start()`` API.
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
index 1e10938f3..b1af9c666 100644
--- a/drivers/raw/ioat/Makefile
+++ b/drivers/raw/ioat/Makefile
@@ -21,6 +21,7 @@ EXPORT_MAP := rte_pmd_ioat_version.map
 
 # library source files
 SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev_test.c
 
 # export include files
 SYMLINK-y-include += rte_ioat_rawdev.h
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index 2bfe2544d..7eeb42f89 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -34,6 +34,81 @@ static struct rte_pci_driver ioat_pmd_drv;
 #define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
 #define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
 
+#define DESC_SZ sizeof(struct rte_ioat_generic_hw_desc)
+#define COMPLETION_SZ sizeof(__m128i)
+
+static int
+ioat_dev_configure(const struct rte_rawdev *dev, rte_rawdev_obj_t config)
+{
+	struct rte_ioat_rawdev_config *params = config;
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	unsigned short i;
+
+	if (dev->started)
+		return -EBUSY;
+
+	if (params == NULL)
+		return -EINVAL;
+
+	if (params->ring_size > 4096 || params->ring_size < 64 ||
+			!rte_is_power_of_2(params->ring_size))
+		return -EINVAL;
+
+	ioat->ring_size = params->ring_size;
+	if (ioat->desc_ring != NULL) {
+		rte_memzone_free(ioat->desc_mz);
+		ioat->desc_ring = NULL;
+		ioat->desc_mz = NULL;
+	}
+
+	/* allocate one block of memory for both descriptors
+	 * and completion handles.
+	 */
+	snprintf(mz_name, sizeof(mz_name), "rawdev%u_desc_ring", dev->dev_id);
+	ioat->desc_mz = rte_memzone_reserve(mz_name,
+			(DESC_SZ + COMPLETION_SZ) * ioat->ring_size,
+			dev->device->numa_node, RTE_MEMZONE_IOVA_CONTIG);
+	if (ioat->desc_mz == NULL)
+		return -ENOMEM;
+	ioat->desc_ring = ioat->desc_mz->addr;
+	ioat->hdls = (void *)&ioat->desc_ring[ioat->ring_size];
+
+	ioat->ring_addr = ioat->desc_mz->iova;
+
+	/* configure descriptor ring - each one points to next */
+	for (i = 0; i < ioat->ring_size; i++) {
+		ioat->desc_ring[i].next = ioat->ring_addr +
+				(((i + 1) % ioat->ring_size) * DESC_SZ);
+	}
+
+	return 0;
+}
+
+static int
+ioat_dev_start(struct rte_rawdev *dev)
+{
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+
+	if (ioat->ring_size == 0 || ioat->desc_ring == NULL)
+		return -EBUSY;
+
+	/* inform hardware of where the descriptor ring is */
+	ioat->regs->chainaddr = ioat->ring_addr;
+	/* inform hardware of where to write the status/completions */
+	ioat->regs->chancmp = ioat->status_addr;
+
+	/* prime the status register to be set to the last element */
+	ioat->status =  ioat->ring_addr + ((ioat->ring_size - 1) * DESC_SZ);
+	return 0;
+}
+
+static void
+ioat_dev_stop(struct rte_rawdev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
 static void
 ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
 {
@@ -44,11 +119,17 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
 		cfg->ring_size = ioat->ring_size;
 }
 
+int ioat_rawdev_test(uint16_t dev_id);
+
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
 	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+			.dev_configure = ioat_dev_configure,
+			.dev_start = ioat_dev_start,
+			.dev_stop = ioat_dev_stop,
 			.dev_info_get = ioat_dev_info_get,
+			.dev_selftest = ioat_rawdev_test,
 	};
 
 	struct rte_rawdev *rawdev = NULL;
@@ -154,6 +235,7 @@ ioat_rawdev_destroy(const char *name)
 	if (rdev->dev_private != NULL) {
 		struct rte_ioat_rawdev *ioat = rdev->dev_private;
 		rdev->dev_private = NULL;
+		rte_memzone_free(ioat->desc_mz);
 		rte_memzone_free(ioat->mz);
 	}
 
diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c
new file mode 100644
index 000000000..5375da26c
--- /dev/null
+++ b/drivers/raw/ioat/ioat_rawdev_test.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include "rte_rawdev.h"
+#include "rte_ioat_rawdev.h"
+
+int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */
+
+int
+ioat_rawdev_test(uint16_t dev_id)
+{
+#define IOAT_TEST_RINGSIZE 512
+	struct rte_ioat_rawdev_config p = { .ring_size = -1 };
+	struct rte_rawdev_info info = { .dev_private = &p };
+
+	rte_rawdev_info_get(dev_id, &info);
+	if (p.ring_size != 0) {
+		printf("Error, initial ring size is non-zero (%d)\n",
+				(int)p.ring_size);
+		return -1;
+	}
+
+	p.ring_size = IOAT_TEST_RINGSIZE;
+	if (rte_rawdev_configure(dev_id, &info) != 0) {
+		printf("Error with rte_rawdev_configure()\n");
+		return -1;
+	}
+	rte_rawdev_info_get(dev_id, &info);
+	if (p.ring_size != IOAT_TEST_RINGSIZE) {
+		printf("Error, ring size is not %d (%d)\n",
+				IOAT_TEST_RINGSIZE, (int)p.ring_size);
+		return -1;
+	}
+
+	if (rte_rawdev_start(dev_id) != 0) {
+		printf("Error with rte_rawdev_start()\n");
+		return -1;
+	}
+	return 0;
+}
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
index ca23e23fc..40fff6654 100644
--- a/drivers/raw/ioat/meson.build
+++ b/drivers/raw/ioat/meson.build
@@ -2,7 +2,8 @@
 # Copyright 2019 Intel Corporation
 
 build = dpdk_conf.has('RTE_ARCH_X86')
-sources = files('ioat_rawdev.c')
+sources = files('ioat_rawdev.c',
+		'ioat_rawdev_test.c')
 deps += ['rawdev', 'bus_pci']
 
 install_headers('rte_ioat_rawdev.h',
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index 1d0dbdd2e..f2cf98cd9 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -14,6 +14,7 @@
  * @b EXPERIMENTAL: these structures and APIs may change without prior notice
  */
 
+#include <x86intrin.h>
 #include <rte_memory.h>
 #include <rte_memzone.h>
 #include <rte_ioat_spec.h>
@@ -51,6 +52,7 @@ struct rte_ioat_rawdev {
 
 	unsigned short ring_size;
 	struct rte_ioat_generic_hw_desc *desc_ring;
+	__m128i *hdls; /* completion handles for returning to user */
 
 	/* to report completions, the device will write status back here */
 	volatile uint64_t status __rte_cache_aligned;
-- 
2.21.0

Re: [dpdk-dev] [PATCH v4 7/9] raw/ioat: add configure, start and stop functions

[Burakov, Anatoly]

On 01-Jul-19 4:55 PM, Bruce Richardson wrote:
> Allow initializing a driver instance. Include selftest to validate these
> functions.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> ---
> 
> V4: Guarantee correct SUCCESS/FAILURE return values from test function
>      Use memzone rather than malloc for ring allocation so we can guarantee
>      contiguous memory.
> V3: don't add a new descriptor format struct, reuse from rte_ioat_spec.h
> V2: test cases placed in self-test routine
> ---

<snip>

>   static void
>   ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
>   {
> @@ -44,11 +119,17 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
>   		cfg->ring_size = ioat->ring_size;
>   }
>   
> +int ioat_rawdev_test(uint16_t dev_id);
> +
>   static int
>   ioat_rawdev_create(const char *name, struct rte_pci_device *dev)

...and...

> +
> +int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */
> +

This looks like you're trying to avoid introducing a header file? (to my 
great shame, i don't quite get how would that even work...)

Otherwise,

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v4 7/9] raw/ioat: add configure, start and stop functions

[Bruce Richardson]

On Tue, Jul 02, 2019 at 10:49:42AM +0100, Burakov, Anatoly wrote:
> On 01-Jul-19 4:55 PM, Bruce Richardson wrote:
> > Allow initializing a driver instance. Include selftest to validate these
> > functions.
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> > ---
> > 
> > V4: Guarantee correct SUCCESS/FAILURE return values from test function
> >      Use memzone rather than malloc for ring allocation so we can guarantee
> >      contiguous memory.
> > V3: don't add a new descriptor format struct, reuse from rte_ioat_spec.h
> > V2: test cases placed in self-test routine
> > ---
> 
> <snip>
> 
> >   static void
> >   ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
> >   {
> > @@ -44,11 +119,17 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
> >   		cfg->ring_size = ioat->ring_size;
> >   }
> > +int ioat_rawdev_test(uint16_t dev_id);
> > +
> >   static int
> >   ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
> 
> ...and...
> 
> > +
> > +int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */
> > +
> 
> This looks like you're trying to avoid introducing a header file? (to my
> great shame, i don't quite get how would that even work...)
> 

Yes, yes I am. I fail to see the need to create a whole new header file
just for a single function prototype, which can be easily stuck in the C
file instead. If the scope of internal data or prototypes needed grows
beyond this, then we can create an internal header to store it, but I don't
foresee that happening any time soon, since things are pretty
self-contained apart from these test cases.

> Otherwise,
> 
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> 
> -- 
> Thanks,
> Anatoly

[dpdk-dev] [PATCH v4 8/9] raw/ioat: add statistics functions

[Bruce Richardson]

Add stats functions to track what is happening in the driver, and put
unit tests to check those.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
---
V4: pulled in memory leak fixes from next patch
V2: test cases moved to self-test routine
---
 doc/guides/rawdevs/ioat_rawdev.rst  | 14 +++++++++
 drivers/raw/ioat/ioat_rawdev.c      | 44 ++++++++++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev_test.c | 45 +++++++++++++++++++++++++++++
 drivers/raw/ioat/rte_ioat_rawdev.h  |  6 ++++
 4 files changed, 109 insertions(+)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index a0594d2cb..40210b3f9 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -149,3 +149,17 @@ The following code shows how the device is configured in
 
 Once configured, the device can then be made ready for use by calling the
 ``rte_rawdev_start()`` API.
+
+Querying Device Statistics
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The statistics from the IOAT rawdev device can be got via the xstats
+functions in the ``rte_rawdev`` library, i.e.
+``rte_rawdev_xstats_names_get()``, ``rte_rawdev_xstats_get()`` and
+``rte_rawdev_xstats_by_name_get``. The statistics returned for each device
+instance are:
+
+* ``failed_enqueues``
+* ``successful_enqueues``
+* ``copies_started``
+* ``copies_completed``
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index 7eeb42f89..06639aa69 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -4,6 +4,7 @@
 
 #include <rte_cycles.h>
 #include <rte_bus_pci.h>
+#include <rte_string_fns.h>
 #include <rte_rawdev_pmd.h>
 
 #include "rte_ioat_rawdev.h"
@@ -119,6 +120,47 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
 		cfg->ring_size = ioat->ring_size;
 }
 
+static const char * const xstat_names[] = {
+		"failed_enqueues", "successful_enqueues",
+		"copies_started", "copies_completed"
+};
+
+static int
+ioat_xstats_get(const struct rte_rawdev *dev, const unsigned int ids[],
+		uint64_t values[], unsigned int n)
+{
+	const struct rte_ioat_rawdev *ioat = dev->dev_private;
+	unsigned int i;
+
+	for (i = 0; i < n; i++) {
+		switch (ids[i]) {
+		case 0: values[i] = ioat->enqueue_failed; break;
+		case 1: values[i] = ioat->enqueued; break;
+		case 2: values[i] = ioat->started; break;
+		case 3: values[i] = ioat->completed; break;
+		default: values[i] = 0; break;
+		}
+	}
+	return n;
+}
+
+static int
+ioat_xstats_get_names(const struct rte_rawdev *dev,
+		struct rte_rawdev_xstats_name *names,
+		unsigned int size)
+{
+	unsigned int i;
+
+	RTE_SET_USED(dev);
+	if (size < RTE_DIM(xstat_names))
+		return RTE_DIM(xstat_names);
+
+	for (i = 0; i < RTE_DIM(xstat_names); i++)
+		strlcpy(names[i].name, xstat_names[i], sizeof(names[i]));
+
+	return RTE_DIM(xstat_names);
+}
+
 int ioat_rawdev_test(uint16_t dev_id);
 
 static int
@@ -129,6 +171,8 @@ ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 			.dev_start = ioat_dev_start,
 			.dev_stop = ioat_dev_stop,
 			.dev_info_get = ioat_dev_info_get,
+			.xstats_get = ioat_xstats_get,
+			.xstats_get_names = ioat_xstats_get_names,
 			.dev_selftest = ioat_rawdev_test,
 	};
 
diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c
index 5375da26c..ab671816d 100644
--- a/drivers/raw/ioat/ioat_rawdev_test.c
+++ b/drivers/raw/ioat/ioat_rawdev_test.c
@@ -2,6 +2,7 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <inttypes.h>
 #include "rte_rawdev.h"
 #include "rte_ioat_rawdev.h"
 
@@ -13,6 +14,11 @@ ioat_rawdev_test(uint16_t dev_id)
 #define IOAT_TEST_RINGSIZE 512
 	struct rte_ioat_rawdev_config p = { .ring_size = -1 };
 	struct rte_rawdev_info info = { .dev_private = &p };
+	struct rte_rawdev_xstats_name *snames = NULL;
+	uint64_t *stats = NULL;
+	unsigned int *ids = NULL;
+	unsigned int nb_xstats;
+	unsigned int i;
 
 	rte_rawdev_info_get(dev_id, &info);
 	if (p.ring_size != 0) {
@@ -37,5 +43,44 @@ ioat_rawdev_test(uint16_t dev_id)
 		printf("Error with rte_rawdev_start()\n");
 		return -1;
 	}
+
+	/* allocate memory for xstats names and values */
+	nb_xstats = rte_rawdev_xstats_names_get(dev_id, NULL, 0);
+
+	snames = malloc(sizeof(*snames) * nb_xstats);
+	if (snames == NULL) {
+		printf("Error allocating xstat names memory\n");
+		goto err;
+	}
+	rte_rawdev_xstats_names_get(dev_id, snames, nb_xstats);
+
+	ids = malloc(sizeof(*ids) * nb_xstats);
+	if (ids == NULL) {
+		printf("Error allocating xstat ids memory\n");
+		goto err;
+	}
+	for (i = 0; i < nb_xstats; i++)
+		ids[i] = i;
+
+	stats = malloc(sizeof(*stats) * nb_xstats);
+	if (stats == NULL) {
+		printf("Error allocating xstat memory\n");
+		goto err;
+	}
+
+	rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
+	for (i = 0; i < nb_xstats; i++)
+		printf("%s: %"PRIu64"   ", snames[i].name, stats[i]);
+	printf("\n");
+
+	free(snames);
+	free(stats);
+	free(ids);
 	return 0;
+
+err:
+	free(snames);
+	free(stats);
+	free(ids);
+	return -1;
 }
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index f2cf98cd9..d5326813c 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -54,6 +54,12 @@ struct rte_ioat_rawdev {
 	struct rte_ioat_generic_hw_desc *desc_ring;
 	__m128i *hdls; /* completion handles for returning to user */
 
+	/* some statistics for tracking, if added/changed update xstats fns*/
+	uint64_t enqueue_failed __rte_cache_aligned;
+	uint64_t enqueued;
+	uint64_t started;
+	uint64_t completed;
+
 	/* to report completions, the device will write status back here */
 	volatile uint64_t status __rte_cache_aligned;
 };
-- 
2.21.0

Re: [dpdk-dev] [PATCH v4 8/9] raw/ioat: add statistics functions

[Burakov, Anatoly]

On 01-Jul-19 4:55 PM, Bruce Richardson wrote:
> Add stats functions to track what is happening in the driver, and put
> unit tests to check those.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> ---
> V4: pulled in memory leak fixes from next patch
> V2: test cases moved to self-test routine
> ---

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

[dpdk-dev] [PATCH v4 9/9] raw/ioat: add local API to perform copies

[Bruce Richardson]

Add local APIs to trigger data copies, and retrieve handle values once
those copies are completed. Included are unit tests to validate the data
is copies correctly.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
---

V3: updated to use descriptor format in rte_ioat_spec.h
V2: test cases moved to self-test routine
---
 doc/guides/rawdevs/ioat_rawdev.rst  | 100 ++++++++++++++++
 drivers/raw/ioat/Makefile           |   1 +
 drivers/raw/ioat/ioat_rawdev_test.c | 155 ++++++++++++++++++++++++-
 drivers/raw/ioat/meson.build        |   2 +-
 drivers/raw/ioat/rte_ioat_rawdev.h  | 169 +++++++++++++++++++++++++++-
 5 files changed, 422 insertions(+), 5 deletions(-)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 40210b3f9..607cb5a86 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -150,6 +150,106 @@ The following code shows how the device is configured in
 Once configured, the device can then be made ready for use by calling the
 ``rte_rawdev_start()`` API.
 
+Performing Data Copies
+~~~~~~~~~~~~~~~~~~~~~~~
+
+To perform data copies using IOAT rawdev devices, the functions
+``rte_ioat_enqueue_copy()`` and ``rte_ioat_do_copies()`` should be used.
+Once copies have been completed, the completion will be reported back when
+the application calls ``rte_ioat_completed_copies()``.
+
+The ``rte_ioat_enqueue_copy()`` function enqueues a single copy to the
+device ring for copying at a later point. The parameters to that function
+include the physical addresses of both the source and destination buffers,
+as well as two "handles" to be returned to the user when the copy is
+completed. These handles can be arbitrary values, but two are provided so
+that the library can track handles for both source and destination on
+behalf of the user, e.g. virtual addresses for the buffers, or mbuf
+pointers if packet data is being copied.
+
+While the ``rte_ioat_enqueue_copy()`` function enqueues a copy operation on
+the device ring, the copy will not actually be performed until after the
+application calls the ``rte_ioat_do_copies()`` function. This function
+informs the device hardware of the elements enqueued on the ring, and the
+device will begin to process them. It is expected that, for efficiency
+reasons, a burst of operations will be enqueued to the device via multiple
+enqueue calls between calls to the ``rte_ioat_do_copies()`` function.
+
+The following code from ``test_ioat_rawdev.c`` demonstrates how to enqueue
+a burst of copies to the device and start the hardware processing of them:
+
+.. code-block:: C
+
+        struct rte_mbuf *srcs[32], *dsts[32];
+        unsigned int j;
+
+        for (i = 0; i < RTE_DIM(srcs); i++) {
+                char *src_data;
+
+                srcs[i] = rte_pktmbuf_alloc(pool);
+                dsts[i] = rte_pktmbuf_alloc(pool);
+                srcs[i]->data_len = srcs[i]->pkt_len = length;
+                dsts[i]->data_len = dsts[i]->pkt_len = length;
+                src_data = rte_pktmbuf_mtod(srcs[i], char *);
+
+                for (j = 0; j < length; j++)
+                        src_data[j] = rand() & 0xFF;
+
+                if (rte_ioat_enqueue_copy(dev_id,
+                                srcs[i]->buf_iova + srcs[i]->data_off,
+                                dsts[i]->buf_iova + dsts[i]->data_off,
+                                length,
+                                (uintptr_t)srcs[i],
+                                (uintptr_t)dsts[i],
+                                0 /* nofence */) != 1) {
+                        printf("Error with rte_ioat_enqueue_copy for buffer %u\n",
+                                        i);
+                        return -1;
+                }
+        }
+        rte_ioat_do_copies(dev_id);
+
+To retrieve information about completed copies, the API
+``rte_ioat_completed_copies()`` should be used. This API will return to the
+application a set of completion handles passed in when the relevant copies
+were enqueued.
+
+The following code from ``test_ioat_rawdev.c`` shows the test code
+retrieving information about the completed copies and validating the data
+is correct before freeing the data buffers using the returned handles:
+
+.. code-block:: C
+
+        if (rte_ioat_completed_copies(dev_id, 64, (void *)completed_src,
+                        (void *)completed_dst) != RTE_DIM(srcs)) {
+                printf("Error with rte_ioat_completed_copies\n");
+                return -1;
+        }
+        for (i = 0; i < RTE_DIM(srcs); i++) {
+                char *src_data, *dst_data;
+
+                if (completed_src[i] != srcs[i]) {
+                        printf("Error with source pointer %u\n", i);
+                        return -1;
+                }
+                if (completed_dst[i] != dsts[i]) {
+                        printf("Error with dest pointer %u\n", i);
+                        return -1;
+                }
+
+                src_data = rte_pktmbuf_mtod(srcs[i], char *);
+                dst_data = rte_pktmbuf_mtod(dsts[i], char *);
+                for (j = 0; j < length; j++)
+                        if (src_data[j] != dst_data[j]) {
+                                printf("Error with copy of packet %u, byte %u\n",
+                                                i, j);
+                                return -1;
+                        }
+                rte_pktmbuf_free(srcs[i]);
+                rte_pktmbuf_free(dsts[i]);
+        }
+
+
 Querying Device Statistics
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
index b1af9c666..32f079845 100644
--- a/drivers/raw/ioat/Makefile
+++ b/drivers/raw/ioat/Makefile
@@ -12,6 +12,7 @@ CFLAGS += $(WERROR_FLAGS)
 
 LDLIBS += -lrte_eal -lrte_rawdev
 LDLIBS += -lrte_pci -lrte_bus_pci
+LDLIBS += -lrte_mbuf -lrte_mempool
 
 # library version
 LIBABIVER := 1
diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c
index ab671816d..f6c7dbb80 100644
--- a/drivers/raw/ioat/ioat_rawdev_test.c
+++ b/drivers/raw/ioat/ioat_rawdev_test.c
@@ -2,12 +2,139 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <unistd.h>
 #include <inttypes.h>
+#include <rte_mbuf.h>
 #include "rte_rawdev.h"
 #include "rte_ioat_rawdev.h"
 
 int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */
 
+static struct rte_mempool *pool;
+
+static int
+test_enqueue_copies(int dev_id)
+{
+	const unsigned int length = 1024;
+	unsigned int i;
+
+	do {
+		struct rte_mbuf *src, *dst;
+		char *src_data, *dst_data;
+		struct rte_mbuf *completed[2] = {0};
+
+		/* test doing a single copy */
+		src = rte_pktmbuf_alloc(pool);
+		dst = rte_pktmbuf_alloc(pool);
+		src->data_len = src->pkt_len = length;
+		dst->data_len = dst->pkt_len = length;
+		src_data = rte_pktmbuf_mtod(src, char *);
+		dst_data = rte_pktmbuf_mtod(dst, char *);
+
+		for (i = 0; i < length; i++)
+			src_data[i] = rand() & 0xFF;
+
+		if (rte_ioat_enqueue_copy(dev_id,
+				src->buf_iova + src->data_off,
+				dst->buf_iova + dst->data_off,
+				length,
+				(uintptr_t)src,
+				(uintptr_t)dst,
+				0 /* no fence */) != 1) {
+			printf("Error with rte_ioat_enqueue_copy\n");
+			return -1;
+		}
+		rte_ioat_do_copies(dev_id);
+		usleep(10);
+
+		if (rte_ioat_completed_copies(dev_id, 1, (void *)&completed[0],
+				(void *)&completed[1]) != 1) {
+			printf("Error with rte_ioat_completed_copies\n");
+			return -1;
+		}
+		if (completed[0] != src || completed[1] != dst) {
+			printf("Error with completions: got (%p, %p), not (%p,%p)\n",
+					completed[0], completed[1], src, dst);
+			return -1;
+		}
+
+		for (i = 0; i < length; i++)
+			if (dst_data[i] != src_data[i]) {
+				printf("Data mismatch at char %u\n", i);
+				return -1;
+			}
+		rte_pktmbuf_free(src);
+		rte_pktmbuf_free(dst);
+	} while (0);
+
+	/* test doing multiple copies */
+	do {
+		struct rte_mbuf *srcs[32], *dsts[32];
+		struct rte_mbuf *completed_src[64];
+		struct rte_mbuf *completed_dst[64];
+		unsigned int j;
+
+		for (i = 0; i < RTE_DIM(srcs); i++) {
+			char *src_data;
+
+			srcs[i] = rte_pktmbuf_alloc(pool);
+			dsts[i] = rte_pktmbuf_alloc(pool);
+			srcs[i]->data_len = srcs[i]->pkt_len = length;
+			dsts[i]->data_len = dsts[i]->pkt_len = length;
+			src_data = rte_pktmbuf_mtod(srcs[i], char *);
+
+			for (j = 0; j < length; j++)
+				src_data[j] = rand() & 0xFF;
+
+			if (rte_ioat_enqueue_copy(dev_id,
+					srcs[i]->buf_iova + srcs[i]->data_off,
+					dsts[i]->buf_iova + dsts[i]->data_off,
+					length,
+					(uintptr_t)srcs[i],
+					(uintptr_t)dsts[i],
+					0 /* nofence */) != 1) {
+				printf("Error with rte_ioat_enqueue_copy for buffer %u\n",
+						i);
+				return -1;
+			}
+		}
+		rte_ioat_do_copies(dev_id);
+		usleep(100);
+
+		if (rte_ioat_completed_copies(dev_id, 64, (void *)completed_src,
+				(void *)completed_dst) != RTE_DIM(srcs)) {
+			printf("Error with rte_ioat_completed_copies\n");
+			return -1;
+		}
+		for (i = 0; i < RTE_DIM(srcs); i++) {
+			char *src_data, *dst_data;
+
+			if (completed_src[i] != srcs[i]) {
+				printf("Error with source pointer %u\n", i);
+				return -1;
+			}
+			if (completed_dst[i] != dsts[i]) {
+				printf("Error with dest pointer %u\n", i);
+				return -1;
+			}
+
+			src_data = rte_pktmbuf_mtod(srcs[i], char *);
+			dst_data = rte_pktmbuf_mtod(dsts[i], char *);
+			for (j = 0; j < length; j++)
+				if (src_data[j] != dst_data[j]) {
+					printf("Error with copy of packet %u, byte %u\n",
+							i, j);
+					return -1;
+				}
+			rte_pktmbuf_free(srcs[i]);
+			rte_pktmbuf_free(dsts[i]);
+		}
+
+	} while (0);
+
+	return 0;
+}
+
 int
 ioat_rawdev_test(uint16_t dev_id)
 {
@@ -44,6 +171,17 @@ ioat_rawdev_test(uint16_t dev_id)
 		return -1;
 	}
 
+	pool = rte_pktmbuf_pool_create("TEST_IOAT_POOL",
+			256, /* n == num elements */
+			32,  /* cache size */
+			0,   /* priv size */
+			2048, /* data room size */
+			info.socket_id);
+	if (pool == NULL) {
+		printf("Error with mempool creation\n");
+		return -1;
+	}
+
 	/* allocate memory for xstats names and values */
 	nb_xstats = rte_rawdev_xstats_names_get(dev_id, NULL, 0);
 
@@ -68,17 +206,28 @@ ioat_rawdev_test(uint16_t dev_id)
 		goto err;
 	}
 
-	rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
-	for (i = 0; i < nb_xstats; i++)
-		printf("%s: %"PRIu64"   ", snames[i].name, stats[i]);
+	/* run the test cases */
+	for (i = 0; i < 100; i++) {
+		unsigned int j;
+
+		if (test_enqueue_copies(dev_id) != 0)
+			goto err;
+
+		rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
+		for (j = 0; j < nb_xstats; j++)
+			printf("%s: %"PRIu64"   ", snames[j].name, stats[j]);
+		printf("\r");
+	}
 	printf("\n");
 
+	rte_mempool_free(pool);
 	free(snames);
 	free(stats);
 	free(ids);
 	return 0;
 
 err:
+	rte_mempool_free(pool);
 	free(snames);
 	free(stats);
 	free(ids);
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
index 40fff6654..247ff88bf 100644
--- a/drivers/raw/ioat/meson.build
+++ b/drivers/raw/ioat/meson.build
@@ -4,7 +4,7 @@
 build = dpdk_conf.has('RTE_ARCH_X86')
 sources = files('ioat_rawdev.c',
 		'ioat_rawdev_test.c')
-deps += ['rawdev', 'bus_pci']
+deps += ['rawdev', 'bus_pci', 'mbuf']
 
 install_headers('rte_ioat_rawdev.h',
 		'rte_ioat_spec.h')
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index d5326813c..3babb822d 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -15,8 +15,10 @@
  */
 
 #include <x86intrin.h>
+#include <rte_atomic.h>
 #include <rte_memory.h>
 #include <rte_memzone.h>
+#include <rte_prefetch.h>
 #include <rte_ioat_spec.h>
 
 /** Name of the device driver */
@@ -54,6 +56,10 @@ struct rte_ioat_rawdev {
 	struct rte_ioat_generic_hw_desc *desc_ring;
 	__m128i *hdls; /* completion handles for returning to user */
 
+
+	unsigned short next_read;
+	unsigned short next_write;
+
 	/* some statistics for tracking, if added/changed update xstats fns*/
 	uint64_t enqueue_failed __rte_cache_aligned;
 	uint64_t enqueued;
@@ -64,4 +70,165 @@ struct rte_ioat_rawdev {
 	volatile uint64_t status __rte_cache_aligned;
 };
 
-#endif
+/**
+ * Enqueue a copy operation onto the ioat device
+ *
+ * This queues up a copy operation to be performed by hardware, but does not
+ * trigger hardware to begin that operation.
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ * @param src
+ *   The physical address of the source buffer
+ * @param dst
+ *   The physical address of the destination buffer
+ * @param length
+ *   The length of the data to be copied
+ * @param src_hdl
+ *   An opaque handle for the source data, to be returned when this operation
+ *   has been completed and the user polls for the completion details
+ * @param dst_hdl
+ *   An opaque handle for the destination data, to be returned when this
+ *   operation has been completed and the user polls for the completion details
+ * @param fence
+ *   A flag parameter indicating that hardware should not begin to perform any
+ *   subsequently enqueued copy operations until after this operation has
+ *   completed
+ * @return
+ *   Number of operations enqueued, either 0 or 1
+ */
+static inline int
+rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst,
+		unsigned int length, uintptr_t src_hdl, uintptr_t dst_hdl,
+		int fence)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	unsigned short read = ioat->next_read;
+	unsigned short write = ioat->next_write;
+	unsigned short mask = ioat->ring_size - 1;
+	unsigned short space = mask + read - write;
+	struct rte_ioat_generic_hw_desc *desc;
+
+	if (space == 0) {
+		ioat->enqueue_failed++;
+		return 0;
+	}
+
+	ioat->next_write = write + 1;
+	write &= mask;
+
+	desc = &ioat->desc_ring[write];
+	desc->size = length;
+	/* set descriptor write-back every 16th descriptor */
+	desc->u.control_raw = (uint32_t)((!!fence << 4) | (!(write & 0xF)) << 3);
+	desc->src_addr = src;
+	desc->dest_addr = dst;
+
+	ioat->hdls[write] = _mm_set_epi64((__m64)((uint64_t)dst_hdl),
+			(__m64)((uint64_t)src_hdl));
+	rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]);
+
+	ioat->enqueued++;
+	return 1;
+}
+
+/**
+ * Trigger hardware to begin performing enqueued copy operations
+ *
+ * This API is used to write the "doorbell" to the hardware to trigger it
+ * to begin the copy operations previously enqueued by rte_ioat_enqueue_copy()
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ */
+static inline void
+rte_ioat_do_copies(int dev_id)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	ioat->desc_ring[(ioat->next_write - 1) & (ioat->ring_size - 1)].u
+			.control.completion_update = 1;
+	rte_compiler_barrier();
+	ioat->regs->dmacount = ioat->next_write;
+	ioat->started = ioat->enqueued;
+}
+
+/**
+ * @internal
+ * Returns the index of the last completed operation.
+ */
+static inline int
+rte_ioat_get_last_completed(struct rte_ioat_rawdev *ioat, int *error)
+{
+	uint64_t status = ioat->status;
+
+	/* lower 3 bits indicate "transfer status" : active, idle, halted.
+	 * We can ignore bit 0.
+	 */
+	*error = status & (RTE_IOAT_CHANSTS_SUSPENDED | RTE_IOAT_CHANSTS_ARMED);
+	return (status - ioat->ring_addr) >> 6;
+}
+
+/**
+ * Returns details of copy operations that have been completed
+ *
+ * Returns to the caller the user-provided "handles" for the copy operations
+ * which have been completed by the hardware, and not already returned by
+ * a previous call to this API.
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ * @param max_copies
+ *   The number of entries which can fit in the src_hdls and dst_hdls
+ *   arrays, i.e. max number of completed operations to report
+ * @param src_hdls
+ *   Array to hold the source handle parameters of the completed copies
+ * @param dst_hdls
+ *   Array to hold the destination handle parameters of the completed copies
+ * @return
+ *   -1 on error, with rte_errno set appropriately.
+ *   Otherwise number of completed operations i.e. number of entries written
+ *   to the src_hdls and dst_hdls array parameters.
+ */
+static inline int
+rte_ioat_completed_copies(int dev_id, uint8_t max_copies,
+		uintptr_t *src_hdls, uintptr_t *dst_hdls)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	unsigned short mask = (ioat->ring_size - 1);
+	unsigned short read = ioat->next_read;
+	unsigned short end_read, count;
+	int error;
+	int i = 0;
+
+	end_read = (rte_ioat_get_last_completed(ioat, &error) + 1) & mask;
+	count = (end_read - (read & mask)) & mask;
+
+	if (error) {
+		rte_errno = EIO;
+		return -1;
+	}
+
+	if (count > max_copies)
+		count = max_copies;
+
+	for (; i < count - 1; i += 2, read += 2) {
+		__m128i hdls0 = _mm_load_si128(&ioat->hdls[read & mask]);
+		__m128i hdls1 = _mm_load_si128(&ioat->hdls[(read + 1) & mask]);
+
+		_mm_storeu_si128((void *)&src_hdls[i],
+				_mm_unpacklo_epi64(hdls0, hdls1));
+		_mm_storeu_si128((void *)&dst_hdls[i],
+				_mm_unpackhi_epi64(hdls0, hdls1));
+	}
+	for (; i < count; i++, read++) {
+		uintptr_t *hdls = (void *)&ioat->hdls[read & mask];
+		src_hdls[i] = hdls[0];
+		dst_hdls[i] = hdls[1];
+	}
+
+	ioat->next_read = read;
+	ioat->completed += count;
+	return count;
+}
+
+#endif /* _RTE_IOAT_RAWDEV_H_ */
-- 
2.21.0

Re: [dpdk-dev] [PATCH v4 0/9] raw/ioat: driver for Intel QuickData Technology

[Bruce Richardson]

On Mon, Jul 01, 2019 at 04:55:51PM +0100, Bruce Richardson wrote:
> This patch series adds support for the Intel QuickData Technology
> device, part of the Intel I/O Acceleration Technology (Intel I/OAT). It
> is a raw device for allowing hardware DMA i.e. data copies in hardware.
> 
> Performing the copies in hardware can provide performance improvements
> for applications where the average copy size is reasonably large, e.g.
> 1k packets. For smaller packets, e.g. 64-256 bytes, offloading the copy
> may reduce performance due to the overhead of using hardware.
> 

Apologies, forgot to mention this set has a dependency on:
http://patches.dpdk.org/project/dpdk/list/?series=5120

/Bruce

[dpdk-dev] [PATCH v5 0/9] raw/ioat: driver for Intel QuickData Technology

[Bruce Richardson]

This patch series adds support for the Intel QuickData Technology
device, part of the Intel I/O Acceleration Technology (Intel I/OAT). It
is a raw device for allowing hardware DMA i.e. data copies in hardware.

Performing the copies in hardware can provide performance improvements
for applications where the average copy size is reasonably large, e.g.
1k packets. For smaller packets, e.g. 64-256 bytes, offloading the copy
may reduce performance due to the overhead of using hardware.

V5:
 * updated doxygen comment for internal rawdev pmd API

V4:
 * changed memory management to use contiguous memzones instead of
   malloc memory
 * added missing device ids for BDX platforms
 * other misc cleanup following review (see individual patches logs)

V3:
 * removed DPDK-specific structure for the descriptor format and reused
   the structure in the imported file rte_ioat_spec.h

V2:
* moved tests to rawdev selftest function
* some checkpatch and other small cleanups
* added extra documentation details on supported hardware
* aligned the changes to dpdk-devbind with the changes in the NTB set
  for consistency


Bruce Richardson (9):
  rawdev: allow devices to skip extra memory allocation
  raw/ioat: add initial support for ioat rawdev driver
  usertools/dpdk-devbind.py: add support for IOAT devices
  raw/ioat: add register definition file
  raw/ioat: create device on probe and destroy on release
  raw/ioat: add device info function
  raw/ioat: add configure, start and stop functions
  raw/ioat: add statistics functions
  raw/ioat: add local API to perform copies

 MAINTAINERS                                 |   5 +
 app/test/test_rawdev.c                      |  20 ++
 config/common_armv8a_linux                  |   1 +
 config/common_base                          |   5 +
 config/defconfig_arm-armv7a-linuxapp-gcc    |   1 +
 config/defconfig_ppc_64-power8-linuxapp-gcc |   1 +
 doc/guides/rawdevs/index.rst                |   1 +
 doc/guides/rawdevs/ioat_rawdev.rst          | 265 +++++++++++++++
 doc/guides/rel_notes/release_19_08.rst      |  11 +
 drivers/raw/Makefile                        |   1 +
 drivers/raw/ioat/Makefile                   |  31 ++
 drivers/raw/ioat/ioat_rawdev.c              | 356 ++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev_test.c         | 235 +++++++++++++
 drivers/raw/ioat/meson.build                |  10 +
 drivers/raw/ioat/rte_ioat_rawdev.h          | 234 +++++++++++++
 drivers/raw/ioat/rte_ioat_spec.h            | 301 +++++++++++++++++
 drivers/raw/ioat/rte_pmd_ioat_version.map   |   4 +
 drivers/raw/meson.build                     |   4 +-
 lib/librte_rawdev/rte_rawdev.c              |  11 +-
 lib/librte_rawdev/rte_rawdev_pmd.h          |   4 +-
 mk/rte.app.mk                               |   1 +
 usertools/dpdk-devbind.py                   |  10 +
 22 files changed, 1505 insertions(+), 7 deletions(-)
 create mode 100644 doc/guides/rawdevs/ioat_rawdev.rst
 create mode 100644 drivers/raw/ioat/Makefile
 create mode 100644 drivers/raw/ioat/ioat_rawdev.c
 create mode 100644 drivers/raw/ioat/ioat_rawdev_test.c
 create mode 100644 drivers/raw/ioat/meson.build
 create mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h
 create mode 100644 drivers/raw/ioat/rte_ioat_spec.h
 create mode 100644 drivers/raw/ioat/rte_pmd_ioat_version.map

-- 
2.21.0

[dpdk-dev] [PATCH v5 1/9] rawdev: allow devices to skip extra memory allocation

[Bruce Richardson]

Some device drivers want to allocate their own private memory, and should
be allowed to do so. Therefore skip memory allocation and associated error
checks if zero-length private memory is requested.

While adjusting the code for new indent level, fix incorrect error
message.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Shreyansh Jain <shreyansh.jain@nxp.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>

---
V5: Added additional info to doxygen for allocate API
---
 lib/librte_rawdev/rte_rawdev.c     | 11 ++++++-----
 lib/librte_rawdev/rte_rawdev_pmd.h |  4 +++-
 2 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/lib/librte_rawdev/rte_rawdev.c b/lib/librte_rawdev/rte_rawdev.c
index 15de2d413..b6f1e1c77 100644
--- a/lib/librte_rawdev/rte_rawdev.c
+++ b/lib/librte_rawdev/rte_rawdev.c
@@ -496,16 +496,17 @@ rte_rawdev_pmd_allocate(const char *name, size_t dev_priv_size, int socket_id)
 
 	rawdev = &rte_rawdevs[dev_id];
 
-	rawdev->dev_private = rte_zmalloc_socket("rawdev private",
+	if (dev_priv_size > 0) {
+		rawdev->dev_private = rte_zmalloc_socket("rawdev private",
 				     dev_priv_size,
 				     RTE_CACHE_LINE_SIZE,
 				     socket_id);
-	if (!rawdev->dev_private) {
-		RTE_RDEV_ERR("Unable to allocate memory to Skeleton dev");
-		return NULL;
+		if (!rawdev->dev_private) {
+			RTE_RDEV_ERR("Unable to allocate memory for rawdev");
+			return NULL;
+		}
 	}
 
-
 	rawdev->dev_id = dev_id;
 	rawdev->socket_id = socket_id;
 	rawdev->started = 0;
diff --git a/lib/librte_rawdev/rte_rawdev_pmd.h b/lib/librte_rawdev/rte_rawdev_pmd.h
index aa6af4a37..cb3555ab5 100644
--- a/lib/librte_rawdev/rte_rawdev_pmd.h
+++ b/lib/librte_rawdev/rte_rawdev_pmd.h
@@ -568,7 +568,9 @@ struct rte_rawdev_ops {
  * @param name
  *   Unique identifier name for each device
  * @param dev_private_size
- *   Private data allocated within rte_rawdev object.
+ *   Size of private data memory allocated within rte_rawdev object.
+ *   Set to 0 to disable internal memory allocation and allow for
+ *   self-allocation.
  * @param socket_id
  *   Socket to allocate resources on.
  * @return
-- 
2.21.0

[dpdk-dev] [PATCH v5 2/9] raw/ioat: add initial support for ioat rawdev driver

[Bruce Richardson]

Add stubs for ioat rawdev driver support in DPDK, specifically:

  * makefile and meson build hooks
  * initial public header file
  * rawdev main C file, with probe and release functions
  * release note update announcing the driver
  * initial documentation for the new section in the rawdev doc
  * unit test stubs for device unit tests

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>

---
V4: add in missing device IDs
V3: remove unneeded whitespace change
V2: don't create a new file for unit testing, add to existing rawdev test
    file, and place test cases in the driver selftest routine (added later
    in set)
    Add new section in document about identifying hardware using lspci
---
 MAINTAINERS                                 |   5 +
 app/test/test_rawdev.c                      |   8 ++
 config/common_armv8a_linux                  |   1 +
 config/common_base                          |   5 +
 config/defconfig_arm-armv7a-linuxapp-gcc    |   1 +
 config/defconfig_ppc_64-power8-linuxapp-gcc |   1 +
 doc/guides/rawdevs/index.rst                |   1 +
 doc/guides/rawdevs/ioat_rawdev.rst          |  63 +++++++++++
 doc/guides/rel_notes/release_19_08.rst      |  11 ++
 drivers/raw/Makefile                        |   1 +
 drivers/raw/ioat/Makefile                   |  28 +++++
 drivers/raw/ioat/ioat_rawdev.c              | 113 ++++++++++++++++++++
 drivers/raw/ioat/meson.build                |   8 ++
 drivers/raw/ioat/rte_ioat_rawdev.h          |  24 +++++
 drivers/raw/ioat/rte_pmd_ioat_version.map   |   4 +
 drivers/raw/meson.build                     |   4 +-
 mk/rte.app.mk                               |   1 +
 17 files changed, 278 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/rawdevs/ioat_rawdev.rst
 create mode 100644 drivers/raw/ioat/Makefile
 create mode 100644 drivers/raw/ioat/ioat_rawdev.c
 create mode 100644 drivers/raw/ioat/meson.build
 create mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h
 create mode 100644 drivers/raw/ioat/rte_pmd_ioat_version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 97a009e43..dac4c866f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1084,6 +1084,11 @@ M: Tianfei zhang <tianfei.zhang@intel.com>
 F: drivers/raw/ifpga_rawdev/
 F: doc/guides/rawdevs/ifpga_rawdev.rst
 
+IOAT Rawdev
+M: Bruce Richardson <bruce.richardson@intel.com>
+F: drivers/raw/ioat/
+F: doc/guides/rawdevs/ioat_rawdev.rst
+
 NXP DPAA2 QDMA
 M: Nipun Gupta <nipun.gupta@nxp.com>
 F: drivers/raw/dpaa2_qdma/
diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
index 043a38a13..88549fb61 100644
--- a/app/test/test_rawdev.c
+++ b/app/test/test_rawdev.c
@@ -25,3 +25,11 @@ test_rawdev_selftest_skeleton(void)
 }
 
 REGISTER_TEST_COMMAND(rawdev_autotest, test_rawdev_selftest_skeleton);
+
+static int
+test_rawdev_selftest_ioat(void)
+{
+	return TEST_SKIPPED;
+}
+
+REGISTER_TEST_COMMAND(ioat_rawdev_autotest, test_rawdev_selftest_ioat);
diff --git a/config/common_armv8a_linux b/config/common_armv8a_linux
index 72091de1c..481712ebc 100644
--- a/config/common_armv8a_linux
+++ b/config/common_armv8a_linux
@@ -34,5 +34,6 @@ CONFIG_RTE_ARCH_ARM64_MEMCPY=n
 CONFIG_RTE_LIBRTE_FM10K_PMD=n
 CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
 
 CONFIG_RTE_SCHED_VECTOR=n
diff --git a/config/common_base b/config/common_base
index e700bf1e7..47ba94315 100644
--- a/config/common_base
+++ b/config/common_base
@@ -752,6 +752,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n
 #
 CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
 
+#
+# Compile PMD for Intel IOAT raw device
+#
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=y
+
 #
 # Compile librte_ring
 #
diff --git a/config/defconfig_arm-armv7a-linuxapp-gcc b/config/defconfig_arm-armv7a-linuxapp-gcc
index 562439c0b..26ab5c57d 100644
--- a/config/defconfig_arm-armv7a-linuxapp-gcc
+++ b/config/defconfig_arm-armv7a-linuxapp-gcc
@@ -55,3 +55,4 @@ CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
 CONFIG_RTE_LIBRTE_NFP_PMD=n
 CONFIG_RTE_LIBRTE_HINIC_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
diff --git a/config/defconfig_ppc_64-power8-linuxapp-gcc b/config/defconfig_ppc_64-power8-linuxapp-gcc
index cec434563..e8dafa3b4 100644
--- a/config/defconfig_ppc_64-power8-linuxapp-gcc
+++ b/config/defconfig_ppc_64-power8-linuxapp-gcc
@@ -57,3 +57,4 @@ CONFIG_RTE_LIBRTE_FM10K_PMD=n
 CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n
 CONFIG_RTE_LIBRTE_AVP_PMD=n
 CONFIG_RTE_LIBRTE_HINIC_PMD=n
+CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n
diff --git a/doc/guides/rawdevs/index.rst b/doc/guides/rawdevs/index.rst
index 7c3bd9586..0a21989e4 100644
--- a/doc/guides/rawdevs/index.rst
+++ b/doc/guides/rawdevs/index.rst
@@ -14,3 +14,4 @@ application through rawdev API.
     dpaa2_cmdif
     dpaa2_qdma
     ifpga_rawdev
+    ioat_rawdev
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
new file mode 100644
index 000000000..0c612e73a
--- /dev/null
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -0,0 +1,63 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+.. include:: <isonum.txt>
+
+IOAT Rawdev Driver for Intel\ |reg| QuickData Technology
+======================================================================
+
+The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg|
+QuickData Technology, part of Intel\ |reg| I/O Acceleration Technology
+`(Intel I/OAT)
+<https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_.
+This PMD, when used on supported hardware, allows data copies, for example,
+cloning packet data, to be accelerated by that hardware rather than having to
+be done by software, freeing up CPU cycles for other tasks.
+
+Hardware Requirements
+----------------------
+
+On Linux, the presence of an Intel\ |reg| QuickData Technology hardware can
+be detected by checking the output of the ``lspci`` command, where the
+hardware will be often listed as "Crystal Beach DMA" or "CBDMA". For
+example, on a system with Intel\ |reg| Xeon\ |reg| CPU E5-2699 v4 @2.20GHz,
+lspci shows:
+
+.. code-block:: console
+
+  # lspci | grep DMA
+  00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01)
+  00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01)
+  00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01)
+  00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01)
+  00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01)
+  00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01)
+  00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01)
+  00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01)
+
+On a system with Intel\ |reg| Xeon\ |reg| Gold 6154 CPU @ 3.00GHz, lspci
+shows:
+
+.. code-block:: console
+
+  # lspci | grep DMA
+  00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+  00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
+
+
+Compilation
+------------
+
+For builds done with ``make``, the driver compilation is enabled by the
+``CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV`` build configuration option. This is
+enabled by default in builds for x86 platforms, and disabled in other
+configurations.
+
+For builds using ``meson`` and ``ninja``, the driver will be built when the
+target platform is x86-based.
diff --git a/doc/guides/rel_notes/release_19_08.rst b/doc/guides/rel_notes/release_19_08.rst
index 60c258136..cf796694f 100644
--- a/doc/guides/rel_notes/release_19_08.rst
+++ b/doc/guides/rel_notes/release_19_08.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright 2019 The DPDK contributors
 
+.. include:: <isonum.txt>
+
 DPDK Release 19.08
 ==================
 
@@ -126,6 +128,15 @@ New Features
   Added telemetry mode to l3fwd-power application to report
   application level busyness, empty and full polls of rte_eth_rx_burst().
 
+* **Added Intel QuickData Technology PMD**
+
+  The PMD for Intel\ |reg|  QuickData Technology, part of
+  Intel\ |reg|  I/O Acceleration Technology `(Intel I/OAT)
+  <https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_,
+  allows data copies to be done by hardware instead
+  of via software, reducing cycles spent copying large blocks of data in
+  applications.
+
 
 Removed Items
 -------------
diff --git a/drivers/raw/Makefile b/drivers/raw/Makefile
index 8e29b4a56..c1b85c8c7 100644
--- a/drivers/raw/Makefile
+++ b/drivers/raw/Makefile
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev
+DIRS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat
 
 include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
new file mode 100644
index 000000000..7726e310a
--- /dev/null
+++ b/drivers/raw/ioat/Makefile
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_pmd_ioat_rawdev.a
+
+# build flags
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+
+LDLIBS += -lrte_eal -lrte_rawdev
+LDLIBS += -lrte_pci -lrte_bus_pci
+
+# library version
+LIBABIVER := 1
+
+# versioning export map
+EXPORT_MAP := rte_pmd_ioat_version.map
+
+# library source files
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
+
+# export include files
+SYMLINK-y-include += rte_ioat_rawdev.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
new file mode 100644
index 000000000..07f6d8dfa
--- /dev/null
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -0,0 +1,113 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include <rte_bus_pci.h>
+#include <rte_rawdev_pmd.h>
+
+#include "rte_ioat_rawdev.h"
+
+/* Dynamic log type identifier */
+int ioat_pmd_logtype;
+
+static struct rte_pci_driver ioat_pmd_drv;
+
+#define IOAT_VENDOR_ID		0x8086
+#define IOAT_DEVICE_ID_SKX	0x2021
+#define IOAT_DEVICE_ID_BDX0	0x6f20
+#define IOAT_DEVICE_ID_BDX1	0x6f21
+#define IOAT_DEVICE_ID_BDX2	0x6f22
+#define IOAT_DEVICE_ID_BDX3	0x6f23
+#define IOAT_DEVICE_ID_BDX4	0x6f24
+#define IOAT_DEVICE_ID_BDX5	0x6f25
+#define IOAT_DEVICE_ID_BDX6	0x6f26
+#define IOAT_DEVICE_ID_BDX7	0x6f27
+#define IOAT_DEVICE_ID_BDXE	0x6f2E
+#define IOAT_DEVICE_ID_BDXF	0x6f2F
+
+#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \
+	ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args)
+
+#define IOAT_PMD_DEBUG(fmt, args...)  IOAT_PMD_LOG(DEBUG, fmt, ## args)
+#define IOAT_PMD_INFO(fmt, args...)   IOAT_PMD_LOG(INFO, fmt, ## args)
+#define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
+#define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
+
+static int
+ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
+{
+	RTE_SET_USED(name);
+	RTE_SET_USED(dev);
+	return 0;
+}
+
+static int
+ioat_rawdev_destroy(const char *name)
+{
+	RTE_SET_USED(name);
+	return 0;
+}
+
+static int
+ioat_rawdev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev)
+{
+	char name[32];
+	int ret = 0;
+
+
+	rte_pci_device_name(&dev->addr, name, sizeof(name));
+	IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node);
+
+	dev->device.driver = &drv->driver;
+	ret = ioat_rawdev_create(name, dev);
+	return ret;
+}
+
+static int
+ioat_rawdev_remove(struct rte_pci_device *dev)
+{
+	char name[32];
+	int ret;
+
+	rte_pci_device_name(&dev->addr, name, sizeof(name));
+
+	IOAT_PMD_INFO("Closing %s on NUMA node %d",
+			name, dev->device.numa_node);
+
+	ret = ioat_rawdev_destroy(name);
+	return ret;
+}
+
+static const struct rte_pci_id pci_id_ioat_map[] = {
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_SKX) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX0) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX1) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX2) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX3) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX4) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX5) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX6) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX7) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXE) },
+	{ RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXF) },
+	{ .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver ioat_pmd_drv = {
+	.id_table = pci_id_ioat_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+		     RTE_PCI_DRV_IOVA_AS_VA,
+	.probe = ioat_rawdev_probe,
+	.remove = ioat_rawdev_remove,
+};
+
+RTE_PMD_REGISTER_PCI(IOAT_PMD_RAWDEV_NAME, ioat_pmd_drv);
+RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_RAWDEV_NAME, pci_id_ioat_map);
+RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_RAWDEV_NAME, "* igb_uio | uio_pci_generic");
+
+RTE_INIT(ioat_pmd_init_log)
+{
+	ioat_pmd_logtype = rte_log_register(IOAT_PMD_LOG_NAME);
+	if (ioat_pmd_logtype >= 0)
+		rte_log_set_level(ioat_pmd_logtype, RTE_LOG_INFO);
+}
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
new file mode 100644
index 000000000..ba7620a68
--- /dev/null
+++ b/drivers/raw/ioat/meson.build
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2019 Intel Corporation
+
+build = dpdk_conf.has('RTE_ARCH_X86')
+sources = files('ioat_rawdev.c')
+deps += ['rawdev', 'bus_pci']
+
+install_headers('rte_ioat_rawdev.h')
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
new file mode 100644
index 000000000..e77406403
--- /dev/null
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_IOAT_RAWDEV_H_
+#define _RTE_IOAT_RAWDEV_H_
+
+/**
+ * @file rte_ioat_rawdev.h
+ *
+ * Definitions for using the ioat rawdev device driver
+ *
+ * @warning
+ * @b EXPERIMENTAL: these structures and APIs may change without prior notice
+ */
+
+/** Name of the device driver */
+#define IOAT_PMD_RAWDEV_NAME rawdev_ioat
+/** String reported as the device driver name by rte_rawdev_info_get() */
+#define IOAT_PMD_RAWDEV_NAME_STR "rawdev_ioat"
+/** Name used to adjust the log level for this driver */
+#define IOAT_PMD_LOG_NAME "rawdev.ioat"
+
+#endif
diff --git a/drivers/raw/ioat/rte_pmd_ioat_version.map b/drivers/raw/ioat/rte_pmd_ioat_version.map
new file mode 100644
index 000000000..9a61188cd
--- /dev/null
+++ b/drivers/raw/ioat/rte_pmd_ioat_version.map
@@ -0,0 +1,4 @@
+DPDK_19.08 {
+
+	local: *;
+};
diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build
index a61cdccef..2af8a70d4 100644
--- a/drivers/raw/meson.build
+++ b/drivers/raw/meson.build
@@ -1,7 +1,9 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2018 NXP
 
-drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
+drivers = ['dpaa2_cmdif', 'dpaa2_qdma',
+	'ifpga_rawdev', 'ioat',
+	'skeleton_rawdev']
 std_deps = ['rawdev']
 config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
 driver_name_fmt = 'rte_pmd_@0@'
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 2b5696a27..d31f5d55f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -307,6 +307,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV)   += -lrte_pmd_ifpga_rawdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD)       += -lrte_pmd_ipn3ke
 endif # CONFIG_RTE_LIBRTE_IFPGA_BUS
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV)   += -lrte_pmd_ioat_rawdev
 endif # CONFIG_RTE_LIBRTE_RAWDEV
 
 endif # !CONFIG_RTE_BUILD_SHARED_LIBS
-- 
2.21.0

Re: [dpdk-dev] [PATCH v5 2/9] raw/ioat: add initial support for ioat rawdev driver

[Hu, Jiayu]

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Tuesday, July 2, 2019 10:12 PM
> To: dev@dpdk.org
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Subject: [PATCH v5 2/9] raw/ioat: add initial support for ioat rawdev driver
> 
> Add stubs for ioat rawdev driver support in DPDK, specifically:
> 
>   * makefile and meson build hooks
>   * initial public header file
>   * rawdev main C file, with probe and release functions
>   * release note update announcing the driver
>   * initial documentation for the new section in the rawdev doc
>   * unit test stubs for device unit tests
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> 
> ---
> V4: add in missing device IDs
> V3: remove unneeded whitespace change
> V2: don't create a new file for unit testing, add to existing rawdev test
>     file, and place test cases in the driver selftest routine (added later
>     in set)
>     Add new section in document about identifying hardware using lspci
> ---
> 

Acked-by: Jiayu Hu <jiayu.hu@intel.com>

[dpdk-dev] [PATCH v5 3/9] usertools/dpdk-devbind.py: add support for IOAT devices

[Bruce Richardson]

In order to allow binding/unbinding of devices for use by the
ioat_rawdev, we need to update the devbind script to add a new class
of device, and add device ids for the specific HW instances.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>

---

V2: rather than adding a DMA section, add to "misc (rawdev)" section where
    other device types, e.g. ntb can also do.

NOTE: this set largely overlaps with the equivalent changes made by the
patchset adding NTB support [1]. Since it's unclear which set will be added
first, this set is based off the latest head.

[1] http://patches.dpdk.org/patch/55127/
---
 doc/guides/rawdevs/ioat_rawdev.rst | 11 +++++++++++
 usertools/dpdk-devbind.py          | 10 ++++++++++
 2 files changed, 21 insertions(+)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 0c612e73a..1a4b0e03e 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -61,3 +61,14 @@ configurations.
 
 For builds using ``meson`` and ``ninja``, the driver will be built when the
 target platform is x86-based.
+
+Device Setup
+-------------
+
+The Intel\ |reg| QuickData Technology HW devices will need to be bound to a
+user-space IO driver for use. The script ``dpdk-devbind.py`` script
+included with DPDK can be used to view the state of the devices and to bind
+them to a suitable DPDK-supported kernel driver. When querying the status
+of the devices, they will appear under the category of "Misc (rawdev)
+devices", i.e. the command ``dpdk-devbind.py --status-dev misc`` can be
+used to see the state of those devices alone.
diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py
index 9e79f0d28..5c1cd3548 100755
--- a/usertools/dpdk-devbind.py
+++ b/usertools/dpdk-devbind.py
@@ -36,11 +36,17 @@
 octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc',
               'SVendor': None, 'SDevice': None}
 
+intel_ioat_bdw = {'Class': '08', 'Vendor': '8086', 'Device': '6f20,6f21,6f22,6f23,6f24,6f25,6f26,6f27,6f2e,6f2f',
+              'SVendor': None, 'SDevice': None}
+intel_ioat_skx = {'Class': '08', 'Vendor': '8086', 'Device': '2021',
+              'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 crypto_devices = [encryption_class, intel_processor_class]
 eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso]
 mempool_devices = [cavium_fpa, octeontx2_npa]
 compress_devices = [cavium_zip]
+misc_devices = [intel_ioat_bdw, intel_ioat_skx]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
@@ -595,6 +601,8 @@ def show_status():
     if status_dev == "compress" or status_dev == "all":
         show_device_status(compress_devices , "Compress")
 
+    if status_dev == "misc" or status_dev == "all":
+        show_device_status(misc_devices, "Misc (rawdev)")
 
 def parse_args():
     '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +678,7 @@ def do_arg_actions():
             get_device_details(eventdev_devices)
             get_device_details(mempool_devices)
             get_device_details(compress_devices)
+            get_device_details(misc_devices)
         show_status()
 
 
@@ -690,6 +699,7 @@ def main():
     get_device_details(eventdev_devices)
     get_device_details(mempool_devices)
     get_device_details(compress_devices)
+    get_device_details(misc_devices)
     do_arg_actions()
 
 if __name__ == "__main__":
-- 
2.21.0

Re: [dpdk-dev] [PATCH v5 3/9] usertools/dpdk-devbind.py: add support for IOAT devices

[Hu, Jiayu]

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Tuesday, July 2, 2019 10:12 PM
> To: dev@dpdk.org
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Subject: [PATCH v5 3/9] usertools/dpdk-devbind.py: add support for IOAT
> devices
> 
> In order to allow binding/unbinding of devices for use by the
> ioat_rawdev, we need to update the devbind script to add a new class
> of device, and add device ids for the specific HW instances.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> 
> ---
> 
> V2: rather than adding a DMA section, add to "misc (rawdev)" section where
>     other device types, e.g. ntb can also do.
> 

Acked-by: Jiayu Hu <jiayu.hu@intel.com>

[dpdk-dev] [PATCH v5 4/9] raw/ioat: add register definition file

[Bruce Richardson]

Add in the list of registers for the device. File is taken from the SPDK
project:

  https://github.com/spdk/spdk/blob/master/include/spdk/ioat_spec.h

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
---
 drivers/raw/ioat/Makefile        |   1 +
 drivers/raw/ioat/meson.build     |   3 +-
 drivers/raw/ioat/rte_ioat_spec.h | 301 +++++++++++++++++++++++++++++++
 3 files changed, 304 insertions(+), 1 deletion(-)
 create mode 100644 drivers/raw/ioat/rte_ioat_spec.h

diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
index 7726e310a..1e10938f3 100644
--- a/drivers/raw/ioat/Makefile
+++ b/drivers/raw/ioat/Makefile
@@ -24,5 +24,6 @@ SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
 
 # export include files
 SYMLINK-y-include += rte_ioat_rawdev.h
+SYMLINK-y-include += rte_ioat_spec.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
index ba7620a68..ca23e23fc 100644
--- a/drivers/raw/ioat/meson.build
+++ b/drivers/raw/ioat/meson.build
@@ -5,4 +5,5 @@ build = dpdk_conf.has('RTE_ARCH_X86')
 sources = files('ioat_rawdev.c')
 deps += ['rawdev', 'bus_pci']
 
-install_headers('rte_ioat_rawdev.h')
+install_headers('rte_ioat_rawdev.h',
+		'rte_ioat_spec.h')
diff --git a/drivers/raw/ioat/rte_ioat_spec.h b/drivers/raw/ioat/rte_ioat_spec.h
new file mode 100644
index 000000000..305e36ded
--- /dev/null
+++ b/drivers/raw/ioat/rte_ioat_spec.h
@@ -0,0 +1,301 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) Intel Corporation
+ */
+
+/**
+ * \file
+ * I/OAT specification definitions
+ *
+ * Taken from ioat_spec.h from SPDK project, with prefix renames and
+ * other minor changes.
+ */
+
+#ifndef RTE_IOAT_SPEC_H
+#define RTE_IOAT_SPEC_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+#define RTE_IOAT_PCI_CHANERR_INT_OFFSET	0x180
+
+#define RTE_IOAT_INTRCTRL_MASTER_INT_EN	0x01
+
+#define RTE_IOAT_VER_3_0                0x30
+#define RTE_IOAT_VER_3_3                0x33
+
+/* DMA Channel Registers */
+#define RTE_IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK	0xF000
+#define RTE_IOAT_CHANCTRL_COMPL_DCA_EN		0x0200
+#define RTE_IOAT_CHANCTRL_CHANNEL_IN_USE		0x0100
+#define RTE_IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL	0x0020
+#define RTE_IOAT_CHANCTRL_ERR_INT_EN		0x0010
+#define RTE_IOAT_CHANCTRL_ANY_ERR_ABORT_EN		0x0008
+#define RTE_IOAT_CHANCTRL_ERR_COMPLETION_EN		0x0004
+#define RTE_IOAT_CHANCTRL_INT_REARM			0x0001
+
+/* DMA Channel Capabilities */
+#define	RTE_IOAT_DMACAP_PB		(1 << 0)
+#define	RTE_IOAT_DMACAP_DCA		(1 << 4)
+#define	RTE_IOAT_DMACAP_BFILL		(1 << 6)
+#define	RTE_IOAT_DMACAP_XOR		(1 << 8)
+#define	RTE_IOAT_DMACAP_PQ		(1 << 9)
+#define	RTE_IOAT_DMACAP_DMA_DIF	(1 << 10)
+
+struct rte_ioat_registers {
+	uint8_t		chancnt;
+	uint8_t		xfercap;
+	uint8_t		genctrl;
+	uint8_t		intrctrl;
+	uint32_t	attnstatus;
+	uint8_t		cbver;		/* 0x08 */
+	uint8_t		reserved4[0x3]; /* 0x09 */
+	uint16_t	intrdelay;	/* 0x0C */
+	uint16_t	cs_status;	/* 0x0E */
+	uint32_t	dmacapability;	/* 0x10 */
+	uint8_t		reserved5[0x6C]; /* 0x14 */
+	uint16_t	chanctrl;	/* 0x80 */
+	uint8_t		reserved6[0x2];	/* 0x82 */
+	uint8_t		chancmd;	/* 0x84 */
+	uint8_t		reserved3[1];	/* 0x85 */
+	uint16_t	dmacount;	/* 0x86 */
+	uint64_t	chansts;	/* 0x88 */
+	uint64_t	chainaddr;	/* 0x90 */
+	uint64_t	chancmp;	/* 0x98 */
+	uint8_t		reserved2[0x8];	/* 0xA0 */
+	uint32_t	chanerr;	/* 0xA8 */
+	uint32_t	chanerrmask;	/* 0xAC */
+} __attribute__((packed));
+
+#define RTE_IOAT_CHANCMD_RESET			0x20
+#define RTE_IOAT_CHANCMD_SUSPEND		0x04
+
+#define RTE_IOAT_CHANSTS_STATUS		0x7ULL
+#define RTE_IOAT_CHANSTS_ACTIVE		0x0
+#define RTE_IOAT_CHANSTS_IDLE			0x1
+#define RTE_IOAT_CHANSTS_SUSPENDED		0x2
+#define RTE_IOAT_CHANSTS_HALTED		0x3
+#define RTE_IOAT_CHANSTS_ARMED			0x4
+
+#define RTE_IOAT_CHANSTS_UNAFFILIATED_ERROR	0x8ULL
+#define RTE_IOAT_CHANSTS_SOFT_ERROR		0x10ULL
+
+#define RTE_IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK	(~0x3FULL)
+
+#define RTE_IOAT_CHANCMP_ALIGN			8	/* CHANCMP address must be 64-bit aligned */
+
+struct rte_ioat_generic_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t reserved2: 1;
+			uint32_t src_page_break: 1;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t op_specific[4];
+};
+
+struct rte_ioat_dma_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t null: 1;
+			uint32_t src_page_break: 1;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+#define RTE_IOAT_OP_COPY 0x00
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t reserved;
+	uint64_t reserved2;
+	uint64_t user1;
+	uint64_t user2;
+};
+
+struct rte_ioat_fill_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t reserved: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t reserved2: 2;
+			uint32_t dest_page_break: 1;
+			uint32_t bundle: 1;
+			uint32_t reserved3: 15;
+#define RTE_IOAT_OP_FILL 0x01
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_data;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t reserved;
+	uint64_t next_dest_addr;
+	uint64_t user1;
+	uint64_t user2;
+};
+
+struct rte_ioat_xor_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_count: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t reserved: 13;
+#define RTE_IOAT_OP_XOR 0x87
+#define RTE_IOAT_OP_XOR_VAL 0x88
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t dest_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t src_addr3;
+	uint64_t src_addr4;
+	uint64_t src_addr5;
+};
+
+struct rte_ioat_xor_ext_hw_desc {
+	uint64_t src_addr6;
+	uint64_t src_addr7;
+	uint64_t src_addr8;
+	uint64_t next;
+	uint64_t reserved[4];
+};
+
+struct rte_ioat_pq_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_count: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t p_disable: 1;
+			uint32_t q_disable: 1;
+			uint32_t reserved: 11;
+#define RTE_IOAT_OP_PQ 0x89
+#define RTE_IOAT_OP_PQ_VAL 0x8a
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t p_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t src_addr3;
+	uint8_t  coef[8];
+	uint64_t q_addr;
+};
+
+struct rte_ioat_pq_ext_hw_desc {
+	uint64_t src_addr4;
+	uint64_t src_addr5;
+	uint64_t src_addr6;
+	uint64_t next;
+	uint64_t src_addr7;
+	uint64_t src_addr8;
+	uint64_t reserved[2];
+};
+
+struct rte_ioat_pq_update_hw_desc {
+	uint32_t size;
+	union {
+		uint32_t control_raw;
+		struct {
+			uint32_t int_enable: 1;
+			uint32_t src_snoop_disable: 1;
+			uint32_t dest_snoop_disable: 1;
+			uint32_t completion_update: 1;
+			uint32_t fence: 1;
+			uint32_t src_cnt: 3;
+			uint32_t bundle: 1;
+			uint32_t dest_dca: 1;
+			uint32_t hint: 1;
+			uint32_t p_disable: 1;
+			uint32_t q_disable: 1;
+			uint32_t reserved: 3;
+			uint32_t coef: 8;
+#define RTE_IOAT_OP_PQ_UP 0x8b
+			uint32_t op: 8;
+		} control;
+	} u;
+	uint64_t src_addr;
+	uint64_t p_addr;
+	uint64_t next;
+	uint64_t src_addr2;
+	uint64_t p_src;
+	uint64_t q_src;
+	uint64_t q_addr;
+};
+
+struct rte_ioat_raw_hw_desc {
+	uint64_t field[8];
+};
+
+union rte_ioat_hw_desc {
+	struct rte_ioat_raw_hw_desc raw;
+	struct rte_ioat_generic_hw_desc generic;
+	struct rte_ioat_dma_hw_desc dma;
+	struct rte_ioat_fill_hw_desc fill;
+	struct rte_ioat_xor_hw_desc xor_desc;
+	struct rte_ioat_xor_ext_hw_desc xor_ext;
+	struct rte_ioat_pq_hw_desc pq;
+	struct rte_ioat_pq_ext_hw_desc pq_ext;
+	struct rte_ioat_pq_update_hw_desc pq_update;
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_IOAT_SPEC_H */
-- 
2.21.0

[dpdk-dev] [PATCH v5 5/9] raw/ioat: create device on probe and destroy on release

[Bruce Richardson]

Add the create/destroy driver functions so that we can actually allocate
a rawdev and destroy it when done. No rawdev API functions are actually
implemented at this point.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
---
V4:
* changed error code for pmd allocation to ENOMEM instead of EINVAL
* Use contiguous memory for device structure allocation
---
 doc/guides/rawdevs/ioat_rawdev.rst |  11 +++
 drivers/raw/ioat/ioat_rawdev.c     | 112 ++++++++++++++++++++++++++++-
 drivers/raw/ioat/rte_ioat_rawdev.h |  24 +++++++
 3 files changed, 144 insertions(+), 3 deletions(-)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 1a4b0e03e..4b7fe8a8f 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -72,3 +72,14 @@ them to a suitable DPDK-supported kernel driver. When querying the status
 of the devices, they will appear under the category of "Misc (rawdev)
 devices", i.e. the command ``dpdk-devbind.py --status-dev misc`` can be
 used to see the state of those devices alone.
+
+Device Probing and Initialization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once bound to a suitable kernel device driver, the HW devices will be found
+as part of the PCI scan done at application initialization time. No vdev
+parameters need to be passed to create or initialize the device.
+
+Once probed successfully, the device will appear as a ``rawdev``, that is a
+"raw device type" inside DPDK, and can be accessed using APIs from the
+``rte_rawdev`` library.
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index 07f6d8dfa..86b5b2a77 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -2,6 +2,7 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <rte_cycles.h>
 #include <rte_bus_pci.h>
 #include <rte_rawdev_pmd.h>
 
@@ -36,15 +37,120 @@ static struct rte_pci_driver ioat_pmd_drv;
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
-	RTE_SET_USED(name);
-	RTE_SET_USED(dev);
+	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+	};
+
+	struct rte_rawdev *rawdev = NULL;
+	struct rte_ioat_rawdev *ioat = NULL;
+	const struct rte_memzone *mz = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	int ret = 0;
+	int retry = 0;
+
+	if (!name) {
+		IOAT_PMD_ERR("Invalid name of the device!");
+		ret = -EINVAL;
+		goto cleanup;
+	}
+
+	/* Allocate device structure */
+	rawdev = rte_rawdev_pmd_allocate(name, sizeof(struct rte_ioat_rawdev),
+					 dev->device.numa_node);
+	if (rawdev == NULL) {
+		IOAT_PMD_ERR("Unable to allocate raw device");
+		ret = -ENOMEM;
+		goto cleanup;
+	}
+
+	snprintf(mz_name, sizeof(mz_name), "rawdev%u_private", rawdev->dev_id);
+	mz = rte_memzone_reserve(mz_name, sizeof(struct rte_ioat_rawdev),
+			dev->device.numa_node, RTE_MEMZONE_IOVA_CONTIG);
+	if (mz == NULL) {
+		IOAT_PMD_ERR("Unable to reserve memzone for private data\n");
+		ret = -ENOMEM;
+		goto cleanup;
+	}
+
+	rawdev->dev_private = mz->addr;
+	rawdev->dev_ops = &ioat_rawdev_ops;
+	rawdev->device = &dev->device;
+	rawdev->driver_name = dev->device.driver->name;
+
+	ioat = rawdev->dev_private;
+	ioat->rawdev = rawdev;
+	ioat->mz = mz;
+	ioat->regs = dev->mem_resource[0].addr;
+	ioat->ring_size = 0;
+	ioat->desc_ring = NULL;
+	ioat->status_addr = ioat->mz->iova +
+			offsetof(struct rte_ioat_rawdev, status);
+
+	/* do device initialization - reset and set error behaviour */
+	if (ioat->regs->chancnt != 1)
+		IOAT_PMD_ERR("%s: Channel count == %d\n", __func__,
+				ioat->regs->chancnt);
+
+	if (ioat->regs->chanctrl & 0x100) { /* locked by someone else */
+		IOAT_PMD_WARN("%s: Channel appears locked\n", __func__);
+		ioat->regs->chanctrl = 0;
+	}
+
+	ioat->regs->chancmd = RTE_IOAT_CHANCMD_SUSPEND;
+	rte_delay_ms(1);
+	ioat->regs->chancmd = RTE_IOAT_CHANCMD_RESET;
+	rte_delay_ms(1);
+	while (ioat->regs->chancmd & RTE_IOAT_CHANCMD_RESET) {
+		ioat->regs->chainaddr = 0;
+		rte_delay_ms(1);
+		if (++retry >= 200) {
+			IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=0x%"PRIx8", CHANSTS=0x%"PRIx64", CHANERR=0x%"PRIx32"\n",
+					__func__,
+					ioat->regs->chancmd,
+					ioat->regs->chansts,
+					ioat->regs->chanerr);
+			ret = -EIO;
+		}
+	}
+	ioat->regs->chanctrl = RTE_IOAT_CHANCTRL_ANY_ERR_ABORT_EN |
+			RTE_IOAT_CHANCTRL_ERR_COMPLETION_EN;
+
 	return 0;
+
+cleanup:
+	if (rawdev)
+		rte_rawdev_pmd_release(rawdev);
+
+	return ret;
 }
 
 static int
 ioat_rawdev_destroy(const char *name)
 {
-	RTE_SET_USED(name);
+	int ret;
+	struct rte_rawdev *rdev;
+
+	if (!name) {
+		IOAT_PMD_ERR("Invalid device name");
+		return -EINVAL;
+	}
+
+	rdev = rte_rawdev_pmd_get_named_dev(name);
+	if (!rdev) {
+		IOAT_PMD_ERR("Invalid device name (%s)", name);
+		return -EINVAL;
+	}
+
+	if (rdev->dev_private != NULL) {
+		struct rte_ioat_rawdev *ioat = rdev->dev_private;
+		rdev->dev_private = NULL;
+		rte_memzone_free(ioat->mz);
+	}
+
+	/* rte_rawdev_close is called by pmd_release */
+	ret = rte_rawdev_pmd_release(rdev);
+	if (ret)
+		IOAT_PMD_DEBUG("Device cleanup failed");
+
 	return 0;
 }
 
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index e77406403..f83d26ef0 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -14,6 +14,10 @@
  * @b EXPERIMENTAL: these structures and APIs may change without prior notice
  */
 
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_ioat_spec.h>
+
 /** Name of the device driver */
 #define IOAT_PMD_RAWDEV_NAME rawdev_ioat
 /** String reported as the device driver name by rte_rawdev_info_get() */
@@ -21,4 +25,24 @@
 /** Name used to adjust the log level for this driver */
 #define IOAT_PMD_LOG_NAME "rawdev.ioat"
 
+/**
+ * @internal
+ * Structure representing a device instance
+ */
+struct rte_ioat_rawdev {
+	struct rte_rawdev *rawdev;
+	const struct rte_memzone *mz;
+	const struct rte_memzone *desc_mz;
+
+	volatile struct rte_ioat_registers *regs;
+	phys_addr_t status_addr;
+	phys_addr_t ring_addr;
+
+	unsigned short ring_size;
+	struct rte_ioat_generic_hw_desc *desc_ring;
+
+	/* to report completions, the device will write status back here */
+	volatile uint64_t status __rte_cache_aligned;
+};
+
 #endif
-- 
2.21.0

Re: [dpdk-dev] [PATCH v5 5/9] raw/ioat: create device on probe and destroy on release

[Hu, Jiayu]

Acked-by: Jiayu Hu <jiayu.hu@intel.com>

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Tuesday, July 2, 2019 10:12 PM
> To: dev@dpdk.org
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Subject: [PATCH v5 5/9] raw/ioat: create device on probe and destroy on
> release
> 
> Add the create/destroy driver functions so that we can actually allocate
> a rawdev and destroy it when done. No rawdev API functions are actually
> implemented at this point.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> ---

[dpdk-dev] [PATCH v5 6/9] raw/ioat: add device info function

[Bruce Richardson]

Add in the "info_get" function to the driver, to allow us to query the
device. This allows us to have the unit test pick up the presence of
supported hardware or not.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
---

V2: Test case is placed in driver self-test routine
V4: Use TEST_SUCCESS in place of 0, when returning from test case.
    Minor documentation updates.
---
 app/test/test_rawdev.c             | 11 ++++++++++
 doc/guides/rawdevs/ioat_rawdev.rst | 34 ++++++++++++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev.c     | 11 ++++++++++
 drivers/raw/ioat/rte_ioat_rawdev.h | 11 ++++++++++
 4 files changed, 67 insertions(+)

diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
index 88549fb61..623117af9 100644
--- a/app/test/test_rawdev.c
+++ b/app/test/test_rawdev.c
@@ -29,6 +29,17 @@ REGISTER_TEST_COMMAND(rawdev_autotest, test_rawdev_selftest_skeleton);
 static int
 test_rawdev_selftest_ioat(void)
 {
+	const int count = rte_rawdev_count();
+	int i;
+
+	for (i = 0; i < count; i++) {
+		struct rte_rawdev_info info = { .dev_private = NULL };
+		if (rte_rawdev_info_get(i, &info) == 0 &&
+				strstr(info.driver_name, "ioat") != NULL)
+			return TEST_SUCCESS;
+	}
+
+	printf("No IOAT rawdev found, skipping tests\n");
 	return TEST_SKIPPED;
 }
 
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 4b7fe8a8f..efed9c64c 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -83,3 +83,37 @@ parameters need to be passed to create or initialize the device.
 Once probed successfully, the device will appear as a ``rawdev``, that is a
 "raw device type" inside DPDK, and can be accessed using APIs from the
 ``rte_rawdev`` library.
+
+Using IOAT Rawdev Devices
+--------------------------
+
+To use the devices from an application, the rawdev API can be used, along
+with definitions taken from the device-specific header file
+``rte_ioat_rawdev.h``. This header is needed to get the definition of
+structure parameters used by some of the rawdev APIs for IOAT rawdev
+devices, as well as providing key functions for using the device for memory
+copies.
+
+Getting Device Information
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Basic information about each rawdev device can be queried using the
+``rte_rawdev_info_get()`` API. For most applications, this API will be
+needed to verify that the rawdev in question is of the expected type. For
+example, the following code snippet can be used to identify an IOAT
+rawdev device for use by an application:
+
+.. code-block:: C
+
+        for (i = 0; i < count && !found; i++) {
+                struct rte_rawdev_info info = { .dev_private = NULL };
+                found = (rte_rawdev_info_get(i, &info) == 0 &&
+                                strcmp(info.driver_name,
+                                                IOAT_PMD_RAWDEV_NAME_STR) == 0);
+        }
+
+When calling the ``rte_rawdev_info_get()`` API for an IOAT rawdev device,
+the ``dev_private`` field in the ``rte_rawdev_info`` struct should either
+be NULL, or else be set to point to a structure of type
+``rte_ioat_rawdev_config``, in which case the size of the configured device
+input ring will be returned in that structure.
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index 86b5b2a77..2bfe2544d 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -34,10 +34,21 @@ static struct rte_pci_driver ioat_pmd_drv;
 #define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
 #define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
 
+static void
+ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
+{
+	struct rte_ioat_rawdev_config *cfg = dev_info;
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+
+	if (cfg != NULL)
+		cfg->ring_size = ioat->ring_size;
+}
+
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
 	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+			.dev_info_get = ioat_dev_info_get,
 	};
 
 	struct rte_rawdev *rawdev = NULL;
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index f83d26ef0..1d0dbdd2e 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -25,6 +25,17 @@
 /** Name used to adjust the log level for this driver */
 #define IOAT_PMD_LOG_NAME "rawdev.ioat"
 
+/**
+ * Configuration structure for an ioat rawdev instance
+ *
+ * This structure is to be passed as the ".dev_private" parameter when
+ * calling the rte_rawdev_get_info() and rte_rawdev_configure() APIs on
+ * an ioat rawdev instance.
+ */
+struct rte_ioat_rawdev_config {
+	unsigned short ring_size;
+};
+
 /**
  * @internal
  * Structure representing a device instance
-- 
2.21.0

Re: [dpdk-dev] [PATCH v5 6/9] raw/ioat: add device info function

[Hu, Jiayu]

Acked-by: Jiayu Hu <jiayu.hu@intel.com>

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Tuesday, July 2, 2019 10:12 PM
> To: dev@dpdk.org
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Subject: [PATCH v5 6/9] raw/ioat: add device info function
> 
> Add in the "info_get" function to the driver, to allow us to query the
> device. This allows us to have the unit test pick up the presence of
> supported hardware or not.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> ---
> 
> V2: Test case is placed in driver self-test routine
> V4: Use TEST_SUCCESS in place of 0, when returning from test case.

[dpdk-dev] [PATCH v5 7/9] raw/ioat: add configure, start and stop functions

[Bruce Richardson]

Allow initializing a driver instance. Include selftest to validate these
functions.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
---

V4: Guarantee correct SUCCESS/FAILURE return values from test function
    Use memzone rather than malloc for ring allocation so we can guarantee
    contiguous memory.
V3: don't add a new descriptor format struct, reuse from rte_ioat_spec.h
V2: test cases placed in self-test routine
---
 app/test/test_rawdev.c              |  3 +-
 doc/guides/rawdevs/ioat_rawdev.rst  | 32 +++++++++++
 drivers/raw/ioat/Makefile           |  1 +
 drivers/raw/ioat/ioat_rawdev.c      | 82 +++++++++++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev_test.c | 41 +++++++++++++++
 drivers/raw/ioat/meson.build        |  3 +-
 drivers/raw/ioat/rte_ioat_rawdev.h  |  2 +
 7 files changed, 162 insertions(+), 2 deletions(-)
 create mode 100644 drivers/raw/ioat/ioat_rawdev_test.c

diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
index 623117af9..524a9d5f3 100644
--- a/app/test/test_rawdev.c
+++ b/app/test/test_rawdev.c
@@ -36,7 +36,8 @@ test_rawdev_selftest_ioat(void)
 		struct rte_rawdev_info info = { .dev_private = NULL };
 		if (rte_rawdev_info_get(i, &info) == 0 &&
 				strstr(info.driver_name, "ioat") != NULL)
-			return TEST_SUCCESS;
+			return rte_rawdev_selftest(i) == 0 ?
+					TEST_SUCCESS : TEST_FAILED;
 	}
 
 	printf("No IOAT rawdev found, skipping tests\n");
diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index efed9c64c..a0594d2cb 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -117,3 +117,35 @@ the ``dev_private`` field in the ``rte_rawdev_info`` struct should either
 be NULL, or else be set to point to a structure of type
 ``rte_ioat_rawdev_config``, in which case the size of the configured device
 input ring will be returned in that structure.
+
+Device Configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+Configuring an IOAT rawdev device is done using the
+``rte_rawdev_configure()`` API, which takes the same structure parameters
+as the, previously referenced, ``rte_rawdev_info_get()`` API. The main
+difference is that, because the parameter is used as input rather than
+output, the ``dev_private`` structure element cannot be NULL, and must
+point to a valid ``rte_ioat_rawdev_config`` structure, containing the ring
+size to be used by the device. The ring size must be a power of two,
+between 64 and 4096.
+
+The following code shows how the device is configured in
+``test_ioat_rawdev.c``:
+
+.. code-block:: C
+
+   #define IOAT_TEST_RINGSIZE 512
+        struct rte_ioat_rawdev_config p = { .ring_size = -1 };
+        struct rte_rawdev_info info = { .dev_private = &p };
+
+        /* ... */
+
+        p.ring_size = IOAT_TEST_RINGSIZE;
+        if (rte_rawdev_configure(dev_id, &info) != 0) {
+                printf("Error with rte_rawdev_configure()\n");
+                return -1;
+        }
+
+Once configured, the device can then be made ready for use by calling the
+``rte_rawdev_start()`` API.
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
index 1e10938f3..b1af9c666 100644
--- a/drivers/raw/ioat/Makefile
+++ b/drivers/raw/ioat/Makefile
@@ -21,6 +21,7 @@ EXPORT_MAP := rte_pmd_ioat_version.map
 
 # library source files
 SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev_test.c
 
 # export include files
 SYMLINK-y-include += rte_ioat_rawdev.h
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index 2bfe2544d..0c91b3579 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -34,6 +34,81 @@ static struct rte_pci_driver ioat_pmd_drv;
 #define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
 #define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
 
+#define DESC_SZ sizeof(struct rte_ioat_generic_hw_desc)
+#define COMPLETION_SZ sizeof(__m128i)
+
+static int
+ioat_dev_configure(const struct rte_rawdev *dev, rte_rawdev_obj_t config)
+{
+	struct rte_ioat_rawdev_config *params = config;
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	unsigned short i;
+
+	if (dev->started)
+		return -EBUSY;
+
+	if (params == NULL)
+		return -EINVAL;
+
+	if (params->ring_size > 4096 || params->ring_size < 64 ||
+			!rte_is_power_of_2(params->ring_size))
+		return -EINVAL;
+
+	ioat->ring_size = params->ring_size;
+	if (ioat->desc_ring != NULL) {
+		rte_memzone_free(ioat->desc_mz);
+		ioat->desc_ring = NULL;
+		ioat->desc_mz = NULL;
+	}
+
+	/* allocate one block of memory for both descriptors
+	 * and completion handles.
+	 */
+	snprintf(mz_name, sizeof(mz_name), "rawdev%u_desc_ring", dev->dev_id);
+	ioat->desc_mz = rte_memzone_reserve(mz_name,
+			(DESC_SZ + COMPLETION_SZ) * ioat->ring_size,
+			dev->device->numa_node, RTE_MEMZONE_IOVA_CONTIG);
+	if (ioat->desc_mz == NULL)
+		return -ENOMEM;
+	ioat->desc_ring = ioat->desc_mz->addr;
+	ioat->hdls = (void *)&ioat->desc_ring[ioat->ring_size];
+
+	ioat->ring_addr = ioat->desc_mz->iova;
+
+	/* configure descriptor ring - each one points to next */
+	for (i = 0; i < ioat->ring_size; i++) {
+		ioat->desc_ring[i].next = ioat->ring_addr +
+				(((i + 1) % ioat->ring_size) * DESC_SZ);
+	}
+
+	return 0;
+}
+
+static int
+ioat_dev_start(struct rte_rawdev *dev)
+{
+	struct rte_ioat_rawdev *ioat = dev->dev_private;
+
+	if (ioat->ring_size == 0 || ioat->desc_ring == NULL)
+		return -EBUSY;
+
+	/* inform hardware of where the descriptor ring is */
+	ioat->regs->chainaddr = ioat->ring_addr;
+	/* inform hardware of where to write the status/completions */
+	ioat->regs->chancmp = ioat->status_addr;
+
+	/* prime the status register to be set to the last element */
+	ioat->status =  ioat->ring_addr + ((ioat->ring_size - 1) * DESC_SZ);
+	return 0;
+}
+
+static void
+ioat_dev_stop(struct rte_rawdev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
 static void
 ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
 {
@@ -44,11 +119,17 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
 		cfg->ring_size = ioat->ring_size;
 }
 
+extern int ioat_rawdev_test(uint16_t dev_id);
+
 static int
 ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 {
 	static const struct rte_rawdev_ops ioat_rawdev_ops = {
+			.dev_configure = ioat_dev_configure,
+			.dev_start = ioat_dev_start,
+			.dev_stop = ioat_dev_stop,
 			.dev_info_get = ioat_dev_info_get,
+			.dev_selftest = ioat_rawdev_test,
 	};
 
 	struct rte_rawdev *rawdev = NULL;
@@ -154,6 +235,7 @@ ioat_rawdev_destroy(const char *name)
 	if (rdev->dev_private != NULL) {
 		struct rte_ioat_rawdev *ioat = rdev->dev_private;
 		rdev->dev_private = NULL;
+		rte_memzone_free(ioat->desc_mz);
 		rte_memzone_free(ioat->mz);
 	}
 
diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c
new file mode 100644
index 000000000..5375da26c
--- /dev/null
+++ b/drivers/raw/ioat/ioat_rawdev_test.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include "rte_rawdev.h"
+#include "rte_ioat_rawdev.h"
+
+int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */
+
+int
+ioat_rawdev_test(uint16_t dev_id)
+{
+#define IOAT_TEST_RINGSIZE 512
+	struct rte_ioat_rawdev_config p = { .ring_size = -1 };
+	struct rte_rawdev_info info = { .dev_private = &p };
+
+	rte_rawdev_info_get(dev_id, &info);
+	if (p.ring_size != 0) {
+		printf("Error, initial ring size is non-zero (%d)\n",
+				(int)p.ring_size);
+		return -1;
+	}
+
+	p.ring_size = IOAT_TEST_RINGSIZE;
+	if (rte_rawdev_configure(dev_id, &info) != 0) {
+		printf("Error with rte_rawdev_configure()\n");
+		return -1;
+	}
+	rte_rawdev_info_get(dev_id, &info);
+	if (p.ring_size != IOAT_TEST_RINGSIZE) {
+		printf("Error, ring size is not %d (%d)\n",
+				IOAT_TEST_RINGSIZE, (int)p.ring_size);
+		return -1;
+	}
+
+	if (rte_rawdev_start(dev_id) != 0) {
+		printf("Error with rte_rawdev_start()\n");
+		return -1;
+	}
+	return 0;
+}
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
index ca23e23fc..40fff6654 100644
--- a/drivers/raw/ioat/meson.build
+++ b/drivers/raw/ioat/meson.build
@@ -2,7 +2,8 @@
 # Copyright 2019 Intel Corporation
 
 build = dpdk_conf.has('RTE_ARCH_X86')
-sources = files('ioat_rawdev.c')
+sources = files('ioat_rawdev.c',
+		'ioat_rawdev_test.c')
 deps += ['rawdev', 'bus_pci']
 
 install_headers('rte_ioat_rawdev.h',
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index 1d0dbdd2e..f2cf98cd9 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -14,6 +14,7 @@
  * @b EXPERIMENTAL: these structures and APIs may change without prior notice
  */
 
+#include <x86intrin.h>
 #include <rte_memory.h>
 #include <rte_memzone.h>
 #include <rte_ioat_spec.h>
@@ -51,6 +52,7 @@ struct rte_ioat_rawdev {
 
 	unsigned short ring_size;
 	struct rte_ioat_generic_hw_desc *desc_ring;
+	__m128i *hdls; /* completion handles for returning to user */
 
 	/* to report completions, the device will write status back here */
 	volatile uint64_t status __rte_cache_aligned;
-- 
2.21.0

Re: [dpdk-dev] [PATCH v5 7/9] raw/ioat: add configure, start and stop functions

[Hu, Jiayu]

Acked-by: Jiayu Hu <jiayu.hu@intel.com>

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Tuesday, July 2, 2019 10:12 PM
> To: dev@dpdk.org
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Subject: [PATCH v5 7/9] raw/ioat: add configure, start and stop functions
> 
> Allow initializing a driver instance. Include selftest to validate these
> functions.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> ---
> 

Re: [dpdk-dev] [PATCH v5 7/9] raw/ioat: add configure, start and stop functions

[Aaron Conole]

Bruce Richardson <bruce.richardson@intel.com> writes:

> Allow initializing a driver instance. Include selftest to validate these
> functions.
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> ---
>
> V4: Guarantee correct SUCCESS/FAILURE return values from test function
>     Use memzone rather than malloc for ring allocation so we can guarantee
>     contiguous memory.
> V3: don't add a new descriptor format struct, reuse from rte_ioat_spec.h
> V2: test cases placed in self-test routine
> ---
>  app/test/test_rawdev.c              |  3 +-
>  doc/guides/rawdevs/ioat_rawdev.rst  | 32 +++++++++++
>  drivers/raw/ioat/Makefile           |  1 +
>  drivers/raw/ioat/ioat_rawdev.c      | 82 +++++++++++++++++++++++++++++
>  drivers/raw/ioat/ioat_rawdev_test.c | 41 +++++++++++++++
>  drivers/raw/ioat/meson.build        |  3 +-
>  drivers/raw/ioat/rte_ioat_rawdev.h  |  2 +
>  7 files changed, 162 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/raw/ioat/ioat_rawdev_test.c
>
> diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c
> index 623117af9..524a9d5f3 100644
> --- a/app/test/test_rawdev.c
> +++ b/app/test/test_rawdev.c
> @@ -36,7 +36,8 @@ test_rawdev_selftest_ioat(void)
>  		struct rte_rawdev_info info = { .dev_private = NULL };
>  		if (rte_rawdev_info_get(i, &info) == 0 &&
>  				strstr(info.driver_name, "ioat") != NULL)
> -			return TEST_SUCCESS;
> +			return rte_rawdev_selftest(i) == 0 ?
> +					TEST_SUCCESS : TEST_FAILED;
>  	}
>  
>  	printf("No IOAT rawdev found, skipping tests\n");
> diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
> index efed9c64c..a0594d2cb 100644
> --- a/doc/guides/rawdevs/ioat_rawdev.rst
> +++ b/doc/guides/rawdevs/ioat_rawdev.rst
> @@ -117,3 +117,35 @@ the ``dev_private`` field in the ``rte_rawdev_info`` struct should either
>  be NULL, or else be set to point to a structure of type
>  ``rte_ioat_rawdev_config``, in which case the size of the configured device
>  input ring will be returned in that structure.
> +
> +Device Configuration
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +Configuring an IOAT rawdev device is done using the
> +``rte_rawdev_configure()`` API, which takes the same structure parameters
> +as the, previously referenced, ``rte_rawdev_info_get()`` API. The main
> +difference is that, because the parameter is used as input rather than
> +output, the ``dev_private`` structure element cannot be NULL, and must
> +point to a valid ``rte_ioat_rawdev_config`` structure, containing the ring
> +size to be used by the device. The ring size must be a power of two,
> +between 64 and 4096.
> +
> +The following code shows how the device is configured in
> +``test_ioat_rawdev.c``:
> +
> +.. code-block:: C
> +
> +   #define IOAT_TEST_RINGSIZE 512
> +        struct rte_ioat_rawdev_config p = { .ring_size = -1 };
> +        struct rte_rawdev_info info = { .dev_private = &p };
> +
> +        /* ... */
> +
> +        p.ring_size = IOAT_TEST_RINGSIZE;
> +        if (rte_rawdev_configure(dev_id, &info) != 0) {
> +                printf("Error with rte_rawdev_configure()\n");
> +                return -1;
> +        }
> +
> +Once configured, the device can then be made ready for use by calling the
> +``rte_rawdev_start()`` API.
> diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
> index 1e10938f3..b1af9c666 100644
> --- a/drivers/raw/ioat/Makefile
> +++ b/drivers/raw/ioat/Makefile
> @@ -21,6 +21,7 @@ EXPORT_MAP := rte_pmd_ioat_version.map
>  
>  # library source files
>  SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c
> +SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev_test.c
>  
>  # export include files
>  SYMLINK-y-include += rte_ioat_rawdev.h
> diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
> index 2bfe2544d..0c91b3579 100644
> --- a/drivers/raw/ioat/ioat_rawdev.c
> +++ b/drivers/raw/ioat/ioat_rawdev.c
> @@ -34,6 +34,81 @@ static struct rte_pci_driver ioat_pmd_drv;
>  #define IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args)
>  #define IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
>  
> +#define DESC_SZ sizeof(struct rte_ioat_generic_hw_desc)
> +#define COMPLETION_SZ sizeof(__m128i)
> +
> +static int
> +ioat_dev_configure(const struct rte_rawdev *dev, rte_rawdev_obj_t config)
> +{
> +	struct rte_ioat_rawdev_config *params = config;
> +	struct rte_ioat_rawdev *ioat = dev->dev_private;
> +	char mz_name[RTE_MEMZONE_NAMESIZE];
> +	unsigned short i;
> +
> +	if (dev->started)
> +		return -EBUSY;
> +
> +	if (params == NULL)
> +		return -EINVAL;
> +
> +	if (params->ring_size > 4096 || params->ring_size < 64 ||
> +			!rte_is_power_of_2(params->ring_size))
> +		return -EINVAL;
> +
> +	ioat->ring_size = params->ring_size;
> +	if (ioat->desc_ring != NULL) {
> +		rte_memzone_free(ioat->desc_mz);
> +		ioat->desc_ring = NULL;
> +		ioat->desc_mz = NULL;
> +	}
> +
> +	/* allocate one block of memory for both descriptors
> +	 * and completion handles.
> +	 */
> +	snprintf(mz_name, sizeof(mz_name), "rawdev%u_desc_ring", dev->dev_id);
> +	ioat->desc_mz = rte_memzone_reserve(mz_name,
> +			(DESC_SZ + COMPLETION_SZ) * ioat->ring_size,
> +			dev->device->numa_node, RTE_MEMZONE_IOVA_CONTIG);
> +	if (ioat->desc_mz == NULL)
> +		return -ENOMEM;
> +	ioat->desc_ring = ioat->desc_mz->addr;
> +	ioat->hdls = (void *)&ioat->desc_ring[ioat->ring_size];
> +
> +	ioat->ring_addr = ioat->desc_mz->iova;
> +
> +	/* configure descriptor ring - each one points to next */
> +	for (i = 0; i < ioat->ring_size; i++) {
> +		ioat->desc_ring[i].next = ioat->ring_addr +
> +				(((i + 1) % ioat->ring_size) * DESC_SZ);
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +ioat_dev_start(struct rte_rawdev *dev)
> +{
> +	struct rte_ioat_rawdev *ioat = dev->dev_private;
> +
> +	if (ioat->ring_size == 0 || ioat->desc_ring == NULL)
> +		return -EBUSY;
> +
> +	/* inform hardware of where the descriptor ring is */
> +	ioat->regs->chainaddr = ioat->ring_addr;
> +	/* inform hardware of where to write the status/completions */
> +	ioat->regs->chancmp = ioat->status_addr;
> +
> +	/* prime the status register to be set to the last element */
> +	ioat->status =  ioat->ring_addr + ((ioat->ring_size - 1) * DESC_SZ);
> +	return 0;
> +}
> +
> +static void
> +ioat_dev_stop(struct rte_rawdev *dev)
> +{
> +	RTE_SET_USED(dev);
> +}
> +
>  static void
>  ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
>  {
> @@ -44,11 +119,17 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
>  		cfg->ring_size = ioat->ring_size;
>  }
>  
> +extern int ioat_rawdev_test(uint16_t dev_id);
> +

This signature doesn't match with the rte_rawdev_ops definition (which
should be int(*func)(void)).

It will generate an error as such:

../drivers/raw/ioat/ioat_rawdev.c:176:20: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
    .dev_selftest = ioat_rawdev_test,
                    ^
../drivers/raw/ioat/ioat_rawdev.c:176:20: note: (near initialization for ‘ioat_rawdev_ops.dev_selftest’)

>  static int
>  ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
>  {
>  	static const struct rte_rawdev_ops ioat_rawdev_ops = {
> +			.dev_configure = ioat_dev_configure,
> +			.dev_start = ioat_dev_start,
> +			.dev_stop = ioat_dev_stop,
>  			.dev_info_get = ioat_dev_info_get,
> +			.dev_selftest = ioat_rawdev_test,
>  	};
>  
>  	struct rte_rawdev *rawdev = NULL;
> @@ -154,6 +235,7 @@ ioat_rawdev_destroy(const char *name)
>  	if (rdev->dev_private != NULL) {
>  		struct rte_ioat_rawdev *ioat = rdev->dev_private;
>  		rdev->dev_private = NULL;
> +		rte_memzone_free(ioat->desc_mz);
>  		rte_memzone_free(ioat->mz);
>  	}
>  
> diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c
> new file mode 100644
> index 000000000..5375da26c
> --- /dev/null
> +++ b/drivers/raw/ioat/ioat_rawdev_test.c
> @@ -0,0 +1,41 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2019 Intel Corporation
> + */
> +
> +#include "rte_rawdev.h"
> +#include "rte_ioat_rawdev.h"
> +
> +int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */
> +
> +int
> +ioat_rawdev_test(uint16_t dev_id)
> +{
> +#define IOAT_TEST_RINGSIZE 512
> +	struct rte_ioat_rawdev_config p = { .ring_size = -1 };
> +	struct rte_rawdev_info info = { .dev_private = &p };
> +
> +	rte_rawdev_info_get(dev_id, &info);
> +	if (p.ring_size != 0) {
> +		printf("Error, initial ring size is non-zero (%d)\n",
> +				(int)p.ring_size);
> +		return -1;
> +	}
> +
> +	p.ring_size = IOAT_TEST_RINGSIZE;
> +	if (rte_rawdev_configure(dev_id, &info) != 0) {
> +		printf("Error with rte_rawdev_configure()\n");
> +		return -1;
> +	}
> +	rte_rawdev_info_get(dev_id, &info);
> +	if (p.ring_size != IOAT_TEST_RINGSIZE) {
> +		printf("Error, ring size is not %d (%d)\n",
> +				IOAT_TEST_RINGSIZE, (int)p.ring_size);
> +		return -1;
> +	}
> +
> +	if (rte_rawdev_start(dev_id) != 0) {
> +		printf("Error with rte_rawdev_start()\n");
> +		return -1;
> +	}
> +	return 0;
> +}
> diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
> index ca23e23fc..40fff6654 100644
> --- a/drivers/raw/ioat/meson.build
> +++ b/drivers/raw/ioat/meson.build
> @@ -2,7 +2,8 @@
>  # Copyright 2019 Intel Corporation
>  
>  build = dpdk_conf.has('RTE_ARCH_X86')
> -sources = files('ioat_rawdev.c')
> +sources = files('ioat_rawdev.c',
> +		'ioat_rawdev_test.c')
>  deps += ['rawdev', 'bus_pci']
>  
>  install_headers('rte_ioat_rawdev.h',
> diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
> index 1d0dbdd2e..f2cf98cd9 100644
> --- a/drivers/raw/ioat/rte_ioat_rawdev.h
> +++ b/drivers/raw/ioat/rte_ioat_rawdev.h
> @@ -14,6 +14,7 @@
>   * @b EXPERIMENTAL: these structures and APIs may change without prior notice
>   */
>  
> +#include <x86intrin.h>
>  #include <rte_memory.h>
>  #include <rte_memzone.h>
>  #include <rte_ioat_spec.h>
> @@ -51,6 +52,7 @@ struct rte_ioat_rawdev {
>  
>  	unsigned short ring_size;
>  	struct rte_ioat_generic_hw_desc *desc_ring;
> +	__m128i *hdls; /* completion handles for returning to user */
>  
>  	/* to report completions, the device will write status back here */
>  	volatile uint64_t status __rte_cache_aligned;

Re: [dpdk-dev] [PATCH v5 7/9] raw/ioat: add configure, start and stop functions

[Bruce Richardson]

On Wed, Jul 03, 2019 at 12:21:39PM -0400, Aaron Conole wrote:
> Bruce Richardson <bruce.richardson@intel.com> writes:
> 
> > Allow initializing a driver instance. Include selftest to validate
> > these functions.
> >
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by:
> > Anatoly Burakov <anatoly.burakov@intel.com> Tested-by: Harry van Haaren
> > <harry.van.haaren@intel.com> ---
> >
> > V4: Guarantee correct SUCCESS/FAILURE return values from test function
> > Use memzone rather than malloc for ring allocation so we can guarantee
> > contiguous memory.  V3: don't add a new descriptor format struct, reuse
> > from rte_ioat_spec.h V2: test cases placed in self-test routine ---
> > app/test/test_rawdev.c              |  3 +-
> > doc/guides/rawdevs/ioat_rawdev.rst  | 32 +++++++++++
> > drivers/raw/ioat/Makefile           |  1 +
> > drivers/raw/ioat/ioat_rawdev.c      | 82 +++++++++++++++++++++++++++++
> > drivers/raw/ioat/ioat_rawdev_test.c | 41 +++++++++++++++
> > drivers/raw/ioat/meson.build        |  3 +-
> > drivers/raw/ioat/rte_ioat_rawdev.h  |  2 + 7 files changed, 162
> > insertions(+), 2 deletions(-) create mode 100644
> > drivers/raw/ioat/ioat_rawdev_test.c
> >
> > diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c index
> > 623117af9..524a9d5f3 100644 --- a/app/test/test_rawdev.c +++
> > b/app/test/test_rawdev.c @@ -36,7 +36,8 @@
> > test_rawdev_selftest_ioat(void) struct rte_rawdev_info info = {
> > .dev_private = NULL }; if (rte_rawdev_info_get(i, &info) == 0 &&
> > strstr(info.driver_name, "ioat") != NULL) -			return
> > TEST_SUCCESS; +			return rte_rawdev_selftest(i) == 0
> > ?  +					TEST_SUCCESS : TEST_FAILED;
> > }
> >  
> >  	printf("No IOAT rawdev found, skipping tests\n"); diff --git
> >  	a/doc/guides/rawdevs/ioat_rawdev.rst
> >  	b/doc/guides/rawdevs/ioat_rawdev.rst index efed9c64c..a0594d2cb
> >  	100644 --- a/doc/guides/rawdevs/ioat_rawdev.rst +++
> >  	b/doc/guides/rawdevs/ioat_rawdev.rst @@ -117,3 +117,35 @@ the
> >  	``dev_private`` field in the ``rte_rawdev_info`` struct should
> >  	either be NULL, or else be set to point to a structure of type
> >  	``rte_ioat_rawdev_config``, in which case the size of the
> >  	configured device input ring will be returned in that structure.  +
> >  	+Device Configuration +~~~~~~~~~~~~~~~~~~~~~ + +Configuring an IOAT
> >  	rawdev device is done using the +``rte_rawdev_configure()`` API,
> >  	which takes the same structure parameters +as the, previously
> >  	referenced, ``rte_rawdev_info_get()`` API. The main +difference is
> >  	that, because the parameter is used as input rather than +output,
> >  	the ``dev_private`` structure element cannot be NULL, and must
> >  	+point to a valid ``rte_ioat_rawdev_config`` structure, containing
> >  	the ring +size to be used by the device. The ring size must be a
> >  	power of two, +between 64 and 4096.  + +The following code shows
> >  	how the device is configured in +``test_ioat_rawdev.c``: + +..
> >  	code-block:: C + +   #define IOAT_TEST_RINGSIZE 512 +        struct
> >  	rte_ioat_rawdev_config p = { .ring_size = -1 }; +        struct
> >  	rte_rawdev_info info = { .dev_private = &p }; + +        /* ... */
> >  	+ +        p.ring_size = IOAT_TEST_RINGSIZE; +        if
> >  	(rte_rawdev_configure(dev_id, &info) != 0) { +
> >  	printf("Error with rte_rawdev_configure()\n"); +
> >  	return -1; +        } + +Once configured, the device can then be
> >  	made ready for use by calling the +``rte_rawdev_start()`` API.
> >  	diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
> >  	index 1e10938f3..b1af9c666 100644 --- a/drivers/raw/ioat/Makefile
> >  	+++ b/drivers/raw/ioat/Makefile @@ -21,6 +21,7 @@ EXPORT_MAP :=
> >  	rte_pmd_ioat_version.map
> >  
> >  # library source files SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) +=
> >  ioat_rawdev.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) +=
> >  ioat_rawdev_test.c
> >  
> >  # export include files SYMLINK-y-include += rte_ioat_rawdev.h diff
> >  --git a/drivers/raw/ioat/ioat_rawdev.c
> >  b/drivers/raw/ioat/ioat_rawdev.c index 2bfe2544d..0c91b3579 100644 ---
> >  a/drivers/raw/ioat/ioat_rawdev.c +++ b/drivers/raw/ioat/ioat_rawdev.c
> >  @@ -34,6 +34,81 @@ static struct rte_pci_driver ioat_pmd_drv; #define
> >  IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args) #define
> >  IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
> >  
> > +#define DESC_SZ sizeof(struct rte_ioat_generic_hw_desc) +#define
> > COMPLETION_SZ sizeof(__m128i) + +static int +ioat_dev_configure(const
> > struct rte_rawdev *dev, rte_rawdev_obj_t config) +{ +	struct
> > rte_ioat_rawdev_config *params = config; +	struct rte_ioat_rawdev
> > *ioat = dev->dev_private; +	char mz_name[RTE_MEMZONE_NAMESIZE]; +
> > unsigned short i; + +	if (dev->started) +		return
> > -EBUSY; + +	if (params == NULL) +		return -EINVAL; + +	if
> > (params->ring_size > 4096 || params->ring_size < 64 || +
> > !rte_is_power_of_2(params->ring_size)) +		return -EINVAL; + +
> > ioat->ring_size = params->ring_size; +	if (ioat->desc_ring !=
> > NULL) { +		rte_memzone_free(ioat->desc_mz); +
> > ioat->desc_ring = NULL; +		ioat->desc_mz = NULL; +	} + +	/*
> > allocate one block of memory for both descriptors +	 * and completion
> > handles.  +	 */ +	snprintf(mz_name, sizeof(mz_name),
> > "rawdev%u_desc_ring", dev->dev_id); +	ioat->desc_mz =
> > rte_memzone_reserve(mz_name, +			(DESC_SZ +
> > COMPLETION_SZ) * ioat->ring_size, +
> > dev->device->numa_node, RTE_MEMZONE_IOVA_CONTIG); +	if (ioat->desc_mz
> > == NULL) +		return -ENOMEM; +	ioat->desc_ring =
> > ioat->desc_mz->addr; +	ioat->hdls = (void
> > *)&ioat->desc_ring[ioat->ring_size]; + +	ioat->ring_addr =
> > ioat->desc_mz->iova; + +	/* configure descriptor ring - each one
> > points to next */ +	for (i = 0; i < ioat->ring_size; i++) { +
> > ioat->desc_ring[i].next = ioat->ring_addr + +
> > (((i + 1) % ioat->ring_size) * DESC_SZ); +	} + +	return 0; +} +
> > +static int +ioat_dev_start(struct rte_rawdev *dev) +{ +	struct
> > rte_ioat_rawdev *ioat = dev->dev_private; + +	if (ioat->ring_size
> > == 0 || ioat->desc_ring == NULL) +		return -EBUSY; + +	/*
> > inform hardware of where the descriptor ring is */ +
> > ioat->regs->chainaddr = ioat->ring_addr; +	/* inform hardware of where
> > to write the status/completions */ +	ioat->regs->chancmp =
> > ioat->status_addr; + +	/* prime the status register to be set to
> > the last element */ +	ioat->status =  ioat->ring_addr +
> > ((ioat->ring_size - 1) * DESC_SZ); +	return 0; +} + +static void
> > +ioat_dev_stop(struct rte_rawdev *dev) +{ +	RTE_SET_USED(dev); +} +
> > static void ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t
> > dev_info) { @@ -44,11 +119,17 @@ ioat_dev_info_get(struct rte_rawdev
> > *dev, rte_rawdev_obj_t dev_info) cfg->ring_size = ioat->ring_size; }
> >  
> > +extern int ioat_rawdev_test(uint16_t dev_id); +
> 
> This signature doesn't match with the rte_rawdev_ops definition (which
> should be int(*func)(void)).
> 
> It will generate an error as such:
> 
> ../drivers/raw/ioat/ioat_rawdev.c:176:20: error: initialization from
> incompatible pointer type [-Werror=incompatible-pointer-types]
> .dev_selftest = ioat_rawdev_test, ^
> ../drivers/raw/ioat/ioat_rawdev.c:176:20: note: (near initialization for
> ‘ioat_rawdev_ops.dev_selftest’)
> 
That was changed by commit 863fd2930bbe ("rawdev: pass the device id as
parameter to selftest") which was part of the patchset on which this set
depended.

/Bruce

Re: [dpdk-dev] [PATCH v5 7/9] raw/ioat: add configure, start and stop functions

[Aaron Conole]

Bruce Richardson <bruce.richardson@intel.com> writes:

> On Wed, Jul 03, 2019 at 12:21:39PM -0400, Aaron Conole wrote:
>> Bruce Richardson <bruce.richardson@intel.com> writes:
>> 
>> > Allow initializing a driver instance. Include selftest to validate
>> > these functions.
>> >
>> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by:
>> > Anatoly Burakov <anatoly.burakov@intel.com> Tested-by: Harry van Haaren
>> > <harry.van.haaren@intel.com> ---
>> >
>> > V4: Guarantee correct SUCCESS/FAILURE return values from test function
>> > Use memzone rather than malloc for ring allocation so we can guarantee
>> > contiguous memory.  V3: don't add a new descriptor format struct, reuse
>> > from rte_ioat_spec.h V2: test cases placed in self-test routine ---
>> > app/test/test_rawdev.c              |  3 +-
>> > doc/guides/rawdevs/ioat_rawdev.rst  | 32 +++++++++++
>> > drivers/raw/ioat/Makefile           |  1 +
>> > drivers/raw/ioat/ioat_rawdev.c      | 82 +++++++++++++++++++++++++++++
>> > drivers/raw/ioat/ioat_rawdev_test.c | 41 +++++++++++++++
>> > drivers/raw/ioat/meson.build        |  3 +-
>> > drivers/raw/ioat/rte_ioat_rawdev.h  |  2 + 7 files changed, 162
>> > insertions(+), 2 deletions(-) create mode 100644
>> > drivers/raw/ioat/ioat_rawdev_test.c
>> >
>> > diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c index
>> > 623117af9..524a9d5f3 100644 --- a/app/test/test_rawdev.c +++
>> > b/app/test/test_rawdev.c @@ -36,7 +36,8 @@
>> > test_rawdev_selftest_ioat(void) struct rte_rawdev_info info = {
>> > .dev_private = NULL }; if (rte_rawdev_info_get(i, &info) == 0 &&
>> > strstr(info.driver_name, "ioat") != NULL) -			return
>> > TEST_SUCCESS; +			return rte_rawdev_selftest(i) == 0
>> > ?  +					TEST_SUCCESS : TEST_FAILED;
>> > }
>> >  
>> >  	printf("No IOAT rawdev found, skipping tests\n"); diff --git
>> >  	a/doc/guides/rawdevs/ioat_rawdev.rst
>> >  	b/doc/guides/rawdevs/ioat_rawdev.rst index efed9c64c..a0594d2cb
>> >  	100644 --- a/doc/guides/rawdevs/ioat_rawdev.rst +++
>> >  	b/doc/guides/rawdevs/ioat_rawdev.rst @@ -117,3 +117,35 @@ the
>> >  	``dev_private`` field in the ``rte_rawdev_info`` struct should
>> >  	either be NULL, or else be set to point to a structure of type
>> >  	``rte_ioat_rawdev_config``, in which case the size of the
>> >  	configured device input ring will be returned in that structure.  +
>> >  	+Device Configuration +~~~~~~~~~~~~~~~~~~~~~ + +Configuring an IOAT
>> >  	rawdev device is done using the +``rte_rawdev_configure()`` API,
>> >  	which takes the same structure parameters +as the, previously
>> >  	referenced, ``rte_rawdev_info_get()`` API. The main +difference is
>> >  	that, because the parameter is used as input rather than +output,
>> >  	the ``dev_private`` structure element cannot be NULL, and must
>> >  	+point to a valid ``rte_ioat_rawdev_config`` structure, containing
>> >  	the ring +size to be used by the device. The ring size must be a
>> >  	power of two, +between 64 and 4096.  + +The following code shows
>> >  	how the device is configured in +``test_ioat_rawdev.c``: + +..
>> >  	code-block:: C + +   #define IOAT_TEST_RINGSIZE 512 +        struct
>> >  	rte_ioat_rawdev_config p = { .ring_size = -1 }; +        struct
>> >  	rte_rawdev_info info = { .dev_private = &p }; + +        /* ... */
>> >  	+ +        p.ring_size = IOAT_TEST_RINGSIZE; +        if
>> >  	(rte_rawdev_configure(dev_id, &info) != 0) { +
>> >  	printf("Error with rte_rawdev_configure()\n"); +
>> >  	return -1; +        } + +Once configured, the device can then be
>> >  	made ready for use by calling the +``rte_rawdev_start()`` API.
>> >  	diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
>> >  	index 1e10938f3..b1af9c666 100644 --- a/drivers/raw/ioat/Makefile
>> >  	+++ b/drivers/raw/ioat/Makefile @@ -21,6 +21,7 @@ EXPORT_MAP :=
>> >  	rte_pmd_ioat_version.map
>> >  
>> >  # library source files SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) +=
>> >  ioat_rawdev.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) +=
>> >  ioat_rawdev_test.c
>> >  
>> >  # export include files SYMLINK-y-include += rte_ioat_rawdev.h diff
>> >  --git a/drivers/raw/ioat/ioat_rawdev.c
>> >  b/drivers/raw/ioat/ioat_rawdev.c index 2bfe2544d..0c91b3579 100644 ---
>> >  a/drivers/raw/ioat/ioat_rawdev.c +++ b/drivers/raw/ioat/ioat_rawdev.c
>> >  @@ -34,6 +34,81 @@ static struct rte_pci_driver ioat_pmd_drv; #define
>> >  IOAT_PMD_ERR(fmt, args...)    IOAT_PMD_LOG(ERR, fmt, ## args) #define
>> >  IOAT_PMD_WARN(fmt, args...)   IOAT_PMD_LOG(WARNING, fmt, ## args)
>> >  
>> > +#define DESC_SZ sizeof(struct rte_ioat_generic_hw_desc) +#define
>> > COMPLETION_SZ sizeof(__m128i) + +static int +ioat_dev_configure(const
>> > struct rte_rawdev *dev, rte_rawdev_obj_t config) +{ +	struct
>> > rte_ioat_rawdev_config *params = config; +	struct rte_ioat_rawdev
>> > *ioat = dev->dev_private; +	char mz_name[RTE_MEMZONE_NAMESIZE]; +
>> > unsigned short i; + +	if (dev->started) +		return
>> > -EBUSY; + +	if (params == NULL) +		return -EINVAL; + +	if
>> > (params->ring_size > 4096 || params->ring_size < 64 || +
>> > !rte_is_power_of_2(params->ring_size)) +		return -EINVAL; + +
>> > ioat->ring_size = params->ring_size; +	if (ioat->desc_ring !=
>> > NULL) { +		rte_memzone_free(ioat->desc_mz); +
>> > ioat->desc_ring = NULL; +		ioat->desc_mz = NULL; +	} + +	/*
>> > allocate one block of memory for both descriptors +	 * and completion
>> > handles.  +	 */ +	snprintf(mz_name, sizeof(mz_name),
>> > "rawdev%u_desc_ring", dev->dev_id); +	ioat->desc_mz =
>> > rte_memzone_reserve(mz_name, +			(DESC_SZ +
>> > COMPLETION_SZ) * ioat->ring_size, +
>> > dev->device->numa_node, RTE_MEMZONE_IOVA_CONTIG); +	if (ioat->desc_mz
>> > == NULL) +		return -ENOMEM; +	ioat->desc_ring =
>> > ioat->desc_mz->addr; +	ioat->hdls = (void
>> > *)&ioat->desc_ring[ioat->ring_size]; + +	ioat->ring_addr =
>> > ioat->desc_mz->iova; + +	/* configure descriptor ring - each one
>> > points to next */ +	for (i = 0; i < ioat->ring_size; i++) { +
>> > ioat->desc_ring[i].next = ioat->ring_addr + +
>> > (((i + 1) % ioat->ring_size) * DESC_SZ); +	} + +	return 0; +} +
>> > +static int +ioat_dev_start(struct rte_rawdev *dev) +{ +	struct
>> > rte_ioat_rawdev *ioat = dev->dev_private; + +	if (ioat->ring_size
>> > == 0 || ioat->desc_ring == NULL) +		return -EBUSY; + +	/*
>> > inform hardware of where the descriptor ring is */ +
>> > ioat->regs->chainaddr = ioat->ring_addr; +	/* inform hardware of where
>> > to write the status/completions */ +	ioat->regs->chancmp =
>> > ioat->status_addr; + +	/* prime the status register to be set to
>> > the last element */ +	ioat->status =  ioat->ring_addr +
>> > ((ioat->ring_size - 1) * DESC_SZ); +	return 0; +} + +static void
>> > +ioat_dev_stop(struct rte_rawdev *dev) +{ +	RTE_SET_USED(dev); +} +
>> > static void ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t
>> > dev_info) { @@ -44,11 +119,17 @@ ioat_dev_info_get(struct rte_rawdev
>> > *dev, rte_rawdev_obj_t dev_info) cfg->ring_size = ioat->ring_size; }
>> >  
>> > +extern int ioat_rawdev_test(uint16_t dev_id); +
>> 
>> This signature doesn't match with the rte_rawdev_ops definition (which
>> should be int(*func)(void)).
>> 
>> It will generate an error as such:
>> 
>> ../drivers/raw/ioat/ioat_rawdev.c:176:20: error: initialization from
>> incompatible pointer type [-Werror=incompatible-pointer-types]
>> .dev_selftest = ioat_rawdev_test, ^
>> ../drivers/raw/ioat/ioat_rawdev.c:176:20: note: (near initialization for
>> ‘ioat_rawdev_ops.dev_selftest’)
>> 
> That was changed by commit 863fd2930bbe ("rawdev: pass the device id as
> parameter to selftest") which was part of the patchset on which this set
> depended.

Strange - the robot didn't pick that commit up.  Was this submitted
before that commit was live?

> /Bruce

[dpdk-dev] [PATCH v5 8/9] raw/ioat: add statistics functions

[Bruce Richardson]

Add stats functions to track what is happening in the driver, and put
unit tests to check those.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
---
V4: pulled in memory leak fixes from next patch
V2: test cases moved to self-test routine
---
 doc/guides/rawdevs/ioat_rawdev.rst  | 14 +++++++++
 drivers/raw/ioat/ioat_rawdev.c      | 44 ++++++++++++++++++++++++++++
 drivers/raw/ioat/ioat_rawdev_test.c | 45 +++++++++++++++++++++++++++++
 drivers/raw/ioat/rte_ioat_rawdev.h  |  6 ++++
 4 files changed, 109 insertions(+)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index a0594d2cb..40210b3f9 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -149,3 +149,17 @@ The following code shows how the device is configured in
 
 Once configured, the device can then be made ready for use by calling the
 ``rte_rawdev_start()`` API.
+
+Querying Device Statistics
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The statistics from the IOAT rawdev device can be got via the xstats
+functions in the ``rte_rawdev`` library, i.e.
+``rte_rawdev_xstats_names_get()``, ``rte_rawdev_xstats_get()`` and
+``rte_rawdev_xstats_by_name_get``. The statistics returned for each device
+instance are:
+
+* ``failed_enqueues``
+* ``successful_enqueues``
+* ``copies_started``
+* ``copies_completed``
diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c
index 0c91b3579..d509b6606 100644
--- a/drivers/raw/ioat/ioat_rawdev.c
+++ b/drivers/raw/ioat/ioat_rawdev.c
@@ -4,6 +4,7 @@
 
 #include <rte_cycles.h>
 #include <rte_bus_pci.h>
+#include <rte_string_fns.h>
 #include <rte_rawdev_pmd.h>
 
 #include "rte_ioat_rawdev.h"
@@ -119,6 +120,47 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info)
 		cfg->ring_size = ioat->ring_size;
 }
 
+static const char * const xstat_names[] = {
+		"failed_enqueues", "successful_enqueues",
+		"copies_started", "copies_completed"
+};
+
+static int
+ioat_xstats_get(const struct rte_rawdev *dev, const unsigned int ids[],
+		uint64_t values[], unsigned int n)
+{
+	const struct rte_ioat_rawdev *ioat = dev->dev_private;
+	unsigned int i;
+
+	for (i = 0; i < n; i++) {
+		switch (ids[i]) {
+		case 0: values[i] = ioat->enqueue_failed; break;
+		case 1: values[i] = ioat->enqueued; break;
+		case 2: values[i] = ioat->started; break;
+		case 3: values[i] = ioat->completed; break;
+		default: values[i] = 0; break;
+		}
+	}
+	return n;
+}
+
+static int
+ioat_xstats_get_names(const struct rte_rawdev *dev,
+		struct rte_rawdev_xstats_name *names,
+		unsigned int size)
+{
+	unsigned int i;
+
+	RTE_SET_USED(dev);
+	if (size < RTE_DIM(xstat_names))
+		return RTE_DIM(xstat_names);
+
+	for (i = 0; i < RTE_DIM(xstat_names); i++)
+		strlcpy(names[i].name, xstat_names[i], sizeof(names[i]));
+
+	return RTE_DIM(xstat_names);
+}
+
 extern int ioat_rawdev_test(uint16_t dev_id);
 
 static int
@@ -129,6 +171,8 @@ ioat_rawdev_create(const char *name, struct rte_pci_device *dev)
 			.dev_start = ioat_dev_start,
 			.dev_stop = ioat_dev_stop,
 			.dev_info_get = ioat_dev_info_get,
+			.xstats_get = ioat_xstats_get,
+			.xstats_get_names = ioat_xstats_get_names,
 			.dev_selftest = ioat_rawdev_test,
 	};
 
diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c
index 5375da26c..ab671816d 100644
--- a/drivers/raw/ioat/ioat_rawdev_test.c
+++ b/drivers/raw/ioat/ioat_rawdev_test.c
@@ -2,6 +2,7 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <inttypes.h>
 #include "rte_rawdev.h"
 #include "rte_ioat_rawdev.h"
 
@@ -13,6 +14,11 @@ ioat_rawdev_test(uint16_t dev_id)
 #define IOAT_TEST_RINGSIZE 512
 	struct rte_ioat_rawdev_config p = { .ring_size = -1 };
 	struct rte_rawdev_info info = { .dev_private = &p };
+	struct rte_rawdev_xstats_name *snames = NULL;
+	uint64_t *stats = NULL;
+	unsigned int *ids = NULL;
+	unsigned int nb_xstats;
+	unsigned int i;
 
 	rte_rawdev_info_get(dev_id, &info);
 	if (p.ring_size != 0) {
@@ -37,5 +43,44 @@ ioat_rawdev_test(uint16_t dev_id)
 		printf("Error with rte_rawdev_start()\n");
 		return -1;
 	}
+
+	/* allocate memory for xstats names and values */
+	nb_xstats = rte_rawdev_xstats_names_get(dev_id, NULL, 0);
+
+	snames = malloc(sizeof(*snames) * nb_xstats);
+	if (snames == NULL) {
+		printf("Error allocating xstat names memory\n");
+		goto err;
+	}
+	rte_rawdev_xstats_names_get(dev_id, snames, nb_xstats);
+
+	ids = malloc(sizeof(*ids) * nb_xstats);
+	if (ids == NULL) {
+		printf("Error allocating xstat ids memory\n");
+		goto err;
+	}
+	for (i = 0; i < nb_xstats; i++)
+		ids[i] = i;
+
+	stats = malloc(sizeof(*stats) * nb_xstats);
+	if (stats == NULL) {
+		printf("Error allocating xstat memory\n");
+		goto err;
+	}
+
+	rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
+	for (i = 0; i < nb_xstats; i++)
+		printf("%s: %"PRIu64"   ", snames[i].name, stats[i]);
+	printf("\n");
+
+	free(snames);
+	free(stats);
+	free(ids);
 	return 0;
+
+err:
+	free(snames);
+	free(stats);
+	free(ids);
+	return -1;
 }
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index f2cf98cd9..d5326813c 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -54,6 +54,12 @@ struct rte_ioat_rawdev {
 	struct rte_ioat_generic_hw_desc *desc_ring;
 	__m128i *hdls; /* completion handles for returning to user */
 
+	/* some statistics for tracking, if added/changed update xstats fns*/
+	uint64_t enqueue_failed __rte_cache_aligned;
+	uint64_t enqueued;
+	uint64_t started;
+	uint64_t completed;
+
 	/* to report completions, the device will write status back here */
 	volatile uint64_t status __rte_cache_aligned;
 };
-- 
2.21.0

Re: [dpdk-dev] [PATCH v5 8/9] raw/ioat: add statistics functions

[Hu, Jiayu]

Acked-by: Jiayu Hu <jiayu.hu@intel.com>

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Tuesday, July 2, 2019 10:12 PM
> To: dev@dpdk.org
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Subject: [PATCH v5 8/9] raw/ioat: add statistics functions
> 
> Add stats functions to track what is happening in the driver, and put
> unit tests to check those.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> ---
> V4: pulled in memory leak fixes from next patch
> V2: test cases moved to self-test routine

[dpdk-dev] [PATCH v5 9/9] raw/ioat: add local API to perform copies

[Bruce Richardson]

Add local APIs to trigger data copies, and retrieve handle values once
those copies are completed. Included are unit tests to validate the data
is copies correctly.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
---

V3: updated to use descriptor format in rte_ioat_spec.h
V2: test cases moved to self-test routine
---
 doc/guides/rawdevs/ioat_rawdev.rst  | 100 ++++++++++++++++
 drivers/raw/ioat/Makefile           |   1 +
 drivers/raw/ioat/ioat_rawdev_test.c | 155 ++++++++++++++++++++++++-
 drivers/raw/ioat/meson.build        |   2 +-
 drivers/raw/ioat/rte_ioat_rawdev.h  | 169 +++++++++++++++++++++++++++-
 5 files changed, 422 insertions(+), 5 deletions(-)

diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst
index 40210b3f9..607cb5a86 100644
--- a/doc/guides/rawdevs/ioat_rawdev.rst
+++ b/doc/guides/rawdevs/ioat_rawdev.rst
@@ -150,6 +150,106 @@ The following code shows how the device is configured in
 Once configured, the device can then be made ready for use by calling the
 ``rte_rawdev_start()`` API.
 
+Performing Data Copies
+~~~~~~~~~~~~~~~~~~~~~~~
+
+To perform data copies using IOAT rawdev devices, the functions
+``rte_ioat_enqueue_copy()`` and ``rte_ioat_do_copies()`` should be used.
+Once copies have been completed, the completion will be reported back when
+the application calls ``rte_ioat_completed_copies()``.
+
+The ``rte_ioat_enqueue_copy()`` function enqueues a single copy to the
+device ring for copying at a later point. The parameters to that function
+include the physical addresses of both the source and destination buffers,
+as well as two "handles" to be returned to the user when the copy is
+completed. These handles can be arbitrary values, but two are provided so
+that the library can track handles for both source and destination on
+behalf of the user, e.g. virtual addresses for the buffers, or mbuf
+pointers if packet data is being copied.
+
+While the ``rte_ioat_enqueue_copy()`` function enqueues a copy operation on
+the device ring, the copy will not actually be performed until after the
+application calls the ``rte_ioat_do_copies()`` function. This function
+informs the device hardware of the elements enqueued on the ring, and the
+device will begin to process them. It is expected that, for efficiency
+reasons, a burst of operations will be enqueued to the device via multiple
+enqueue calls between calls to the ``rte_ioat_do_copies()`` function.
+
+The following code from ``test_ioat_rawdev.c`` demonstrates how to enqueue
+a burst of copies to the device and start the hardware processing of them:
+
+.. code-block:: C
+
+        struct rte_mbuf *srcs[32], *dsts[32];
+        unsigned int j;
+
+        for (i = 0; i < RTE_DIM(srcs); i++) {
+                char *src_data;
+
+                srcs[i] = rte_pktmbuf_alloc(pool);
+                dsts[i] = rte_pktmbuf_alloc(pool);
+                srcs[i]->data_len = srcs[i]->pkt_len = length;
+                dsts[i]->data_len = dsts[i]->pkt_len = length;
+                src_data = rte_pktmbuf_mtod(srcs[i], char *);
+
+                for (j = 0; j < length; j++)
+                        src_data[j] = rand() & 0xFF;
+
+                if (rte_ioat_enqueue_copy(dev_id,
+                                srcs[i]->buf_iova + srcs[i]->data_off,
+                                dsts[i]->buf_iova + dsts[i]->data_off,
+                                length,
+                                (uintptr_t)srcs[i],
+                                (uintptr_t)dsts[i],
+                                0 /* nofence */) != 1) {
+                        printf("Error with rte_ioat_enqueue_copy for buffer %u\n",
+                                        i);
+                        return -1;
+                }
+        }
+        rte_ioat_do_copies(dev_id);
+
+To retrieve information about completed copies, the API
+``rte_ioat_completed_copies()`` should be used. This API will return to the
+application a set of completion handles passed in when the relevant copies
+were enqueued.
+
+The following code from ``test_ioat_rawdev.c`` shows the test code
+retrieving information about the completed copies and validating the data
+is correct before freeing the data buffers using the returned handles:
+
+.. code-block:: C
+
+        if (rte_ioat_completed_copies(dev_id, 64, (void *)completed_src,
+                        (void *)completed_dst) != RTE_DIM(srcs)) {
+                printf("Error with rte_ioat_completed_copies\n");
+                return -1;
+        }
+        for (i = 0; i < RTE_DIM(srcs); i++) {
+                char *src_data, *dst_data;
+
+                if (completed_src[i] != srcs[i]) {
+                        printf("Error with source pointer %u\n", i);
+                        return -1;
+                }
+                if (completed_dst[i] != dsts[i]) {
+                        printf("Error with dest pointer %u\n", i);
+                        return -1;
+                }
+
+                src_data = rte_pktmbuf_mtod(srcs[i], char *);
+                dst_data = rte_pktmbuf_mtod(dsts[i], char *);
+                for (j = 0; j < length; j++)
+                        if (src_data[j] != dst_data[j]) {
+                                printf("Error with copy of packet %u, byte %u\n",
+                                                i, j);
+                                return -1;
+                        }
+                rte_pktmbuf_free(srcs[i]);
+                rte_pktmbuf_free(dsts[i]);
+        }
+
+
 Querying Device Statistics
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile
index b1af9c666..32f079845 100644
--- a/drivers/raw/ioat/Makefile
+++ b/drivers/raw/ioat/Makefile
@@ -12,6 +12,7 @@ CFLAGS += $(WERROR_FLAGS)
 
 LDLIBS += -lrte_eal -lrte_rawdev
 LDLIBS += -lrte_pci -lrte_bus_pci
+LDLIBS += -lrte_mbuf -lrte_mempool
 
 # library version
 LIBABIVER := 1
diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c
index ab671816d..f6c7dbb80 100644
--- a/drivers/raw/ioat/ioat_rawdev_test.c
+++ b/drivers/raw/ioat/ioat_rawdev_test.c
@@ -2,12 +2,139 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
+#include <unistd.h>
 #include <inttypes.h>
+#include <rte_mbuf.h>
 #include "rte_rawdev.h"
 #include "rte_ioat_rawdev.h"
 
 int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */
 
+static struct rte_mempool *pool;
+
+static int
+test_enqueue_copies(int dev_id)
+{
+	const unsigned int length = 1024;
+	unsigned int i;
+
+	do {
+		struct rte_mbuf *src, *dst;
+		char *src_data, *dst_data;
+		struct rte_mbuf *completed[2] = {0};
+
+		/* test doing a single copy */
+		src = rte_pktmbuf_alloc(pool);
+		dst = rte_pktmbuf_alloc(pool);
+		src->data_len = src->pkt_len = length;
+		dst->data_len = dst->pkt_len = length;
+		src_data = rte_pktmbuf_mtod(src, char *);
+		dst_data = rte_pktmbuf_mtod(dst, char *);
+
+		for (i = 0; i < length; i++)
+			src_data[i] = rand() & 0xFF;
+
+		if (rte_ioat_enqueue_copy(dev_id,
+				src->buf_iova + src->data_off,
+				dst->buf_iova + dst->data_off,
+				length,
+				(uintptr_t)src,
+				(uintptr_t)dst,
+				0 /* no fence */) != 1) {
+			printf("Error with rte_ioat_enqueue_copy\n");
+			return -1;
+		}
+		rte_ioat_do_copies(dev_id);
+		usleep(10);
+
+		if (rte_ioat_completed_copies(dev_id, 1, (void *)&completed[0],
+				(void *)&completed[1]) != 1) {
+			printf("Error with rte_ioat_completed_copies\n");
+			return -1;
+		}
+		if (completed[0] != src || completed[1] != dst) {
+			printf("Error with completions: got (%p, %p), not (%p,%p)\n",
+					completed[0], completed[1], src, dst);
+			return -1;
+		}
+
+		for (i = 0; i < length; i++)
+			if (dst_data[i] != src_data[i]) {
+				printf("Data mismatch at char %u\n", i);
+				return -1;
+			}
+		rte_pktmbuf_free(src);
+		rte_pktmbuf_free(dst);
+	} while (0);
+
+	/* test doing multiple copies */
+	do {
+		struct rte_mbuf *srcs[32], *dsts[32];
+		struct rte_mbuf *completed_src[64];
+		struct rte_mbuf *completed_dst[64];
+		unsigned int j;
+
+		for (i = 0; i < RTE_DIM(srcs); i++) {
+			char *src_data;
+
+			srcs[i] = rte_pktmbuf_alloc(pool);
+			dsts[i] = rte_pktmbuf_alloc(pool);
+			srcs[i]->data_len = srcs[i]->pkt_len = length;
+			dsts[i]->data_len = dsts[i]->pkt_len = length;
+			src_data = rte_pktmbuf_mtod(srcs[i], char *);
+
+			for (j = 0; j < length; j++)
+				src_data[j] = rand() & 0xFF;
+
+			if (rte_ioat_enqueue_copy(dev_id,
+					srcs[i]->buf_iova + srcs[i]->data_off,
+					dsts[i]->buf_iova + dsts[i]->data_off,
+					length,
+					(uintptr_t)srcs[i],
+					(uintptr_t)dsts[i],
+					0 /* nofence */) != 1) {
+				printf("Error with rte_ioat_enqueue_copy for buffer %u\n",
+						i);
+				return -1;
+			}
+		}
+		rte_ioat_do_copies(dev_id);
+		usleep(100);
+
+		if (rte_ioat_completed_copies(dev_id, 64, (void *)completed_src,
+				(void *)completed_dst) != RTE_DIM(srcs)) {
+			printf("Error with rte_ioat_completed_copies\n");
+			return -1;
+		}
+		for (i = 0; i < RTE_DIM(srcs); i++) {
+			char *src_data, *dst_data;
+
+			if (completed_src[i] != srcs[i]) {
+				printf("Error with source pointer %u\n", i);
+				return -1;
+			}
+			if (completed_dst[i] != dsts[i]) {
+				printf("Error with dest pointer %u\n", i);
+				return -1;
+			}
+
+			src_data = rte_pktmbuf_mtod(srcs[i], char *);
+			dst_data = rte_pktmbuf_mtod(dsts[i], char *);
+			for (j = 0; j < length; j++)
+				if (src_data[j] != dst_data[j]) {
+					printf("Error with copy of packet %u, byte %u\n",
+							i, j);
+					return -1;
+				}
+			rte_pktmbuf_free(srcs[i]);
+			rte_pktmbuf_free(dsts[i]);
+		}
+
+	} while (0);
+
+	return 0;
+}
+
 int
 ioat_rawdev_test(uint16_t dev_id)
 {
@@ -44,6 +171,17 @@ ioat_rawdev_test(uint16_t dev_id)
 		return -1;
 	}
 
+	pool = rte_pktmbuf_pool_create("TEST_IOAT_POOL",
+			256, /* n == num elements */
+			32,  /* cache size */
+			0,   /* priv size */
+			2048, /* data room size */
+			info.socket_id);
+	if (pool == NULL) {
+		printf("Error with mempool creation\n");
+		return -1;
+	}
+
 	/* allocate memory for xstats names and values */
 	nb_xstats = rte_rawdev_xstats_names_get(dev_id, NULL, 0);
 
@@ -68,17 +206,28 @@ ioat_rawdev_test(uint16_t dev_id)
 		goto err;
 	}
 
-	rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
-	for (i = 0; i < nb_xstats; i++)
-		printf("%s: %"PRIu64"   ", snames[i].name, stats[i]);
+	/* run the test cases */
+	for (i = 0; i < 100; i++) {
+		unsigned int j;
+
+		if (test_enqueue_copies(dev_id) != 0)
+			goto err;
+
+		rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats);
+		for (j = 0; j < nb_xstats; j++)
+			printf("%s: %"PRIu64"   ", snames[j].name, stats[j]);
+		printf("\r");
+	}
 	printf("\n");
 
+	rte_mempool_free(pool);
 	free(snames);
 	free(stats);
 	free(ids);
 	return 0;
 
 err:
+	rte_mempool_free(pool);
 	free(snames);
 	free(stats);
 	free(ids);
diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build
index 40fff6654..247ff88bf 100644
--- a/drivers/raw/ioat/meson.build
+++ b/drivers/raw/ioat/meson.build
@@ -4,7 +4,7 @@
 build = dpdk_conf.has('RTE_ARCH_X86')
 sources = files('ioat_rawdev.c',
 		'ioat_rawdev_test.c')
-deps += ['rawdev', 'bus_pci']
+deps += ['rawdev', 'bus_pci', 'mbuf']
 
 install_headers('rte_ioat_rawdev.h',
 		'rte_ioat_spec.h')
diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h
index d5326813c..3babb822d 100644
--- a/drivers/raw/ioat/rte_ioat_rawdev.h
+++ b/drivers/raw/ioat/rte_ioat_rawdev.h
@@ -15,8 +15,10 @@
  */
 
 #include <x86intrin.h>
+#include <rte_atomic.h>
 #include <rte_memory.h>
 #include <rte_memzone.h>
+#include <rte_prefetch.h>
 #include <rte_ioat_spec.h>
 
 /** Name of the device driver */
@@ -54,6 +56,10 @@ struct rte_ioat_rawdev {
 	struct rte_ioat_generic_hw_desc *desc_ring;
 	__m128i *hdls; /* completion handles for returning to user */
 
+
+	unsigned short next_read;
+	unsigned short next_write;
+
 	/* some statistics for tracking, if added/changed update xstats fns*/
 	uint64_t enqueue_failed __rte_cache_aligned;
 	uint64_t enqueued;
@@ -64,4 +70,165 @@ struct rte_ioat_rawdev {
 	volatile uint64_t status __rte_cache_aligned;
 };
 
-#endif
+/**
+ * Enqueue a copy operation onto the ioat device
+ *
+ * This queues up a copy operation to be performed by hardware, but does not
+ * trigger hardware to begin that operation.
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ * @param src
+ *   The physical address of the source buffer
+ * @param dst
+ *   The physical address of the destination buffer
+ * @param length
+ *   The length of the data to be copied
+ * @param src_hdl
+ *   An opaque handle for the source data, to be returned when this operation
+ *   has been completed and the user polls for the completion details
+ * @param dst_hdl
+ *   An opaque handle for the destination data, to be returned when this
+ *   operation has been completed and the user polls for the completion details
+ * @param fence
+ *   A flag parameter indicating that hardware should not begin to perform any
+ *   subsequently enqueued copy operations until after this operation has
+ *   completed
+ * @return
+ *   Number of operations enqueued, either 0 or 1
+ */
+static inline int
+rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst,
+		unsigned int length, uintptr_t src_hdl, uintptr_t dst_hdl,
+		int fence)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	unsigned short read = ioat->next_read;
+	unsigned short write = ioat->next_write;
+	unsigned short mask = ioat->ring_size - 1;
+	unsigned short space = mask + read - write;
+	struct rte_ioat_generic_hw_desc *desc;
+
+	if (space == 0) {
+		ioat->enqueue_failed++;
+		return 0;
+	}
+
+	ioat->next_write = write + 1;
+	write &= mask;
+
+	desc = &ioat->desc_ring[write];
+	desc->size = length;
+	/* set descriptor write-back every 16th descriptor */
+	desc->u.control_raw = (uint32_t)((!!fence << 4) | (!(write & 0xF)) << 3);
+	desc->src_addr = src;
+	desc->dest_addr = dst;
+
+	ioat->hdls[write] = _mm_set_epi64((__m64)((uint64_t)dst_hdl),
+			(__m64)((uint64_t)src_hdl));
+	rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]);
+
+	ioat->enqueued++;
+	return 1;
+}
+
+/**
+ * Trigger hardware to begin performing enqueued copy operations
+ *
+ * This API is used to write the "doorbell" to the hardware to trigger it
+ * to begin the copy operations previously enqueued by rte_ioat_enqueue_copy()
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ */
+static inline void
+rte_ioat_do_copies(int dev_id)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	ioat->desc_ring[(ioat->next_write - 1) & (ioat->ring_size - 1)].u
+			.control.completion_update = 1;
+	rte_compiler_barrier();
+	ioat->regs->dmacount = ioat->next_write;
+	ioat->started = ioat->enqueued;
+}
+
+/**
+ * @internal
+ * Returns the index of the last completed operation.
+ */
+static inline int
+rte_ioat_get_last_completed(struct rte_ioat_rawdev *ioat, int *error)
+{
+	uint64_t status = ioat->status;
+
+	/* lower 3 bits indicate "transfer status" : active, idle, halted.
+	 * We can ignore bit 0.
+	 */
+	*error = status & (RTE_IOAT_CHANSTS_SUSPENDED | RTE_IOAT_CHANSTS_ARMED);
+	return (status - ioat->ring_addr) >> 6;
+}
+
+/**
+ * Returns details of copy operations that have been completed
+ *
+ * Returns to the caller the user-provided "handles" for the copy operations
+ * which have been completed by the hardware, and not already returned by
+ * a previous call to this API.
+ *
+ * @param dev_id
+ *   The rawdev device id of the ioat instance
+ * @param max_copies
+ *   The number of entries which can fit in the src_hdls and dst_hdls
+ *   arrays, i.e. max number of completed operations to report
+ * @param src_hdls
+ *   Array to hold the source handle parameters of the completed copies
+ * @param dst_hdls
+ *   Array to hold the destination handle parameters of the completed copies
+ * @return
+ *   -1 on error, with rte_errno set appropriately.
+ *   Otherwise number of completed operations i.e. number of entries written
+ *   to the src_hdls and dst_hdls array parameters.
+ */
+static inline int
+rte_ioat_completed_copies(int dev_id, uint8_t max_copies,
+		uintptr_t *src_hdls, uintptr_t *dst_hdls)
+{
+	struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private;
+	unsigned short mask = (ioat->ring_size - 1);
+	unsigned short read = ioat->next_read;
+	unsigned short end_read, count;
+	int error;
+	int i = 0;
+
+	end_read = (rte_ioat_get_last_completed(ioat, &error) + 1) & mask;
+	count = (end_read - (read & mask)) & mask;
+
+	if (error) {
+		rte_errno = EIO;
+		return -1;
+	}
+
+	if (count > max_copies)
+		count = max_copies;
+
+	for (; i < count - 1; i += 2, read += 2) {
+		__m128i hdls0 = _mm_load_si128(&ioat->hdls[read & mask]);
+		__m128i hdls1 = _mm_load_si128(&ioat->hdls[(read + 1) & mask]);
+
+		_mm_storeu_si128((void *)&src_hdls[i],
+				_mm_unpacklo_epi64(hdls0, hdls1));
+		_mm_storeu_si128((void *)&dst_hdls[i],
+				_mm_unpackhi_epi64(hdls0, hdls1));
+	}
+	for (; i < count; i++, read++) {
+		uintptr_t *hdls = (void *)&ioat->hdls[read & mask];
+		src_hdls[i] = hdls[0];
+		dst_hdls[i] = hdls[1];
+	}
+
+	ioat->next_read = read;
+	ioat->completed += count;
+	return count;
+}
+
+#endif /* _RTE_IOAT_RAWDEV_H_ */
-- 
2.21.0

Re: [dpdk-dev] [PATCH v5 9/9] raw/ioat: add local API to perform copies

[Hu, Jiayu]

Acked-by: Jiayu Hu <jiayu.hu@intel.com>

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Tuesday, July 2, 2019 10:13 PM
> To: dev@dpdk.org
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Subject: [PATCH v5 9/9] raw/ioat: add local API to perform copies
> 
> Add local APIs to trigger data copies, and retrieve handle values once
> those copies are completed. Included are unit tests to validate the data
> is copies correctly.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> ---
> 
> V3: updated to use descriptor format in rte_ioat_spec.h
> V2: test cases moved to self-test routine
> ---

Re: [dpdk-dev] [PATCH v5 0/9] raw/ioat: driver for Intel QuickData Technology

[Thomas Monjalon]

02/07/2019 16:12, Bruce Richardson:
> This patch series adds support for the Intel QuickData Technology
> device, part of the Intel I/O Acceleration Technology (Intel I/OAT). It
> is a raw device for allowing hardware DMA i.e. data copies in hardware.
> 
> Performing the copies in hardware can provide performance improvements
> for applications where the average copy size is reasonably large, e.g.
> 1k packets. For smaller packets, e.g. 64-256 bytes, offloading the copy
> may reduce performance due to the overhead of using hardware.
[...]
> Bruce Richardson (9):
>   rawdev: allow devices to skip extra memory allocation
>   raw/ioat: add initial support for ioat rawdev driver
>   usertools/dpdk-devbind.py: add support for IOAT devices
>   raw/ioat: add register definition file
>   raw/ioat: create device on probe and destroy on release
>   raw/ioat: add device info function
>   raw/ioat: add configure, start and stop functions
>   raw/ioat: add statistics functions
>   raw/ioat: add local API to perform copies

Applied, thanks

Re: [dpdk-dev] [PATCH v5 0/9] raw/ioat: driver for Intel QuickData Technology

[Ferruh Yigit]

On 7/2/2019 3:12 PM, Bruce Richardson wrote:
> This patch series adds support for the Intel QuickData Technology
> device, part of the Intel I/O Acceleration Technology (Intel I/OAT). It
> is a raw device for allowing hardware DMA i.e. data copies in hardware.
> 
> Performing the copies in hardware can provide performance improvements
> for applications where the average copy size is reasonably large, e.g.
> 1k packets. For smaller packets, e.g. 64-256 bytes, offloading the copy
> may reduce performance due to the overhead of using hardware.
> 
> V5:
>  * updated doxygen comment for internal rawdev pmd API
> 
> V4:
>  * changed memory management to use contiguous memzones instead of
>    malloc memory
>  * added missing device ids for BDX platforms
>  * other misc cleanup following review (see individual patches logs)
> 
> V3:
>  * removed DPDK-specific structure for the descriptor format and reused
>    the structure in the imported file rte_ioat_spec.h
> 
> V2:
> * moved tests to rawdev selftest function
> * some checkpatch and other small cleanups
> * added extra documentation details on supported hardware
> * aligned the changes to dpdk-devbind with the changes in the NTB set
>   for consistency
> 
> 
> Bruce Richardson (9):
>   rawdev: allow devices to skip extra memory allocation
>   raw/ioat: add initial support for ioat rawdev driver
>   usertools/dpdk-devbind.py: add support for IOAT devices
>   raw/ioat: add register definition file
>   raw/ioat: create device on probe and destroy on release
>   raw/ioat: add device info function
>   raw/ioat: add configure, start and stop functions
>   raw/ioat: add statistics functions
>   raw/ioat: add local API to perform copies


Hi Bruce, Thomas,

Getting following build error with icc, both for 32-bit & 64-bit:

.../dpdk/drivers/raw/ioat/ioat_rawdev_test.c(28): (col. 9) error #13203: No EMMS
instruction before call to function