From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2F978A00C3; Tue, 20 Sep 2022 17:14:24 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2017B410E8; Tue, 20 Sep 2022 17:14:24 +0200 (CEST) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id BF1FD40DFB for ; Tue, 20 Sep 2022 17:14:21 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663686861; x=1695222861; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=iBSTySQyVkvtYeyXWY7787Bq0T31CPvIBdZqCmfdIOs=; b=l3xgXeytRVZ3Tt36sAws6zei2KOXC65glbU/ziio/sHjzoePK4O1MZ4O Qtwod5HCm9IgdesSxgV4+m4G1bfAH3CqhGUn7bP7GaZlsuQSbFnZPwf6E ay6vtOQwbBIQblaClC8JJF/h/tMUY6bXq/abLiD+r1zlUMh0XprRddHvW /4IprsHzeLYfUx3J+iNQS6iQwvyMyjNH/2ZWHl0AUAEuQcSYS2sTLRY5u BHmyxeP096ezmaOVPeok6r8UWYP+2Dh2Y1lyh1b+QFlMVrJxHQ9m5zipf W5QJFbE9wdTWU7bhB7vfy1OoOE30JNWh3nJ0wlOUcEmo4xIXDScrV9h93 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10476"; a="298447870" X-IronPort-AV: E=Sophos;i="5.93,330,1654585200"; d="scan'208";a="298447870" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Sep 2022 08:14:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,330,1654585200"; d="scan'208";a="744568220" Received: from silpixa00401385.ir.intel.com ([10.237.214.41]) by orsmga004.jf.intel.com with ESMTP; 20 Sep 2022 08:14:13 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Bruce Richardson Subject: [PATCH] raw/ioat: remove deprecated driver Date: Tue, 20 Sep 2022 16:14:04 +0100 Message-Id: <20220920151404.374080-1-bruce.richardson@intel.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The ioat rawdev driver has been superceded by the ioat and idxd dmadev drivers, and has been deprecated for some time, so remove it. Signed-off-by: Bruce Richardson --- doc/guides/rawdevs/index.rst | 1 - doc/guides/rawdevs/ioat.rst | 333 ----------- doc/guides/rel_notes/deprecation.rst | 7 - drivers/raw/ioat/dpdk_idxd_cfg.py | 1 - drivers/raw/ioat/idxd_bus.c | 365 ------------ drivers/raw/ioat/idxd_pci.c | 380 ------------- drivers/raw/ioat/ioat_common.c | 273 --------- drivers/raw/ioat/ioat_private.h | 84 --- drivers/raw/ioat/ioat_rawdev.c | 332 ----------- drivers/raw/ioat/ioat_rawdev_test.c | 734 ------------------------- drivers/raw/ioat/ioat_spec.h | 336 ----------- drivers/raw/ioat/meson.build | 36 -- drivers/raw/ioat/rte_idxd_rawdev_fns.h | 394 ------------- drivers/raw/ioat/rte_ioat_rawdev.h | 214 ------- drivers/raw/ioat/rte_ioat_rawdev_fns.h | 379 ------------- drivers/raw/ioat/version.map | 3 - drivers/raw/meson.build | 1 - 17 files changed, 3873 deletions(-) delete mode 100644 doc/guides/rawdevs/ioat.rst delete mode 120000 drivers/raw/ioat/dpdk_idxd_cfg.py delete mode 100644 drivers/raw/ioat/idxd_bus.c delete mode 100644 drivers/raw/ioat/idxd_pci.c delete mode 100644 drivers/raw/ioat/ioat_common.c delete mode 100644 drivers/raw/ioat/ioat_private.h delete mode 100644 drivers/raw/ioat/ioat_rawdev.c delete mode 100644 drivers/raw/ioat/ioat_rawdev_test.c delete mode 100644 drivers/raw/ioat/ioat_spec.h delete mode 100644 drivers/raw/ioat/meson.build delete mode 100644 drivers/raw/ioat/rte_idxd_rawdev_fns.h delete mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h delete mode 100644 drivers/raw/ioat/rte_ioat_rawdev_fns.h delete mode 100644 drivers/raw/ioat/version.map diff --git a/doc/guides/rawdevs/index.rst b/doc/guides/rawdevs/index.rst index cf69633064..f34315f051 100644 --- a/doc/guides/rawdevs/index.rst +++ b/doc/guides/rawdevs/index.rst @@ -15,5 +15,4 @@ application through rawdev API. cnxk_gpio dpaa2_cmdif ifpga - ioat ntb diff --git a/doc/guides/rawdevs/ioat.rst b/doc/guides/rawdevs/ioat.rst deleted file mode 100644 index 98d15dd032..0000000000 --- a/doc/guides/rawdevs/ioat.rst +++ /dev/null @@ -1,333 +0,0 @@ -.. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2019 Intel Corporation. - -.. include:: - -IOAT Rawdev Driver -=================== - -.. warning:: - As of DPDK 21.11 the rawdev implementation of the IOAT driver has been deprecated. - Please use the dmadev library instead. - -The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg| -Data Streaming Accelerator `(Intel DSA) -`_ and for Intel\ |reg| -QuickData Technology, part of Intel\ |reg| I/O Acceleration Technology -`(Intel I/OAT) -`_. -This PMD, when used on supported hardware, allows data copies, for example, -cloning packet data, to be accelerated by that hardware rather than having to -be done by software, freeing up CPU cycles for other tasks. - -Hardware Requirements ----------------------- - -The ``dpdk-devbind.py`` script, included with DPDK, -can be used to show the presence of supported hardware. -Running ``dpdk-devbind.py --status-dev misc`` will show all the miscellaneous, -or rawdev-based devices on the system. -For Intel\ |reg| QuickData Technology devices, the hardware will be often listed as "Crystal Beach DMA", -or "CBDMA". -For Intel\ |reg| DSA devices, they are currently (at time of writing) appearing as devices with type "0b25", -due to the absence of pci-id database entries for them at this point. - -Compilation ------------- - -For builds using ``meson`` and ``ninja``, the driver will be built when the target platform is x86-based. -No additional compilation steps are necessary. - -.. note:: - Since the addition of the dmadev library, the ``ioat`` and ``idxd`` parts of this driver - will only be built if their ``dmadev`` counterparts are not built. - The following can be used to disable the ``dmadev`` drivers, - if the raw drivers are to be used instead:: - - $ meson -Ddisable_drivers=dma/* - -Device Setup -------------- - -Depending on support provided by the PMD, HW devices can either use the kernel configured driver -or be bound to a user-space IO driver for use. -For example, Intel\ |reg| DSA devices can use the IDXD kernel driver or DPDK-supported drivers, -such as ``vfio-pci``. - -Intel\ |reg| DSA devices using idxd kernel driver -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To use a Intel\ |reg| DSA device bound to the IDXD kernel driver, the device must first be configured. -The `accel-config `_ utility library can be used for configuration. - -.. note:: - The device configuration can also be done by directly interacting with the sysfs nodes. - An example of how this may be done can be seen in the script ``dpdk_idxd_cfg.py`` - included in the driver source directory. - -There are some mandatory configuration steps before being able to use a device with an application. -The internal engines, which do the copies or other operations, -and the work-queues, which are used by applications to assign work to the device, -need to be assigned to groups, and the various other configuration options, -such as priority or queue depth, need to be set for each queue. - -To assign an engine to a group:: - - $ accel-config config-engine dsa0/engine0.0 --group-id=0 - $ accel-config config-engine dsa0/engine0.1 --group-id=1 - -To assign work queues to groups for passing descriptors to the engines a similar accel-config command can be used. -However, the work queues also need to be configured depending on the use case. -Some configuration options include: - -* mode (Dedicated/Shared): Indicates whether a WQ may accept jobs from multiple queues simultaneously. -* priority: WQ priority between 1 and 15. Larger value means higher priority. -* wq-size: the size of the WQ. Sum of all WQ sizes must be less that the total-size defined by the device. -* type: WQ type (kernel/mdev/user). Determines how the device is presented. -* name: identifier given to the WQ. - -Example configuration for a work queue:: - - $ accel-config config-wq dsa0/wq0.0 --group-id=0 \ - --mode=dedicated --priority=10 --wq-size=8 \ - --type=user --name=dpdk_app1 - -Once the devices have been configured, they need to be enabled:: - - $ accel-config enable-device dsa0 - $ accel-config enable-wq dsa0/wq0.0 - -Check the device configuration:: - - $ accel-config list - -Devices using VFIO/UIO drivers -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The HW devices to be used will need to be bound to a user-space IO driver for use. -The ``dpdk-devbind.py`` script can be used to view the state of the devices -and to bind them to a suitable DPDK-supported driver, such as ``vfio-pci``. -For example:: - - $ dpdk-devbind.py -b vfio-pci 00:04.0 00:04.1 - -Device Probing and Initialization -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For devices bound to a suitable DPDK-supported VFIO/UIO driver, the HW devices will -be found as part of the device scan done at application initialization time without -the need to pass parameters to the application. - -For Intel\ |reg| DSA devices, DPDK will automatically configure the device with the -maximum number of workqueues available on it, partitioning all resources equally -among the queues. -If fewer workqueues are required, then the ``max_queues`` parameter may be passed to -the device driver on the EAL commandline, via the ``allowlist`` or ``-a`` flag e.g.:: - - $ dpdk-test -a ,max_queues=4 - -For devices bound to the IDXD kernel driver, -the DPDK ioat driver will automatically perform a scan for available workqueues to use. -Any workqueues found listed in ``/dev/dsa`` on the system will be checked in ``/sys``, -and any which have ``dpdk_`` prefix in their name will be automatically probed by the -driver to make them available to the application. -Alternatively, to support use by multiple DPDK processes simultaneously, -the value used as the DPDK ``--file-prefix`` parameter may be used as a workqueue name prefix, -instead of ``dpdk_``, -allowing each DPDK application instance to only use a subset of configured queues. - -Once probed successfully, irrespective of kernel driver, the device will appear as a ``rawdev``, -that is a "raw device type" inside DPDK, and can be accessed using APIs from the -``rte_rawdev`` library. - -Using IOAT Rawdev Devices --------------------------- - -To use the devices from an application, the rawdev API can be used, along -with definitions taken from the device-specific header file -``rte_ioat_rawdev.h``. This header is needed to get the definition of -structure parameters used by some of the rawdev APIs for IOAT rawdev -devices, as well as providing key functions for using the device for memory -copies. - -Getting Device Information -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Basic information about each rawdev device can be queried using the -``rte_rawdev_info_get()`` API. For most applications, this API will be -needed to verify that the rawdev in question is of the expected type. For -example, the following code snippet can be used to identify an IOAT -rawdev device for use by an application: - -.. code-block:: C - - for (i = 0; i < count && !found; i++) { - struct rte_rawdev_info info = { .dev_private = NULL }; - found = (rte_rawdev_info_get(i, &info, 0) == 0 && - strcmp(info.driver_name, - IOAT_PMD_RAWDEV_NAME_STR) == 0); - } - -When calling the ``rte_rawdev_info_get()`` API for an IOAT rawdev device, -the ``dev_private`` field in the ``rte_rawdev_info`` struct should either -be NULL, or else be set to point to a structure of type -``rte_ioat_rawdev_config``, in which case the size of the configured device -input ring will be returned in that structure. - -Device Configuration -~~~~~~~~~~~~~~~~~~~~~ - -Configuring an IOAT rawdev device is done using the -``rte_rawdev_configure()`` API, which takes the same structure parameters -as the, previously referenced, ``rte_rawdev_info_get()`` API. The main -difference is that, because the parameter is used as input rather than -output, the ``dev_private`` structure element cannot be NULL, and must -point to a valid ``rte_ioat_rawdev_config`` structure, containing the ring -size to be used by the device. The ring size must be a power of two, -between 64 and 4096. -If it is not needed, the tracking by the driver of user-provided completion -handles may be disabled by setting the ``hdls_disable`` flag in -the configuration structure also. - -The following code shows how the device is configured in -``test_ioat_rawdev.c``: - -.. code-block:: C - - #define IOAT_TEST_RINGSIZE 512 - struct rte_ioat_rawdev_config p = { .ring_size = -1 }; - struct rte_rawdev_info info = { .dev_private = &p }; - - /* ... */ - - p.ring_size = IOAT_TEST_RINGSIZE; - if (rte_rawdev_configure(dev_id, &info, sizeof(p)) != 0) { - printf("Error with rte_rawdev_configure()\n"); - return -1; - } - -Once configured, the device can then be made ready for use by calling the -``rte_rawdev_start()`` API. - -Performing Data Copies -~~~~~~~~~~~~~~~~~~~~~~~ - -To perform data copies using IOAT rawdev devices, the functions -``rte_ioat_enqueue_copy()`` and ``rte_ioat_perform_ops()`` should be used. -Once copies have been completed, the completion will be reported back when -the application calls ``rte_ioat_completed_ops()``. - -The ``rte_ioat_enqueue_copy()`` function enqueues a single copy to the -device ring for copying at a later point. The parameters to that function -include the IOVA addresses of both the source and destination buffers, -as well as two "handles" to be returned to the user when the copy is -completed. These handles can be arbitrary values, but two are provided so -that the library can track handles for both source and destination on -behalf of the user, e.g. virtual addresses for the buffers, or mbuf -pointers if packet data is being copied. - -While the ``rte_ioat_enqueue_copy()`` function enqueues a copy operation on -the device ring, the copy will not actually be performed until after the -application calls the ``rte_ioat_perform_ops()`` function. This function -informs the device hardware of the elements enqueued on the ring, and the -device will begin to process them. It is expected that, for efficiency -reasons, a burst of operations will be enqueued to the device via multiple -enqueue calls between calls to the ``rte_ioat_perform_ops()`` function. - -The following code from ``test_ioat_rawdev.c`` demonstrates how to enqueue -a burst of copies to the device and start the hardware processing of them: - -.. code-block:: C - - struct rte_mbuf *srcs[32], *dsts[32]; - unsigned int j; - - for (i = 0; i < RTE_DIM(srcs); i++) { - char *src_data; - - srcs[i] = rte_pktmbuf_alloc(pool); - dsts[i] = rte_pktmbuf_alloc(pool); - srcs[i]->data_len = srcs[i]->pkt_len = length; - dsts[i]->data_len = dsts[i]->pkt_len = length; - src_data = rte_pktmbuf_mtod(srcs[i], char *); - - for (j = 0; j < length; j++) - src_data[j] = rand() & 0xFF; - - if (rte_ioat_enqueue_copy(dev_id, - srcs[i]->buf_iova + srcs[i]->data_off, - dsts[i]->buf_iova + dsts[i]->data_off, - length, - (uintptr_t)srcs[i], - (uintptr_t)dsts[i]) != 1) { - printf("Error with rte_ioat_enqueue_copy for buffer %u\n", - i); - return -1; - } - } - rte_ioat_perform_ops(dev_id); - -To retrieve information about completed copies, the API -``rte_ioat_completed_ops()`` should be used. This API will return to the -application a set of completion handles passed in when the relevant copies -were enqueued. - -The following code from ``test_ioat_rawdev.c`` shows the test code -retrieving information about the completed copies and validating the data -is correct before freeing the data buffers using the returned handles: - -.. code-block:: C - - if (rte_ioat_completed_ops(dev_id, 64, (void *)completed_src, - (void *)completed_dst) != RTE_DIM(srcs)) { - printf("Error with rte_ioat_completed_ops\n"); - return -1; - } - for (i = 0; i < RTE_DIM(srcs); i++) { - char *src_data, *dst_data; - - if (completed_src[i] != srcs[i]) { - printf("Error with source pointer %u\n", i); - return -1; - } - if (completed_dst[i] != dsts[i]) { - printf("Error with dest pointer %u\n", i); - return -1; - } - - src_data = rte_pktmbuf_mtod(srcs[i], char *); - dst_data = rte_pktmbuf_mtod(dsts[i], char *); - for (j = 0; j < length; j++) - if (src_data[j] != dst_data[j]) { - printf("Error with copy of packet %u, byte %u\n", - i, j); - return -1; - } - rte_pktmbuf_free(srcs[i]); - rte_pktmbuf_free(dsts[i]); - } - - -Filling an Area of Memory -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The IOAT driver also has support for the ``fill`` operation, where an area -of memory is overwritten, or filled, with a short pattern of data. -Fill operations can be performed in much the same was as copy operations -described above, just using the ``rte_ioat_enqueue_fill()`` function rather -than the ``rte_ioat_enqueue_copy()`` function. - - -Querying Device Statistics -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The statistics from the IOAT rawdev device can be got via the xstats -functions in the ``rte_rawdev`` library, i.e. -``rte_rawdev_xstats_names_get()``, ``rte_rawdev_xstats_get()`` and -``rte_rawdev_xstats_by_name_get``. The statistics returned for each device -instance are: - -* ``failed_enqueues`` -* ``successful_enqueues`` -* ``copies_started`` -* ``copies_completed`` diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst index e7583cae4c..c1975d6c3e 100644 --- a/doc/guides/rel_notes/deprecation.rst +++ b/doc/guides/rel_notes/deprecation.rst @@ -230,10 +230,3 @@ Deprecation Notices * raw/ifgpa: The function ``rte_pmd_ifpga_get_pci_bus`` will be removed in DPDK 22.11. - -* raw/ioat: The ``ioat`` rawdev driver has been deprecated, since it's - functionality is provided through the new ``dmadev`` infrastructure. - To continue to use hardware previously supported by the ``ioat`` rawdev driver, - applications should be updated to use the ``dmadev`` library instead, - with the underlying HW-functionality being provided by the ``ioat`` or - ``idxd`` dma drivers diff --git a/drivers/raw/ioat/dpdk_idxd_cfg.py b/drivers/raw/ioat/dpdk_idxd_cfg.py deleted file mode 120000 index 85545548d1..0000000000 --- a/drivers/raw/ioat/dpdk_idxd_cfg.py +++ /dev/null @@ -1 +0,0 @@ -../../dma/idxd/dpdk_idxd_cfg.py \ No newline at end of file diff --git a/drivers/raw/ioat/idxd_bus.c b/drivers/raw/ioat/idxd_bus.c deleted file mode 100644 index 539f51b1b1..0000000000 --- a/drivers/raw/ioat/idxd_bus.c +++ /dev/null @@ -1,365 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2021 Intel Corporation - */ - -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include "ioat_private.h" - -/* default value for DSA paths, but allow override in environment for testing */ -#define DSA_DEV_PATH "/dev/dsa" -#define DSA_SYSFS_PATH "/sys/bus/dsa/devices" - -static unsigned int devcount; - -/** unique identifier for a DSA device/WQ instance */ -struct dsa_wq_addr { - uint16_t device_id; - uint16_t wq_id; -}; - -/** a DSA device instance */ -struct rte_dsa_device { - struct rte_device device; /**< Inherit core device */ - TAILQ_ENTRY(rte_dsa_device) next; /**< next dev in list */ - - char wq_name[32]; /**< the workqueue name/number e.g. wq0.1 */ - struct dsa_wq_addr addr; /**< Identifies the specific WQ */ -}; - -/* forward prototypes */ -struct dsa_bus; -static int dsa_scan(void); -static int dsa_probe(void); -static struct rte_device *dsa_find_device(const struct rte_device *start, - rte_dev_cmp_t cmp, const void *data); -static enum rte_iova_mode dsa_get_iommu_class(void); -static int dsa_addr_parse(const char *name, void *addr); - -/** List of devices */ -TAILQ_HEAD(dsa_device_list, rte_dsa_device); - -/** - * Structure describing the DSA bus - */ -struct dsa_bus { - struct rte_bus bus; /**< Inherit the generic class */ - struct rte_driver driver; /**< Driver struct for devices to point to */ - struct dsa_device_list device_list; /**< List of PCI devices */ -}; - -struct dsa_bus dsa_bus = { - .bus = { - .scan = dsa_scan, - .probe = dsa_probe, - .find_device = dsa_find_device, - .get_iommu_class = dsa_get_iommu_class, - .parse = dsa_addr_parse, - }, - .driver = { - .name = "rawdev_idxd" - }, - .device_list = TAILQ_HEAD_INITIALIZER(dsa_bus.device_list), -}; - -static inline const char * -dsa_get_dev_path(void) -{ - const char *path = getenv("DSA_DEV_PATH"); - return path ? path : DSA_DEV_PATH; -} - -static inline const char * -dsa_get_sysfs_path(void) -{ - const char *path = getenv("DSA_SYSFS_PATH"); - return path ? path : DSA_SYSFS_PATH; -} - -static const struct rte_rawdev_ops idxd_vdev_ops = { - .dev_close = idxd_rawdev_close, - .dev_selftest = ioat_rawdev_test, - .dump = idxd_dev_dump, - .dev_configure = idxd_dev_configure, - .dev_info_get = idxd_dev_info_get, - .xstats_get = ioat_xstats_get, - .xstats_get_names = ioat_xstats_get_names, - .xstats_reset = ioat_xstats_reset, -}; - -static void * -idxd_vdev_mmap_wq(struct rte_dsa_device *dev) -{ - void *addr; - char path[PATH_MAX]; - int fd; - - snprintf(path, sizeof(path), "%s/%s", dsa_get_dev_path(), dev->wq_name); - fd = open(path, O_RDWR); - if (fd < 0) { - IOAT_PMD_ERR("Failed to open device path: %s", path); - return NULL; - } - - addr = mmap(NULL, 0x1000, PROT_WRITE, MAP_SHARED, fd, 0); - close(fd); - if (addr == MAP_FAILED) { - IOAT_PMD_ERR("Failed to mmap device %s", path); - return NULL; - } - - return addr; -} - -static int -read_wq_string(struct rte_dsa_device *dev, const char *filename, - char *value, size_t valuelen) -{ - char sysfs_node[PATH_MAX]; - int len; - int fd; - - snprintf(sysfs_node, sizeof(sysfs_node), "%s/%s/%s", - dsa_get_sysfs_path(), dev->wq_name, filename); - fd = open(sysfs_node, O_RDONLY); - if (fd < 0) { - IOAT_PMD_ERR("%s(): opening file '%s' failed: %s", - __func__, sysfs_node, strerror(errno)); - return -1; - } - - len = read(fd, value, valuelen - 1); - close(fd); - if (len < 0) { - IOAT_PMD_ERR("%s(): error reading file '%s': %s", - __func__, sysfs_node, strerror(errno)); - return -1; - } - value[len] = '\0'; - return 0; -} - -static int -read_wq_int(struct rte_dsa_device *dev, const char *filename, - int *value) -{ - char sysfs_node[PATH_MAX]; - FILE *f; - int ret = 0; - - snprintf(sysfs_node, sizeof(sysfs_node), "%s/%s/%s", - dsa_get_sysfs_path(), dev->wq_name, filename); - f = fopen(sysfs_node, "r"); - if (f == NULL) { - IOAT_PMD_ERR("%s(): opening file '%s' failed: %s", - __func__, sysfs_node, strerror(errno)); - return -1; - } - - if (fscanf(f, "%d", value) != 1) { - IOAT_PMD_ERR("%s(): error reading file '%s': %s", - __func__, sysfs_node, strerror(errno)); - ret = -1; - } - - fclose(f); - return ret; -} - -static int -read_device_int(struct rte_dsa_device *dev, const char *filename, - int *value) -{ - char sysfs_node[PATH_MAX]; - FILE *f; - int ret = 0; - - snprintf(sysfs_node, sizeof(sysfs_node), "%s/dsa%d/%s", - dsa_get_sysfs_path(), dev->addr.device_id, filename); - f = fopen(sysfs_node, "r"); - if (f == NULL) { - IOAT_PMD_ERR("%s(): opening file '%s' failed: %s", - __func__, sysfs_node, strerror(errno)); - return -1; - } - - if (fscanf(f, "%d", value) != 1) { - IOAT_PMD_ERR("%s(): error reading file '%s': %s", - __func__, sysfs_node, strerror(errno)); - ret = -1; - } - - fclose(f); - return ret; -} - -static int -idxd_rawdev_probe_dsa(struct rte_dsa_device *dev) -{ - struct idxd_rawdev idxd = {{0}}; /* double {} to avoid error on BSD12 */ - int ret = 0; - - IOAT_PMD_INFO("Probing device %s on numa node %d", - dev->wq_name, dev->device.numa_node); - if (read_wq_int(dev, "size", &ret) < 0) - return -1; - idxd.max_batches = ret; - idxd.qid = dev->addr.wq_id; - idxd.u.vdev.dsa_id = dev->addr.device_id; - - idxd.public.portal = idxd_vdev_mmap_wq(dev); - if (idxd.public.portal == NULL) { - IOAT_PMD_ERR("WQ mmap failed"); - return -ENOENT; - } - - ret = idxd_rawdev_create(dev->wq_name, &dev->device, &idxd, &idxd_vdev_ops); - if (ret) { - IOAT_PMD_ERR("Failed to create rawdev %s", dev->wq_name); - return ret; - } - - return 0; -} - -static int -is_for_this_process_use(const char *name) -{ - char *runtime_dir = strdup(rte_eal_get_runtime_dir()); - char *prefix = basename(runtime_dir); - int prefixlen = strlen(prefix); - int retval = 0; - - if (strncmp(name, "dpdk_", 5) == 0) - retval = 1; - if (strncmp(name, prefix, prefixlen) == 0 && name[prefixlen] == '_') - retval = 1; - - free(runtime_dir); - return retval; -} - -static int -dsa_probe(void) -{ - struct rte_dsa_device *dev; - - TAILQ_FOREACH(dev, &dsa_bus.device_list, next) { - char type[64], name[64]; - - if (read_wq_string(dev, "type", type, sizeof(type)) < 0 || - read_wq_string(dev, "name", name, sizeof(name)) < 0) - continue; - - if (strncmp(type, "user", 4) == 0 && is_for_this_process_use(name)) { - dev->device.driver = &dsa_bus.driver; - idxd_rawdev_probe_dsa(dev); - continue; - } - IOAT_PMD_DEBUG("WQ '%s', not allocated to DPDK", dev->wq_name); - } - - return 0; -} - -static int -dsa_scan(void) -{ - const char *path = dsa_get_dev_path(); - struct dirent *wq; - DIR *dev_dir; - - dev_dir = opendir(path); - if (dev_dir == NULL) { - if (errno == ENOENT) - return 0; /* no bus, return without error */ - IOAT_PMD_ERR("%s(): opendir '%s' failed: %s", - __func__, path, strerror(errno)); - return -1; - } - - while ((wq = readdir(dev_dir)) != NULL) { - struct rte_dsa_device *dev; - int numa_node = -1; - - if (strncmp(wq->d_name, "wq", 2) != 0) - continue; - if (strnlen(wq->d_name, sizeof(dev->wq_name)) == sizeof(dev->wq_name)) { - IOAT_PMD_ERR("%s(): wq name too long: '%s', skipping", - __func__, wq->d_name); - continue; - } - IOAT_PMD_DEBUG("%s(): found %s/%s", __func__, path, wq->d_name); - - dev = malloc(sizeof(*dev)); - if (dsa_addr_parse(wq->d_name, &dev->addr) < 0) { - IOAT_PMD_ERR("Error parsing WQ name: %s", wq->d_name); - free(dev); - continue; - } - dev->device.bus = &dsa_bus.bus; - strlcpy(dev->wq_name, wq->d_name, sizeof(dev->wq_name)); - TAILQ_INSERT_TAIL(&dsa_bus.device_list, dev, next); - devcount++; - - read_device_int(dev, "numa_node", &numa_node); - dev->device.numa_node = numa_node; - dev->device.name = dev->wq_name; - } - - closedir(dev_dir); - return 0; -} - -static struct rte_device * -dsa_find_device(const struct rte_device *start, rte_dev_cmp_t cmp, - const void *data) -{ - struct rte_dsa_device *dev = TAILQ_FIRST(&dsa_bus.device_list); - - /* the rte_device struct must be at start of dsa structure */ - RTE_BUILD_BUG_ON(offsetof(struct rte_dsa_device, device) != 0); - - if (start != NULL) /* jump to start point if given */ - dev = TAILQ_NEXT((const struct rte_dsa_device *)start, next); - while (dev != NULL) { - if (cmp(&dev->device, data) == 0) - return &dev->device; - dev = TAILQ_NEXT(dev, next); - } - return NULL; -} - -static enum rte_iova_mode -dsa_get_iommu_class(void) -{ - /* if there are no devices, report don't care, otherwise VA mode */ - return devcount > 0 ? RTE_IOVA_VA : RTE_IOVA_DC; -} - -static int -dsa_addr_parse(const char *name, void *addr) -{ - struct dsa_wq_addr *wq = addr; - unsigned int device_id, wq_id; - - if (sscanf(name, "wq%u.%u", &device_id, &wq_id) != 2) { - IOAT_PMD_DEBUG("Parsing WQ name failed: %s", name); - return -1; - } - - wq->device_id = device_id; - wq->wq_id = wq_id; - return 0; -} - -RTE_REGISTER_BUS(dsa, dsa_bus.bus); diff --git a/drivers/raw/ioat/idxd_pci.c b/drivers/raw/ioat/idxd_pci.c deleted file mode 100644 index 13515dbc6c..0000000000 --- a/drivers/raw/ioat/idxd_pci.c +++ /dev/null @@ -1,380 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2020 Intel Corporation - */ - -#include -#include -#include - -#include "ioat_private.h" -#include "ioat_spec.h" - -#define IDXD_VENDOR_ID 0x8086 -#define IDXD_DEVICE_ID_SPR 0x0B25 - -#define IDXD_PMD_RAWDEV_NAME_PCI rawdev_idxd_pci - -const struct rte_pci_id pci_id_idxd_map[] = { - { RTE_PCI_DEVICE(IDXD_VENDOR_ID, IDXD_DEVICE_ID_SPR) }, - { .vendor_id = 0, /* sentinel */ }, -}; - -static inline int -idxd_pci_dev_command(struct idxd_rawdev *idxd, enum rte_idxd_cmds command) -{ - uint8_t err_code; - uint16_t qid = idxd->qid; - int i = 0; - - if (command >= idxd_disable_wq && command <= idxd_reset_wq) - qid = (1 << qid); - rte_spinlock_lock(&idxd->u.pci->lk); - idxd->u.pci->regs->cmd = (command << IDXD_CMD_SHIFT) | qid; - - do { - rte_pause(); - err_code = idxd->u.pci->regs->cmdstatus; - if (++i >= 1000) { - IOAT_PMD_ERR("Timeout waiting for command response from HW"); - rte_spinlock_unlock(&idxd->u.pci->lk); - return err_code; - } - } while (idxd->u.pci->regs->cmdstatus & CMDSTATUS_ACTIVE_MASK); - rte_spinlock_unlock(&idxd->u.pci->lk); - - return err_code & CMDSTATUS_ERR_MASK; -} - -static uint32_t * -idxd_get_wq_cfg(struct idxd_pci_common *pci, uint8_t wq_idx) -{ - return RTE_PTR_ADD(pci->wq_regs_base, - (uintptr_t)wq_idx << (5 + pci->wq_cfg_sz)); -} - -static int -idxd_is_wq_enabled(struct idxd_rawdev *idxd) -{ - uint32_t state = idxd_get_wq_cfg(idxd->u.pci, idxd->qid)[WQ_STATE_IDX]; - return ((state >> WQ_STATE_SHIFT) & WQ_STATE_MASK) == 0x1; -} - -static void -idxd_pci_dev_stop(struct rte_rawdev *dev) -{ - struct idxd_rawdev *idxd = dev->dev_private; - uint8_t err_code; - - if (!idxd_is_wq_enabled(idxd)) { - IOAT_PMD_ERR("Work queue %d already disabled", idxd->qid); - return; - } - - err_code = idxd_pci_dev_command(idxd, idxd_disable_wq); - if (err_code || idxd_is_wq_enabled(idxd)) { - IOAT_PMD_ERR("Failed disabling work queue %d, error code: %#x", - idxd->qid, err_code); - return; - } - IOAT_PMD_DEBUG("Work queue %d disabled OK", idxd->qid); -} - -static int -idxd_pci_dev_start(struct rte_rawdev *dev) -{ - struct idxd_rawdev *idxd = dev->dev_private; - uint8_t err_code; - - if (idxd_is_wq_enabled(idxd)) { - IOAT_PMD_WARN("WQ %d already enabled", idxd->qid); - return 0; - } - - if (idxd->public.desc_ring == NULL) { - IOAT_PMD_ERR("WQ %d has not been fully configured", idxd->qid); - return -EINVAL; - } - - err_code = idxd_pci_dev_command(idxd, idxd_enable_wq); - if (err_code || !idxd_is_wq_enabled(idxd)) { - IOAT_PMD_ERR("Failed enabling work queue %d, error code: %#x", - idxd->qid, err_code); - return err_code == 0 ? -1 : err_code; - } - - IOAT_PMD_DEBUG("Work queue %d enabled OK", idxd->qid); - - return 0; -} - -static const struct rte_rawdev_ops idxd_pci_ops = { - .dev_close = idxd_rawdev_close, - .dev_selftest = ioat_rawdev_test, - .dump = idxd_dev_dump, - .dev_configure = idxd_dev_configure, - .dev_start = idxd_pci_dev_start, - .dev_stop = idxd_pci_dev_stop, - .dev_info_get = idxd_dev_info_get, - .xstats_get = ioat_xstats_get, - .xstats_get_names = ioat_xstats_get_names, - .xstats_reset = ioat_xstats_reset, -}; - -/* each portal uses 4 x 4k pages */ -#define IDXD_PORTAL_SIZE (4096 * 4) - -static int -init_pci_device(struct rte_pci_device *dev, struct idxd_rawdev *idxd, - unsigned int max_queues) -{ - struct idxd_pci_common *pci; - uint8_t nb_groups, nb_engines, nb_wqs; - uint16_t grp_offset, wq_offset; /* how far into bar0 the regs are */ - uint16_t wq_size, total_wq_size; - uint8_t lg2_max_batch, lg2_max_copy_size; - unsigned int i, err_code; - - pci = malloc(sizeof(*pci)); - if (pci == NULL) { - IOAT_PMD_ERR("%s: Can't allocate memory", __func__); - goto err; - } - rte_spinlock_init(&pci->lk); - - /* assign the bar registers, and then configure device */ - pci->regs = dev->mem_resource[0].addr; - grp_offset = (uint16_t)pci->regs->offsets[0]; - pci->grp_regs = RTE_PTR_ADD(pci->regs, grp_offset * 0x100); - wq_offset = (uint16_t)(pci->regs->offsets[0] >> 16); - pci->wq_regs_base = RTE_PTR_ADD(pci->regs, wq_offset * 0x100); - pci->portals = dev->mem_resource[2].addr; - pci->wq_cfg_sz = (pci->regs->wqcap >> 24) & 0x0F; - - /* sanity check device status */ - if (pci->regs->gensts & GENSTS_DEV_STATE_MASK) { - /* need function-level-reset (FLR) or is enabled */ - IOAT_PMD_ERR("Device status is not disabled, cannot init"); - goto err; - } - if (pci->regs->cmdstatus & CMDSTATUS_ACTIVE_MASK) { - /* command in progress */ - IOAT_PMD_ERR("Device has a command in progress, cannot init"); - goto err; - } - - /* read basic info about the hardware for use when configuring */ - nb_groups = (uint8_t)pci->regs->grpcap; - nb_engines = (uint8_t)pci->regs->engcap; - nb_wqs = (uint8_t)(pci->regs->wqcap >> 16); - total_wq_size = (uint16_t)pci->regs->wqcap; - lg2_max_copy_size = (uint8_t)(pci->regs->gencap >> 16) & 0x1F; - lg2_max_batch = (uint8_t)(pci->regs->gencap >> 21) & 0x0F; - - IOAT_PMD_DEBUG("nb_groups = %u, nb_engines = %u, nb_wqs = %u", - nb_groups, nb_engines, nb_wqs); - - /* zero out any old config */ - for (i = 0; i < nb_groups; i++) { - pci->grp_regs[i].grpengcfg = 0; - pci->grp_regs[i].grpwqcfg[0] = 0; - } - for (i = 0; i < nb_wqs; i++) - idxd_get_wq_cfg(pci, i)[0] = 0; - - /* limit queues if necessary */ - if (max_queues != 0 && nb_wqs > max_queues) { - nb_wqs = max_queues; - if (nb_engines > max_queues) - nb_engines = max_queues; - if (nb_groups > max_queues) - nb_engines = max_queues; - IOAT_PMD_DEBUG("Limiting queues to %u", nb_wqs); - } - - /* put each engine into a separate group to avoid reordering */ - if (nb_groups > nb_engines) - nb_groups = nb_engines; - if (nb_groups < nb_engines) - nb_engines = nb_groups; - - /* assign engines to groups, round-robin style */ - for (i = 0; i < nb_engines; i++) { - IOAT_PMD_DEBUG("Assigning engine %u to group %u", - i, i % nb_groups); - pci->grp_regs[i % nb_groups].grpengcfg |= (1ULL << i); - } - - /* now do the same for queues and give work slots to each queue */ - wq_size = total_wq_size / nb_wqs; - IOAT_PMD_DEBUG("Work queue size = %u, max batch = 2^%u, max copy = 2^%u", - wq_size, lg2_max_batch, lg2_max_copy_size); - for (i = 0; i < nb_wqs; i++) { - /* add engine "i" to a group */ - IOAT_PMD_DEBUG("Assigning work queue %u to group %u", - i, i % nb_groups); - pci->grp_regs[i % nb_groups].grpwqcfg[0] |= (1ULL << i); - /* now configure it, in terms of size, max batch, mode */ - idxd_get_wq_cfg(pci, i)[WQ_SIZE_IDX] = wq_size; - idxd_get_wq_cfg(pci, i)[WQ_MODE_IDX] = (1 << WQ_PRIORITY_SHIFT) | - WQ_MODE_DEDICATED; - idxd_get_wq_cfg(pci, i)[WQ_SIZES_IDX] = lg2_max_copy_size | - (lg2_max_batch << WQ_BATCH_SZ_SHIFT); - } - - /* dump the group configuration to output */ - for (i = 0; i < nb_groups; i++) { - IOAT_PMD_DEBUG("## Group %d", i); - IOAT_PMD_DEBUG(" GRPWQCFG: %"PRIx64, pci->grp_regs[i].grpwqcfg[0]); - IOAT_PMD_DEBUG(" GRPENGCFG: %"PRIx64, pci->grp_regs[i].grpengcfg); - IOAT_PMD_DEBUG(" GRPFLAGS: %"PRIx32, pci->grp_regs[i].grpflags); - } - - idxd->u.pci = pci; - idxd->max_batches = wq_size; - - /* enable the device itself */ - err_code = idxd_pci_dev_command(idxd, idxd_enable_dev); - if (err_code) { - IOAT_PMD_ERR("Error enabling device: code %#x", err_code); - return err_code; - } - IOAT_PMD_DEBUG("IDXD Device enabled OK"); - - return nb_wqs; - -err: - free(pci); - return -1; -} - -static int -idxd_rawdev_probe_pci(struct rte_pci_driver *drv, struct rte_pci_device *dev) -{ - struct idxd_rawdev idxd = {{0}}; /* Double {} to avoid error on BSD12 */ - uint8_t nb_wqs; - int qid, ret = 0; - char name[PCI_PRI_STR_SIZE]; - unsigned int max_queues = 0; - - rte_pci_device_name(&dev->addr, name, sizeof(name)); - IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); - dev->device.driver = &drv->driver; - - if (dev->device.devargs && dev->device.devargs->args[0] != '\0') { - /* if the number of devargs grows beyond just 1, use rte_kvargs */ - if (sscanf(dev->device.devargs->args, - "max_queues=%u", &max_queues) != 1) { - IOAT_PMD_ERR("Invalid device parameter: '%s'", - dev->device.devargs->args); - return -1; - } - } - - ret = init_pci_device(dev, &idxd, max_queues); - if (ret < 0) { - IOAT_PMD_ERR("Error initializing PCI hardware"); - return ret; - } - nb_wqs = (uint8_t)ret; - - /* set up one device for each queue */ - for (qid = 0; qid < nb_wqs; qid++) { - char qname[32]; - - /* add the queue number to each device name */ - snprintf(qname, sizeof(qname), "%s-q%d", name, qid); - idxd.qid = qid; - idxd.public.portal = RTE_PTR_ADD(idxd.u.pci->portals, - qid * IDXD_PORTAL_SIZE); - if (idxd_is_wq_enabled(&idxd)) - IOAT_PMD_ERR("Error, WQ %u seems enabled", qid); - ret = idxd_rawdev_create(qname, &dev->device, - &idxd, &idxd_pci_ops); - if (ret != 0) { - IOAT_PMD_ERR("Failed to create rawdev %s", name); - if (qid == 0) /* if no devices using this, free pci */ - free(idxd.u.pci); - return ret; - } - } - - return 0; -} - -static int -idxd_rawdev_destroy(const char *name) -{ - int ret; - uint8_t err_code; - struct rte_rawdev *rdev; - struct idxd_rawdev *idxd; - - if (!name) { - IOAT_PMD_ERR("Invalid device name"); - return -EINVAL; - } - - rdev = rte_rawdev_pmd_get_named_dev(name); - if (!rdev) { - IOAT_PMD_ERR("Invalid device name (%s)", name); - return -EINVAL; - } - - idxd = rdev->dev_private; - if (!idxd) { - IOAT_PMD_ERR("Error getting dev_private"); - return -EINVAL; - } - - /* disable the device */ - err_code = idxd_pci_dev_command(idxd, idxd_disable_dev); - if (err_code) { - IOAT_PMD_ERR("Error disabling device: code %#x", err_code); - return err_code; - } - IOAT_PMD_DEBUG("IDXD Device disabled OK"); - - /* free device memory */ - IOAT_PMD_DEBUG("Freeing device driver memory"); - rdev->dev_private = NULL; - rte_free(idxd->public.batch_idx_ring); - rte_free(idxd->public.desc_ring); - rte_free(idxd->public.hdl_ring); - rte_memzone_free(idxd->mz); - - /* rte_rawdev_close is called by pmd_release */ - ret = rte_rawdev_pmd_release(rdev); - if (ret) - IOAT_PMD_DEBUG("Device cleanup failed"); - - return 0; -} - -static int -idxd_rawdev_remove_pci(struct rte_pci_device *dev) -{ - char name[PCI_PRI_STR_SIZE]; - int ret = 0; - - rte_pci_device_name(&dev->addr, name, sizeof(name)); - - IOAT_PMD_INFO("Closing %s on NUMA node %d", - name, dev->device.numa_node); - - ret = idxd_rawdev_destroy(name); - - return ret; -} - -struct rte_pci_driver idxd_pmd_drv_pci = { - .id_table = pci_id_idxd_map, - .drv_flags = RTE_PCI_DRV_NEED_MAPPING, - .probe = idxd_rawdev_probe_pci, - .remove = idxd_rawdev_remove_pci, -}; - -RTE_PMD_REGISTER_PCI(IDXD_PMD_RAWDEV_NAME_PCI, idxd_pmd_drv_pci); -RTE_PMD_REGISTER_PCI_TABLE(IDXD_PMD_RAWDEV_NAME_PCI, pci_id_idxd_map); -RTE_PMD_REGISTER_KMOD_DEP(IDXD_PMD_RAWDEV_NAME_PCI, - "* igb_uio | uio_pci_generic | vfio-pci"); -RTE_PMD_REGISTER_PARAM_STRING(rawdev_idxd_pci, "max_queues=0"); diff --git a/drivers/raw/ioat/ioat_common.c b/drivers/raw/ioat/ioat_common.c deleted file mode 100644 index e44e181e58..0000000000 --- a/drivers/raw/ioat/ioat_common.c +++ /dev/null @@ -1,273 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2020 Intel Corporation - */ - -#include -#include -#include -#include - -#include "ioat_private.h" - -RTE_LOG_REGISTER_DEFAULT(ioat_rawdev_logtype, INFO); - -static const char * const xstat_names[] = { - "failed_enqueues", "successful_enqueues", - "copies_started", "copies_completed" -}; - -int -ioat_xstats_get(const struct rte_rawdev *dev, const unsigned int ids[], - uint64_t values[], unsigned int n) -{ - const struct rte_ioat_rawdev *ioat = dev->dev_private; - const uint64_t *stats = (const void *)&ioat->xstats; - unsigned int i; - - for (i = 0; i < n; i++) { - if (ids[i] > sizeof(ioat->xstats)/sizeof(*stats)) - values[i] = 0; - else - values[i] = stats[ids[i]]; - } - return n; -} - -int -ioat_xstats_get_names(const struct rte_rawdev *dev, - struct rte_rawdev_xstats_name *names, - unsigned int size) -{ - unsigned int i; - - RTE_SET_USED(dev); - if (size < RTE_DIM(xstat_names)) - return RTE_DIM(xstat_names); - - for (i = 0; i < RTE_DIM(xstat_names); i++) - strlcpy(names[i].name, xstat_names[i], sizeof(names[i])); - - return RTE_DIM(xstat_names); -} - -int -ioat_xstats_reset(struct rte_rawdev *dev, const uint32_t *ids, uint32_t nb_ids) -{ - struct rte_ioat_rawdev *ioat = dev->dev_private; - uint64_t *stats = (void *)&ioat->xstats; - unsigned int i; - - if (!ids) { - memset(&ioat->xstats, 0, sizeof(ioat->xstats)); - return 0; - } - - for (i = 0; i < nb_ids; i++) - if (ids[i] < sizeof(ioat->xstats)/sizeof(*stats)) - stats[ids[i]] = 0; - - return 0; -} - -int -idxd_rawdev_close(struct rte_rawdev *dev __rte_unused) -{ - return 0; -} - -int -idxd_dev_dump(struct rte_rawdev *dev, FILE *f) -{ - struct idxd_rawdev *idxd = dev->dev_private; - struct rte_idxd_rawdev *rte_idxd = &idxd->public; - int i; - - fprintf(f, "Raw Device #%d\n", dev->dev_id); - fprintf(f, "Driver: %s\n\n", dev->driver_name); - - fprintf(f, "Portal: %p\n", rte_idxd->portal); - fprintf(f, "Config: {ring_size: %u, hdls_disable: %u}\n\n", - rte_idxd->cfg.ring_size, rte_idxd->cfg.hdls_disable); - - fprintf(f, "max batches: %u\n", rte_idxd->max_batches); - fprintf(f, "batch idx read: %u\n", rte_idxd->batch_idx_read); - fprintf(f, "batch idx write: %u\n", rte_idxd->batch_idx_write); - fprintf(f, "batch idxes:"); - for (i = 0; i < rte_idxd->max_batches + 1; i++) - fprintf(f, "%u ", rte_idxd->batch_idx_ring[i]); - fprintf(f, "\n\n"); - - fprintf(f, "hdls read: %u\n", rte_idxd->max_batches); - fprintf(f, "hdls avail: %u\n", rte_idxd->hdls_avail); - fprintf(f, "batch start: %u\n", rte_idxd->batch_start); - fprintf(f, "batch size: %u\n", rte_idxd->batch_size); - - return 0; -} - -int -idxd_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info, - size_t info_size) -{ - struct rte_ioat_rawdev_config *cfg = dev_info; - struct idxd_rawdev *idxd = dev->dev_private; - struct rte_idxd_rawdev *rte_idxd = &idxd->public; - - if (info_size != sizeof(*cfg)) - return -EINVAL; - - if (cfg != NULL) - *cfg = rte_idxd->cfg; - return 0; -} - -int -idxd_dev_configure(const struct rte_rawdev *dev, - rte_rawdev_obj_t config, size_t config_size) -{ - struct idxd_rawdev *idxd = dev->dev_private; - struct rte_idxd_rawdev *rte_idxd = &idxd->public; - struct rte_ioat_rawdev_config *cfg = config; - uint16_t max_desc = cfg->ring_size; - - if (config_size != sizeof(*cfg)) - return -EINVAL; - - if (dev->started) { - IOAT_PMD_ERR("%s: Error, device is started.", __func__); - return -EAGAIN; - } - - rte_idxd->cfg = *cfg; - - if (!rte_is_power_of_2(max_desc)) - max_desc = rte_align32pow2(max_desc); - IOAT_PMD_DEBUG("Rawdev %u using %u descriptors", - dev->dev_id, max_desc); - rte_idxd->desc_ring_mask = max_desc - 1; - - /* in case we are reconfiguring a device, free any existing memory */ - rte_free(rte_idxd->desc_ring); - rte_free(rte_idxd->hdl_ring); - rte_free(rte_idxd->hdl_ring_flags); - - /* allocate the descriptor ring at 2x size as batches can't wrap */ - rte_idxd->desc_ring = rte_zmalloc(NULL, - sizeof(*rte_idxd->desc_ring) * max_desc * 2, 0); - if (rte_idxd->desc_ring == NULL) - return -ENOMEM; - rte_idxd->desc_iova = rte_mem_virt2iova(rte_idxd->desc_ring); - - rte_idxd->hdl_ring = rte_zmalloc(NULL, - sizeof(*rte_idxd->hdl_ring) * max_desc, 0); - if (rte_idxd->hdl_ring == NULL) { - rte_free(rte_idxd->desc_ring); - rte_idxd->desc_ring = NULL; - return -ENOMEM; - } - rte_idxd->hdl_ring_flags = rte_zmalloc(NULL, - sizeof(*rte_idxd->hdl_ring_flags) * max_desc, 0); - if (rte_idxd->hdl_ring_flags == NULL) { - rte_free(rte_idxd->desc_ring); - rte_free(rte_idxd->hdl_ring); - rte_idxd->desc_ring = NULL; - rte_idxd->hdl_ring = NULL; - return -ENOMEM; - } - rte_idxd->hdls_read = rte_idxd->batch_start = 0; - rte_idxd->batch_size = 0; - rte_idxd->hdls_avail = 0; - - return 0; -} - -int -idxd_rawdev_create(const char *name, struct rte_device *dev, - const struct idxd_rawdev *base_idxd, - const struct rte_rawdev_ops *ops) -{ - struct idxd_rawdev *idxd; - struct rte_idxd_rawdev *public; - struct rte_rawdev *rawdev = NULL; - const struct rte_memzone *mz = NULL; - char mz_name[RTE_MEMZONE_NAMESIZE]; - int ret = 0; - - RTE_BUILD_BUG_ON(sizeof(struct rte_idxd_hw_desc) != 64); - RTE_BUILD_BUG_ON(offsetof(struct rte_idxd_hw_desc, size) != 32); - RTE_BUILD_BUG_ON(sizeof(struct rte_idxd_completion) != 32); - - if (!name) { - IOAT_PMD_ERR("Invalid name of the device!"); - ret = -EINVAL; - goto cleanup; - } - - /* Allocate device structure */ - rawdev = rte_rawdev_pmd_allocate(name, sizeof(struct idxd_rawdev), - dev->numa_node); - if (rawdev == NULL) { - IOAT_PMD_ERR("Unable to allocate raw device"); - ret = -ENOMEM; - goto cleanup; - } - - /* Allocate memory for the primary process or else return the memory - * of primary memzone for the secondary process. - */ - snprintf(mz_name, sizeof(mz_name), "rawdev%u_private", rawdev->dev_id); - if (rte_eal_process_type() == RTE_PROC_SECONDARY) { - mz = rte_memzone_lookup(mz_name); - if (mz == NULL) { - IOAT_PMD_ERR("Unable lookup memzone for private data\n"); - ret = -ENOMEM; - goto cleanup; - } - rawdev->dev_private = mz->addr; - rawdev->dev_ops = ops; - rawdev->device = dev; - return 0; - } - mz = rte_memzone_reserve(mz_name, sizeof(struct idxd_rawdev), - dev->numa_node, RTE_MEMZONE_IOVA_CONTIG); - if (mz == NULL) { - IOAT_PMD_ERR("Unable to reserve memzone for private data\n"); - ret = -ENOMEM; - goto cleanup; - } - rawdev->dev_private = mz->addr; - rawdev->dev_ops = ops; - rawdev->device = dev; - rawdev->driver_name = IOAT_PMD_RAWDEV_NAME_STR; - - idxd = rawdev->dev_private; - *idxd = *base_idxd; /* copy over the main fields already passed in */ - idxd->rawdev = rawdev; - idxd->mz = mz; - - public = &idxd->public; - public->type = RTE_IDXD_DEV; - public->max_batches = idxd->max_batches; - public->batch_idx_read = 0; - public->batch_idx_write = 0; - /* allocate batch index ring. The +1 is because we can never fully use - * the ring, otherwise read == write means both full and empty. - */ - public->batch_idx_ring = rte_zmalloc(NULL, - sizeof(uint16_t) * (idxd->max_batches + 1), 0); - if (public->batch_idx_ring == NULL) { - IOAT_PMD_ERR("Unable to reserve memory for batch data\n"); - ret = -ENOMEM; - goto cleanup; - } - - return 0; - -cleanup: - if (mz) - rte_memzone_free(mz); - if (rawdev) - rte_rawdev_pmd_release(rawdev); - - return ret; -} diff --git a/drivers/raw/ioat/ioat_private.h b/drivers/raw/ioat/ioat_private.h deleted file mode 100644 index 7fb685a3d7..0000000000 --- a/drivers/raw/ioat/ioat_private.h +++ /dev/null @@ -1,84 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2020 Intel Corporation - */ - -#ifndef _IOAT_PRIVATE_H_ -#define _IOAT_PRIVATE_H_ - -/** - * @file idxd_private.h - * - * Private data structures for the idxd/DSA part of ioat device driver - * - * @warning - * @b EXPERIMENTAL: these structures and APIs may change without prior notice - */ - -#include -#include -#include "rte_ioat_rawdev.h" - -extern int ioat_rawdev_logtype; - -#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ - ioat_rawdev_logtype, "IOAT: %s(): " fmt "\n", __func__, ##args) - -#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) -#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) -#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) -#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) - -struct idxd_pci_common { - rte_spinlock_t lk; - - uint8_t wq_cfg_sz; - volatile struct rte_idxd_bar0 *regs; - volatile uint32_t *wq_regs_base; - volatile struct rte_idxd_grpcfg *grp_regs; - volatile void *portals; -}; - -struct idxd_rawdev { - struct rte_idxd_rawdev public; /* the public members, must be first */ - - struct rte_rawdev *rawdev; - const struct rte_memzone *mz; - uint8_t qid; - uint16_t max_batches; - - union { - struct { - unsigned int dsa_id; - } vdev; - - struct idxd_pci_common *pci; - } u; -}; - -int ioat_xstats_get(const struct rte_rawdev *dev, const unsigned int ids[], - uint64_t values[], unsigned int n); - -int ioat_xstats_get_names(const struct rte_rawdev *dev, - struct rte_rawdev_xstats_name *names, - unsigned int size); - -int ioat_xstats_reset(struct rte_rawdev *dev, const uint32_t *ids, - uint32_t nb_ids); - -extern int ioat_rawdev_test(uint16_t dev_id); - -extern int idxd_rawdev_create(const char *name, struct rte_device *dev, - const struct idxd_rawdev *idxd, - const struct rte_rawdev_ops *ops); - -extern int idxd_rawdev_close(struct rte_rawdev *dev); - -extern int idxd_dev_configure(const struct rte_rawdev *dev, - rte_rawdev_obj_t config, size_t config_size); - -extern int idxd_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info, - size_t info_size); - -extern int idxd_dev_dump(struct rte_rawdev *dev, FILE *f); - -#endif /* _IOAT_PRIVATE_H_ */ diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c deleted file mode 100644 index 11341fcf5d..0000000000 --- a/drivers/raw/ioat/ioat_rawdev.c +++ /dev/null @@ -1,332 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2019 Intel Corporation - */ - -#include -#include -#include -#include -#include - -#include "rte_ioat_rawdev.h" -#include "ioat_spec.h" -#include "ioat_private.h" - -static struct rte_pci_driver ioat_pmd_drv; - -#define IOAT_VENDOR_ID 0x8086 -#define IOAT_DEVICE_ID_SKX 0x2021 -#define IOAT_DEVICE_ID_BDX0 0x6f20 -#define IOAT_DEVICE_ID_BDX1 0x6f21 -#define IOAT_DEVICE_ID_BDX2 0x6f22 -#define IOAT_DEVICE_ID_BDX3 0x6f23 -#define IOAT_DEVICE_ID_BDX4 0x6f24 -#define IOAT_DEVICE_ID_BDX5 0x6f25 -#define IOAT_DEVICE_ID_BDX6 0x6f26 -#define IOAT_DEVICE_ID_BDX7 0x6f27 -#define IOAT_DEVICE_ID_BDXE 0x6f2E -#define IOAT_DEVICE_ID_BDXF 0x6f2F -#define IOAT_DEVICE_ID_ICX 0x0b00 - -#define DESC_SZ sizeof(struct rte_ioat_generic_hw_desc) -#define COMPLETION_SZ sizeof(__m128i) - -static int -ioat_dev_configure(const struct rte_rawdev *dev, rte_rawdev_obj_t config, - size_t config_size) -{ - struct rte_ioat_rawdev_config *params = config; - struct rte_ioat_rawdev *ioat = dev->dev_private; - char mz_name[RTE_MEMZONE_NAMESIZE]; - unsigned short i; - - if (dev->started) - return -EBUSY; - - if (params == NULL || config_size != sizeof(*params)) - return -EINVAL; - - if (params->ring_size > 4096 || params->ring_size < 64 || - !rte_is_power_of_2(params->ring_size)) - return -EINVAL; - - ioat->ring_size = params->ring_size; - ioat->hdls_disable = params->hdls_disable; - if (ioat->desc_ring != NULL) { - rte_memzone_free(ioat->desc_mz); - ioat->desc_ring = NULL; - ioat->desc_mz = NULL; - } - - /* allocate one block of memory for both descriptors - * and completion handles. - */ - snprintf(mz_name, sizeof(mz_name), "rawdev%u_desc_ring", dev->dev_id); - ioat->desc_mz = rte_memzone_reserve(mz_name, - (DESC_SZ + COMPLETION_SZ) * ioat->ring_size, - dev->device->numa_node, RTE_MEMZONE_IOVA_CONTIG); - if (ioat->desc_mz == NULL) - return -ENOMEM; - ioat->desc_ring = ioat->desc_mz->addr; - ioat->hdls = (void *)&ioat->desc_ring[ioat->ring_size]; - - ioat->ring_addr = ioat->desc_mz->iova; - - /* configure descriptor ring - each one points to next */ - for (i = 0; i < ioat->ring_size; i++) { - ioat->desc_ring[i].next = ioat->ring_addr + - (((i + 1) % ioat->ring_size) * DESC_SZ); - } - - return 0; -} - -static int -ioat_dev_start(struct rte_rawdev *dev) -{ - struct rte_ioat_rawdev *ioat = dev->dev_private; - - if (ioat->ring_size == 0 || ioat->desc_ring == NULL) - return -EBUSY; - - /* inform hardware of where the descriptor ring is */ - ioat->regs->chainaddr = ioat->ring_addr; - /* inform hardware of where to write the status/completions */ - ioat->regs->chancmp = ioat->status_addr; - - /* prime the status register to be set to the last element */ - ioat->status = ioat->ring_addr + ((ioat->ring_size - 1) * DESC_SZ); - return 0; -} - -static void -ioat_dev_stop(struct rte_rawdev *dev) -{ - RTE_SET_USED(dev); -} - -static int -ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info, - size_t dev_info_size) -{ - struct rte_ioat_rawdev_config *cfg = dev_info; - struct rte_ioat_rawdev *ioat = dev->dev_private; - - if (dev_info == NULL || dev_info_size != sizeof(*cfg)) - return -EINVAL; - - cfg->ring_size = ioat->ring_size; - cfg->hdls_disable = ioat->hdls_disable; - return 0; -} - -static int -ioat_dev_close(struct rte_rawdev *dev __rte_unused) -{ - return 0; -} - -static int -ioat_rawdev_create(const char *name, struct rte_pci_device *dev) -{ - static const struct rte_rawdev_ops ioat_rawdev_ops = { - .dev_configure = ioat_dev_configure, - .dev_start = ioat_dev_start, - .dev_stop = ioat_dev_stop, - .dev_close = ioat_dev_close, - .dev_info_get = ioat_dev_info_get, - .xstats_get = ioat_xstats_get, - .xstats_get_names = ioat_xstats_get_names, - .xstats_reset = ioat_xstats_reset, - .dev_selftest = ioat_rawdev_test, - }; - - struct rte_rawdev *rawdev = NULL; - struct rte_ioat_rawdev *ioat = NULL; - const struct rte_memzone *mz = NULL; - char mz_name[RTE_MEMZONE_NAMESIZE]; - int ret = 0; - int retry = 0; - - if (!name) { - IOAT_PMD_ERR("Invalid name of the device!"); - ret = -EINVAL; - goto cleanup; - } - - /* Allocate device structure */ - rawdev = rte_rawdev_pmd_allocate(name, sizeof(struct rte_ioat_rawdev), - dev->device.numa_node); - if (rawdev == NULL) { - IOAT_PMD_ERR("Unable to allocate raw device"); - ret = -ENOMEM; - goto cleanup; - } - - /* Allocate memory for the primary process or else return the memory - * of primary memzone for the secondary process. - */ - snprintf(mz_name, sizeof(mz_name), "rawdev%u_private", rawdev->dev_id); - if (rte_eal_process_type() == RTE_PROC_SECONDARY) { - mz = rte_memzone_lookup(mz_name); - if (mz == NULL) { - IOAT_PMD_ERR("Unable lookup memzone for private data\n"); - ret = -ENOMEM; - goto cleanup; - } - rawdev->dev_private = mz->addr; - rawdev->dev_ops = &ioat_rawdev_ops; - rawdev->device = &dev->device; - rawdev->driver_name = dev->device.driver->name; - return 0; - } - mz = rte_memzone_reserve(mz_name, sizeof(struct rte_ioat_rawdev), - dev->device.numa_node, RTE_MEMZONE_IOVA_CONTIG); - if (mz == NULL) { - IOAT_PMD_ERR("Unable to reserve memzone for private data\n"); - ret = -ENOMEM; - goto cleanup; - } - - rawdev->dev_private = mz->addr; - rawdev->dev_ops = &ioat_rawdev_ops; - rawdev->device = &dev->device; - rawdev->driver_name = dev->device.driver->name; - - ioat = rawdev->dev_private; - ioat->type = RTE_IOAT_DEV; - ioat->rawdev = rawdev; - ioat->mz = mz; - ioat->regs = dev->mem_resource[0].addr; - ioat->doorbell = &ioat->regs->dmacount; - ioat->ring_size = 0; - ioat->desc_ring = NULL; - ioat->status_addr = ioat->mz->iova + - offsetof(struct rte_ioat_rawdev, status); - - /* do device initialization - reset and set error behaviour */ - if (ioat->regs->chancnt != 1) - IOAT_PMD_ERR("%s: Channel count == %d\n", __func__, - ioat->regs->chancnt); - - if (ioat->regs->chanctrl & 0x100) { /* locked by someone else */ - IOAT_PMD_WARN("%s: Channel appears locked\n", __func__); - ioat->regs->chanctrl = 0; - } - - ioat->regs->chancmd = RTE_IOAT_CHANCMD_SUSPEND; - rte_delay_ms(1); - ioat->regs->chancmd = RTE_IOAT_CHANCMD_RESET; - rte_delay_ms(1); - while (ioat->regs->chancmd & RTE_IOAT_CHANCMD_RESET) { - ioat->regs->chainaddr = 0; - rte_delay_ms(1); - if (++retry >= 200) { - IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=0x%"PRIx8", CHANSTS=0x%"PRIx64", CHANERR=0x%"PRIx32"\n", - __func__, - ioat->regs->chancmd, - ioat->regs->chansts, - ioat->regs->chanerr); - ret = -EIO; - } - } - ioat->regs->chanctrl = RTE_IOAT_CHANCTRL_ANY_ERR_ABORT_EN | - RTE_IOAT_CHANCTRL_ERR_COMPLETION_EN; - - return 0; - -cleanup: - if (rawdev) - rte_rawdev_pmd_release(rawdev); - - return ret; -} - -static int -ioat_rawdev_destroy(const char *name) -{ - int ret; - struct rte_rawdev *rdev; - - if (!name) { - IOAT_PMD_ERR("Invalid device name"); - return -EINVAL; - } - - rdev = rte_rawdev_pmd_get_named_dev(name); - if (!rdev) { - IOAT_PMD_ERR("Invalid device name (%s)", name); - return -EINVAL; - } - - if (rdev->dev_private != NULL) { - struct rte_ioat_rawdev *ioat = rdev->dev_private; - rdev->dev_private = NULL; - rte_memzone_free(ioat->desc_mz); - rte_memzone_free(ioat->mz); - } - - /* rte_rawdev_close is called by pmd_release */ - ret = rte_rawdev_pmd_release(rdev); - if (ret) - IOAT_PMD_DEBUG("Device cleanup failed"); - - return 0; -} - -static int -ioat_rawdev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) -{ - char name[32]; - int ret = 0; - - - rte_pci_device_name(&dev->addr, name, sizeof(name)); - IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); - - dev->device.driver = &drv->driver; - ret = ioat_rawdev_create(name, dev); - return ret; -} - -static int -ioat_rawdev_remove(struct rte_pci_device *dev) -{ - char name[32]; - int ret; - - rte_pci_device_name(&dev->addr, name, sizeof(name)); - - IOAT_PMD_INFO("Closing %s on NUMA node %d", - name, dev->device.numa_node); - - ret = ioat_rawdev_destroy(name); - return ret; -} - -static const struct rte_pci_id pci_id_ioat_map[] = { - { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_SKX) }, - { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX0) }, - { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX1) }, - { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX2) }, - { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX3) }, - { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX4) }, - { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX5) }, - { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX6) }, - { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDX7) }, - { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXE) }, - { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_BDXF) }, - { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID_ICX) }, - { .vendor_id = 0, /* sentinel */ }, -}; - -static struct rte_pci_driver ioat_pmd_drv = { - .id_table = pci_id_ioat_map, - .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC, - .probe = ioat_rawdev_probe, - .remove = ioat_rawdev_remove, -}; - -RTE_PMD_REGISTER_PCI(IOAT_PMD_RAWDEV_NAME, ioat_pmd_drv); -RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_RAWDEV_NAME, pci_id_ioat_map); -RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_RAWDEV_NAME, "* igb_uio | uio_pci_generic"); diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c deleted file mode 100644 index 5e33669699..0000000000 --- a/drivers/raw/ioat/ioat_rawdev_test.c +++ /dev/null @@ -1,734 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2019 Intel Corporation - */ - -#include -#include -#include -#include "rte_rawdev.h" -#include "rte_ioat_rawdev.h" -#include "ioat_private.h" - -#define MAX_SUPPORTED_RAWDEVS 64 -#define TEST_SKIPPED 77 -#define COPY_LEN 1024 - -int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */ - -static struct rte_mempool *pool; -static unsigned short expected_ring_size[MAX_SUPPORTED_RAWDEVS]; - -#define PRINT_ERR(...) print_err(__func__, __LINE__, __VA_ARGS__) - -static inline int -__rte_format_printf(3, 4) -print_err(const char *func, int lineno, const char *format, ...) -{ - va_list ap; - int ret; - - ret = fprintf(stderr, "In %s:%d - ", func, lineno); - va_start(ap, format); - ret += vfprintf(stderr, format, ap); - va_end(ap); - - return ret; -} - -static int -do_multi_copies(int dev_id, int split_batches, int split_completions) -{ - struct rte_mbuf *srcs[32], *dsts[32]; - struct rte_mbuf *completed_src[64]; - struct rte_mbuf *completed_dst[64]; - unsigned int i, j; - - for (i = 0; i < RTE_DIM(srcs); i++) { - char *src_data; - - if (split_batches && i == RTE_DIM(srcs) / 2) - rte_ioat_perform_ops(dev_id); - - srcs[i] = rte_pktmbuf_alloc(pool); - dsts[i] = rte_pktmbuf_alloc(pool); - src_data = rte_pktmbuf_mtod(srcs[i], char *); - - for (j = 0; j < COPY_LEN; j++) - src_data[j] = rand() & 0xFF; - - if (rte_ioat_enqueue_copy(dev_id, - srcs[i]->buf_iova + srcs[i]->data_off, - dsts[i]->buf_iova + dsts[i]->data_off, - COPY_LEN, - (uintptr_t)srcs[i], - (uintptr_t)dsts[i]) != 1) { - PRINT_ERR("Error with rte_ioat_enqueue_copy for buffer %u\n", - i); - return -1; - } - } - rte_ioat_perform_ops(dev_id); - usleep(100); - - if (split_completions) { - /* gather completions in two halves */ - uint16_t half_len = RTE_DIM(srcs) / 2; - if (rte_ioat_completed_ops(dev_id, half_len, NULL, NULL, - (void *)completed_src, - (void *)completed_dst) != half_len) { - PRINT_ERR("Error with rte_ioat_completed_ops - first half request\n"); - rte_rawdev_dump(dev_id, stdout); - return -1; - } - if (rte_ioat_completed_ops(dev_id, half_len, NULL, NULL, - (void *)&completed_src[half_len], - (void *)&completed_dst[half_len]) != half_len) { - PRINT_ERR("Error with rte_ioat_completed_ops - second half request\n"); - rte_rawdev_dump(dev_id, stdout); - return -1; - } - } else { - /* gather all completions in one go */ - if (rte_ioat_completed_ops(dev_id, RTE_DIM(completed_src), NULL, NULL, - (void *)completed_src, - (void *)completed_dst) != RTE_DIM(srcs)) { - PRINT_ERR("Error with rte_ioat_completed_ops\n"); - rte_rawdev_dump(dev_id, stdout); - return -1; - } - } - for (i = 0; i < RTE_DIM(srcs); i++) { - char *src_data, *dst_data; - - if (completed_src[i] != srcs[i]) { - PRINT_ERR("Error with source pointer %u\n", i); - return -1; - } - if (completed_dst[i] != dsts[i]) { - PRINT_ERR("Error with dest pointer %u\n", i); - return -1; - } - - src_data = rte_pktmbuf_mtod(srcs[i], char *); - dst_data = rte_pktmbuf_mtod(dsts[i], char *); - for (j = 0; j < COPY_LEN; j++) - if (src_data[j] != dst_data[j]) { - PRINT_ERR("Error with copy of packet %u, byte %u\n", - i, j); - return -1; - } - rte_pktmbuf_free(srcs[i]); - rte_pktmbuf_free(dsts[i]); - } - return 0; -} - -static int -test_enqueue_copies(int dev_id) -{ - unsigned int i; - - /* test doing a single copy */ - do { - struct rte_mbuf *src, *dst; - char *src_data, *dst_data; - struct rte_mbuf *completed[2] = {0}; - - src = rte_pktmbuf_alloc(pool); - dst = rte_pktmbuf_alloc(pool); - src_data = rte_pktmbuf_mtod(src, char *); - dst_data = rte_pktmbuf_mtod(dst, char *); - - for (i = 0; i < COPY_LEN; i++) - src_data[i] = rand() & 0xFF; - - if (rte_ioat_enqueue_copy(dev_id, - src->buf_iova + src->data_off, - dst->buf_iova + dst->data_off, - COPY_LEN, - (uintptr_t)src, - (uintptr_t)dst) != 1) { - PRINT_ERR("Error with rte_ioat_enqueue_copy\n"); - return -1; - } - rte_ioat_perform_ops(dev_id); - usleep(10); - - if (rte_ioat_completed_ops(dev_id, 1, NULL, NULL, (void *)&completed[0], - (void *)&completed[1]) != 1) { - PRINT_ERR("Error with rte_ioat_completed_ops\n"); - return -1; - } - if (completed[0] != src || completed[1] != dst) { - PRINT_ERR("Error with completions: got (%p, %p), not (%p,%p)\n", - completed[0], completed[1], src, dst); - return -1; - } - - for (i = 0; i < COPY_LEN; i++) - if (dst_data[i] != src_data[i]) { - PRINT_ERR("Data mismatch at char %u [Got %02x not %02x]\n", - i, dst_data[i], src_data[i]); - return -1; - } - rte_pktmbuf_free(src); - rte_pktmbuf_free(dst); - - /* check ring is now empty */ - if (rte_ioat_completed_ops(dev_id, 1, NULL, NULL, (void *)&completed[0], - (void *)&completed[1]) != 0) { - PRINT_ERR("Error: got unexpected returned handles from rte_ioat_completed_ops\n"); - return -1; - } - } while (0); - - /* test doing a multiple single copies */ - do { - const uint16_t max_ops = 4; - struct rte_mbuf *src, *dst; - char *src_data, *dst_data; - struct rte_mbuf *completed[32] = {0}; - const uint16_t max_completions = RTE_DIM(completed) / 2; - - src = rte_pktmbuf_alloc(pool); - dst = rte_pktmbuf_alloc(pool); - src_data = rte_pktmbuf_mtod(src, char *); - dst_data = rte_pktmbuf_mtod(dst, char *); - - for (i = 0; i < COPY_LEN; i++) - src_data[i] = rand() & 0xFF; - - /* perform the same copy times */ - for (i = 0; i < max_ops; i++) { - if (rte_ioat_enqueue_copy(dev_id, - src->buf_iova + src->data_off, - dst->buf_iova + dst->data_off, - COPY_LEN, - (uintptr_t)src, - (uintptr_t)dst) != 1) { - PRINT_ERR("Error with rte_ioat_enqueue_copy\n"); - return -1; - } - rte_ioat_perform_ops(dev_id); - } - usleep(10); - - if (rte_ioat_completed_ops(dev_id, max_completions, NULL, NULL, - (void *)&completed[0], - (void *)&completed[max_completions]) != max_ops) { - PRINT_ERR("Error with rte_ioat_completed_ops\n"); - rte_rawdev_dump(dev_id, stdout); - return -1; - } - if (completed[0] != src || completed[max_completions] != dst) { - PRINT_ERR("Error with completions: got (%p, %p), not (%p,%p)\n", - completed[0], completed[max_completions], src, dst); - return -1; - } - - for (i = 0; i < COPY_LEN; i++) - if (dst_data[i] != src_data[i]) { - PRINT_ERR("Data mismatch at char %u\n", i); - return -1; - } - rte_pktmbuf_free(src); - rte_pktmbuf_free(dst); - } while (0); - - /* test doing multiple copies */ - do_multi_copies(dev_id, 0, 0); /* enqueue and complete one batch at a time */ - do_multi_copies(dev_id, 1, 0); /* enqueue 2 batches and then complete both */ - do_multi_copies(dev_id, 0, 1); /* enqueue 1 batch, then complete in two halves */ - return 0; -} - -static int -test_enqueue_fill(int dev_id) -{ - const unsigned int lengths[] = {8, 64, 1024, 50, 100, 89}; - struct rte_mbuf *dst = rte_pktmbuf_alloc(pool); - char *dst_data = rte_pktmbuf_mtod(dst, char *); - struct rte_mbuf *completed[2] = {0}; - uint64_t pattern = 0xfedcba9876543210; - unsigned int i, j; - - for (i = 0; i < RTE_DIM(lengths); i++) { - /* reset dst_data */ - memset(dst_data, 0, lengths[i]); - - /* perform the fill operation */ - if (rte_ioat_enqueue_fill(dev_id, pattern, - dst->buf_iova + dst->data_off, lengths[i], - (uintptr_t)dst) != 1) { - PRINT_ERR("Error with rte_ioat_enqueue_fill\n"); - return -1; - } - - rte_ioat_perform_ops(dev_id); - usleep(100); - - if (rte_ioat_completed_ops(dev_id, 1, NULL, NULL, (void *)&completed[0], - (void *)&completed[1]) != 1) { - PRINT_ERR("Error with completed ops\n"); - return -1; - } - /* check the result */ - for (j = 0; j < lengths[i]; j++) { - char pat_byte = ((char *)&pattern)[j % 8]; - if (dst_data[j] != pat_byte) { - PRINT_ERR("Error with fill operation (lengths = %u): got (%x), not (%x)\n", - lengths[i], dst_data[j], pat_byte); - return -1; - } - } - } - - rte_pktmbuf_free(dst); - return 0; -} - -static int -test_burst_capacity(int dev_id) -{ -#define BURST_SIZE 64 - const unsigned int ring_space = rte_ioat_burst_capacity(dev_id); - struct rte_mbuf *src, *dst; - unsigned int length = 1024; - unsigned int i, j, iter; - unsigned int old_cap, cap; - uintptr_t completions[BURST_SIZE]; - - src = rte_pktmbuf_alloc(pool); - dst = rte_pktmbuf_alloc(pool); - - old_cap = ring_space; - /* to test capacity, we enqueue elements and check capacity is reduced - * by one each time - rebaselining the expected value after each burst - * as the capacity is only for a burst. We enqueue multiple bursts to - * fill up half the ring, before emptying it again. We do this twice to - * ensure that we get to test scenarios where we get ring wrap-around - */ - for (iter = 0; iter < 2; iter++) { - for (i = 0; i < ring_space / (2 * BURST_SIZE); i++) { - cap = rte_ioat_burst_capacity(dev_id); - if (cap > old_cap) { - PRINT_ERR("Error, avail ring capacity has gone up, not down\n"); - return -1; - } - old_cap = cap; - - for (j = 0; j < BURST_SIZE; j++) { - if (rte_ioat_enqueue_copy(dev_id, rte_pktmbuf_iova(src), - rte_pktmbuf_iova(dst), length, 0, 0) != 1) { - PRINT_ERR("Error with rte_ioat_enqueue_copy\n"); - return -1; - } - if (cap - rte_ioat_burst_capacity(dev_id) != j + 1) { - PRINT_ERR("Error, ring capacity did not change as expected\n"); - return -1; - } - } - rte_ioat_perform_ops(dev_id); - } - usleep(100); - for (i = 0; i < ring_space / (2 * BURST_SIZE); i++) { - if (rte_ioat_completed_ops(dev_id, BURST_SIZE, - NULL, NULL, - completions, completions) != BURST_SIZE) { - PRINT_ERR("Error with completions\n"); - return -1; - } - } - if (rte_ioat_burst_capacity(dev_id) != ring_space) { - PRINT_ERR("Error, ring capacity has not reset to original value\n"); - return -1; - } - old_cap = ring_space; - } - - rte_pktmbuf_free(src); - rte_pktmbuf_free(dst); - - return 0; -} - -static int -test_completion_status(int dev_id) -{ -#define COMP_BURST_SZ 16 - const unsigned int fail_copy[] = {0, 7, 15}; - struct rte_mbuf *srcs[COMP_BURST_SZ], *dsts[COMP_BURST_SZ]; - struct rte_mbuf *completed_src[COMP_BURST_SZ * 2]; - struct rte_mbuf *completed_dst[COMP_BURST_SZ * 2]; - unsigned int length = 1024; - unsigned int i; - uint8_t not_ok = 0; - - /* Test single full batch statuses */ - for (i = 0; i < RTE_DIM(fail_copy); i++) { - uint32_t status[COMP_BURST_SZ] = {0}; - unsigned int j; - - for (j = 0; j < COMP_BURST_SZ; j++) { - srcs[j] = rte_pktmbuf_alloc(pool); - dsts[j] = rte_pktmbuf_alloc(pool); - - if (rte_ioat_enqueue_copy(dev_id, - (j == fail_copy[i] ? (phys_addr_t)NULL : - (srcs[j]->buf_iova + srcs[j]->data_off)), - dsts[j]->buf_iova + dsts[j]->data_off, - length, - (uintptr_t)srcs[j], - (uintptr_t)dsts[j]) != 1) { - PRINT_ERR("Error with rte_ioat_enqueue_copy for buffer %u\n", j); - return -1; - } - } - rte_ioat_perform_ops(dev_id); - usleep(100); - - if (rte_ioat_completed_ops(dev_id, COMP_BURST_SZ, status, ¬_ok, - (void *)completed_src, (void *)completed_dst) != COMP_BURST_SZ) { - PRINT_ERR("Error with rte_ioat_completed_ops\n"); - rte_rawdev_dump(dev_id, stdout); - return -1; - } - if (not_ok != 1 || status[fail_copy[i]] == RTE_IOAT_OP_SUCCESS) { - unsigned int j; - PRINT_ERR("Error, missing expected failed copy, %u\n", fail_copy[i]); - for (j = 0; j < COMP_BURST_SZ; j++) - printf("%u ", status[j]); - printf("<-- Statuses\n"); - return -1; - } - for (j = 0; j < COMP_BURST_SZ; j++) { - rte_pktmbuf_free(completed_src[j]); - rte_pktmbuf_free(completed_dst[j]); - } - } - - /* Test gathering status for two batches at once */ - for (i = 0; i < RTE_DIM(fail_copy); i++) { - uint32_t status[COMP_BURST_SZ] = {0}; - unsigned int batch, j; - unsigned int expected_failures = 0; - - for (batch = 0; batch < 2; batch++) { - for (j = 0; j < COMP_BURST_SZ/2; j++) { - srcs[j] = rte_pktmbuf_alloc(pool); - dsts[j] = rte_pktmbuf_alloc(pool); - - if (j == fail_copy[i]) - expected_failures++; - if (rte_ioat_enqueue_copy(dev_id, - (j == fail_copy[i] ? (phys_addr_t)NULL : - (srcs[j]->buf_iova + srcs[j]->data_off)), - dsts[j]->buf_iova + dsts[j]->data_off, - length, - (uintptr_t)srcs[j], - (uintptr_t)dsts[j]) != 1) { - PRINT_ERR("Error with rte_ioat_enqueue_copy for buffer %u\n", - j); - return -1; - } - } - rte_ioat_perform_ops(dev_id); - } - usleep(100); - - if (rte_ioat_completed_ops(dev_id, COMP_BURST_SZ, status, ¬_ok, - (void *)completed_src, (void *)completed_dst) != COMP_BURST_SZ) { - PRINT_ERR("Error with rte_ioat_completed_ops\n"); - rte_rawdev_dump(dev_id, stdout); - return -1; - } - if (not_ok != expected_failures) { - unsigned int j; - PRINT_ERR("Error, missing expected failed copy, got %u, not %u\n", - not_ok, expected_failures); - for (j = 0; j < COMP_BURST_SZ; j++) - printf("%u ", status[j]); - printf("<-- Statuses\n"); - return -1; - } - for (j = 0; j < COMP_BURST_SZ; j++) { - rte_pktmbuf_free(completed_src[j]); - rte_pktmbuf_free(completed_dst[j]); - } - } - - /* Test gathering status for half batch at a time */ - for (i = 0; i < RTE_DIM(fail_copy); i++) { - uint32_t status[COMP_BURST_SZ] = {0}; - unsigned int j; - - for (j = 0; j < COMP_BURST_SZ; j++) { - srcs[j] = rte_pktmbuf_alloc(pool); - dsts[j] = rte_pktmbuf_alloc(pool); - - if (rte_ioat_enqueue_copy(dev_id, - (j == fail_copy[i] ? (phys_addr_t)NULL : - (srcs[j]->buf_iova + srcs[j]->data_off)), - dsts[j]->buf_iova + dsts[j]->data_off, - length, - (uintptr_t)srcs[j], - (uintptr_t)dsts[j]) != 1) { - PRINT_ERR("Error with rte_ioat_enqueue_copy for buffer %u\n", j); - return -1; - } - } - rte_ioat_perform_ops(dev_id); - usleep(100); - - if (rte_ioat_completed_ops(dev_id, COMP_BURST_SZ / 2, status, ¬_ok, - (void *)completed_src, - (void *)completed_dst) != (COMP_BURST_SZ / 2)) { - PRINT_ERR("Error with rte_ioat_completed_ops\n"); - rte_rawdev_dump(dev_id, stdout); - return -1; - } - if (fail_copy[i] < COMP_BURST_SZ / 2 && - (not_ok != 1 || status[fail_copy[i]] == RTE_IOAT_OP_SUCCESS)) { - PRINT_ERR("Missing expected failure in first half-batch\n"); - rte_rawdev_dump(dev_id, stdout); - return -1; - } - if (rte_ioat_completed_ops(dev_id, COMP_BURST_SZ / 2, status, ¬_ok, - (void *)&completed_src[COMP_BURST_SZ / 2], - (void *)&completed_dst[COMP_BURST_SZ / 2]) != (COMP_BURST_SZ / 2)) { - PRINT_ERR("Error with rte_ioat_completed_ops\n"); - rte_rawdev_dump(dev_id, stdout); - return -1; - } - if (fail_copy[i] >= COMP_BURST_SZ / 2 && (not_ok != 1 || - status[fail_copy[i] - (COMP_BURST_SZ / 2)] - == RTE_IOAT_OP_SUCCESS)) { - PRINT_ERR("Missing expected failure in second half-batch\n"); - rte_rawdev_dump(dev_id, stdout); - return -1; - } - - for (j = 0; j < COMP_BURST_SZ; j++) { - rte_pktmbuf_free(completed_src[j]); - rte_pktmbuf_free(completed_dst[j]); - } - } - - /* Test gathering statuses with fence */ - for (i = 1; i < RTE_DIM(fail_copy); i++) { - uint32_t status[COMP_BURST_SZ * 2] = {0}; - unsigned int j; - uint16_t count; - - for (j = 0; j < COMP_BURST_SZ; j++) { - srcs[j] = rte_pktmbuf_alloc(pool); - dsts[j] = rte_pktmbuf_alloc(pool); - - /* always fail the first copy */ - if (rte_ioat_enqueue_copy(dev_id, - (j == 0 ? (phys_addr_t)NULL : - (srcs[j]->buf_iova + srcs[j]->data_off)), - dsts[j]->buf_iova + dsts[j]->data_off, - length, - (uintptr_t)srcs[j], - (uintptr_t)dsts[j]) != 1) { - PRINT_ERR("Error with rte_ioat_enqueue_copy for buffer %u\n", j); - return -1; - } - /* put in a fence which will stop any further transactions - * because we had a previous failure. - */ - if (j == fail_copy[i]) - rte_ioat_fence(dev_id); - } - rte_ioat_perform_ops(dev_id); - usleep(100); - - count = rte_ioat_completed_ops(dev_id, COMP_BURST_SZ * 2, status, ¬_ok, - (void *)completed_src, (void *)completed_dst); - if (count != COMP_BURST_SZ) { - PRINT_ERR("Error with rte_ioat_completed_ops, got %u not %u\n", - count, COMP_BURST_SZ); - for (j = 0; j < count; j++) - printf("%u ", status[j]); - printf("<-- Statuses\n"); - return -1; - } - if (not_ok != COMP_BURST_SZ - fail_copy[i]) { - PRINT_ERR("Unexpected failed copy count, got %u, expected %u\n", - not_ok, COMP_BURST_SZ - fail_copy[i]); - for (j = 0; j < COMP_BURST_SZ; j++) - printf("%u ", status[j]); - printf("<-- Statuses\n"); - return -1; - } - if (status[0] == RTE_IOAT_OP_SUCCESS || status[0] == RTE_IOAT_OP_SKIPPED) { - PRINT_ERR("Error, op 0 unexpectedly did not fail.\n"); - return -1; - } - for (j = 1; j <= fail_copy[i]; j++) { - if (status[j] != RTE_IOAT_OP_SUCCESS) { - PRINT_ERR("Error, op %u unexpectedly failed\n", j); - return -1; - } - } - for (j = fail_copy[i] + 1; j < COMP_BURST_SZ; j++) { - if (status[j] != RTE_IOAT_OP_SKIPPED) { - PRINT_ERR("Error, all descriptors after fence should be invalid\n"); - return -1; - } - } - for (j = 0; j < COMP_BURST_SZ; j++) { - rte_pktmbuf_free(completed_src[j]); - rte_pktmbuf_free(completed_dst[j]); - } - } - - return 0; -} - -int -ioat_rawdev_test(uint16_t dev_id) -{ -#define IOAT_TEST_RINGSIZE 512 - const struct rte_idxd_rawdev *idxd = - (struct rte_idxd_rawdev *)rte_rawdevs[dev_id].dev_private; - const enum rte_ioat_dev_type ioat_type = idxd->type; - struct rte_ioat_rawdev_config p = { .ring_size = -1 }; - struct rte_rawdev_info info = { .dev_private = &p }; - struct rte_rawdev_xstats_name *snames = NULL; - uint64_t *stats = NULL; - unsigned int *ids = NULL; - unsigned int nb_xstats; - unsigned int i; - - if (dev_id >= MAX_SUPPORTED_RAWDEVS) { - printf("Skipping test. Cannot test rawdevs with id's greater than %d\n", - MAX_SUPPORTED_RAWDEVS); - return TEST_SKIPPED; - } - - rte_rawdev_info_get(dev_id, &info, sizeof(p)); - if (p.ring_size != expected_ring_size[dev_id]) { - PRINT_ERR("Error, initial ring size is not as expected (Actual: %d, Expected: %d)\n", - (int)p.ring_size, expected_ring_size[dev_id]); - return -1; - } - - p.ring_size = IOAT_TEST_RINGSIZE; - if (rte_rawdev_configure(dev_id, &info, sizeof(p)) != 0) { - PRINT_ERR("Error with rte_rawdev_configure()\n"); - return -1; - } - rte_rawdev_info_get(dev_id, &info, sizeof(p)); - if (p.ring_size != IOAT_TEST_RINGSIZE) { - PRINT_ERR("Error, ring size is not %d (%d)\n", - IOAT_TEST_RINGSIZE, (int)p.ring_size); - return -1; - } - expected_ring_size[dev_id] = p.ring_size; - - if (rte_rawdev_start(dev_id) != 0) { - PRINT_ERR("Error with rte_rawdev_start()\n"); - return -1; - } - - pool = rte_pktmbuf_pool_create("TEST_IOAT_POOL", - p.ring_size * 2, /* n == num elements */ - 32, /* cache size */ - 0, /* priv size */ - 2048, /* data room size */ - info.socket_id); - if (pool == NULL) { - PRINT_ERR("Error with mempool creation\n"); - return -1; - } - - /* allocate memory for xstats names and values */ - nb_xstats = rte_rawdev_xstats_names_get(dev_id, NULL, 0); - - snames = malloc(sizeof(*snames) * nb_xstats); - if (snames == NULL) { - PRINT_ERR("Error allocating xstat names memory\n"); - goto err; - } - rte_rawdev_xstats_names_get(dev_id, snames, nb_xstats); - - ids = malloc(sizeof(*ids) * nb_xstats); - if (ids == NULL) { - PRINT_ERR("Error allocating xstat ids memory\n"); - goto err; - } - for (i = 0; i < nb_xstats; i++) - ids[i] = i; - - stats = malloc(sizeof(*stats) * nb_xstats); - if (stats == NULL) { - PRINT_ERR("Error allocating xstat memory\n"); - goto err; - } - - /* run the test cases */ - printf("Running Copy Tests\n"); - for (i = 0; i < 100; i++) { - unsigned int j; - - if (test_enqueue_copies(dev_id) != 0) - goto err; - - rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats); - for (j = 0; j < nb_xstats; j++) - printf("%s: %"PRIu64" ", snames[j].name, stats[j]); - printf("\r"); - } - printf("\n"); - - /* test enqueue fill operation */ - printf("Running Fill Tests\n"); - for (i = 0; i < 100; i++) { - unsigned int j; - - if (test_enqueue_fill(dev_id) != 0) - goto err; - - rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats); - for (j = 0; j < nb_xstats; j++) - printf("%s: %"PRIu64" ", snames[j].name, stats[j]); - printf("\r"); - } - printf("\n"); - - printf("Running Burst Capacity Test\n"); - if (test_burst_capacity(dev_id) != 0) - goto err; - - /* only DSA devices report address errors, and we can only use null pointers - * to generate those errors when DPDK is in VA mode. - */ - if (rte_eal_iova_mode() == RTE_IOVA_VA && ioat_type == RTE_IDXD_DEV) { - printf("Running Completions Status Test\n"); - if (test_completion_status(dev_id) != 0) - goto err; - } - - rte_rawdev_stop(dev_id); - if (rte_rawdev_xstats_reset(dev_id, NULL, 0) != 0) { - PRINT_ERR("Error resetting xstat values\n"); - goto err; - } - - rte_mempool_free(pool); - free(snames); - free(stats); - free(ids); - return 0; - -err: - rte_rawdev_stop(dev_id); - rte_rawdev_xstats_reset(dev_id, NULL, 0); - rte_mempool_free(pool); - free(snames); - free(stats); - free(ids); - return -1; -} diff --git a/drivers/raw/ioat/ioat_spec.h b/drivers/raw/ioat/ioat_spec.h deleted file mode 100644 index 6aa467e4cb..0000000000 --- a/drivers/raw/ioat/ioat_spec.h +++ /dev/null @@ -1,336 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) Intel Corporation - */ - -/** - * \file - * I/OAT specification definitions - * - * Taken from ioat_spec.h from SPDK project, with prefix renames and - * other minor changes. - */ - -#ifndef RTE_IOAT_SPEC_H -#define RTE_IOAT_SPEC_H - -#ifdef __cplusplus -extern "C" { -#endif - -#include - -#define RTE_IOAT_PCI_CHANERR_INT_OFFSET 0x180 - -#define RTE_IOAT_INTRCTRL_MASTER_INT_EN 0x01 - -#define RTE_IOAT_VER_3_0 0x30 -#define RTE_IOAT_VER_3_3 0x33 - -/* DMA Channel Registers */ -#define RTE_IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK 0xF000 -#define RTE_IOAT_CHANCTRL_COMPL_DCA_EN 0x0200 -#define RTE_IOAT_CHANCTRL_CHANNEL_IN_USE 0x0100 -#define RTE_IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL 0x0020 -#define RTE_IOAT_CHANCTRL_ERR_INT_EN 0x0010 -#define RTE_IOAT_CHANCTRL_ANY_ERR_ABORT_EN 0x0008 -#define RTE_IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 -#define RTE_IOAT_CHANCTRL_INT_REARM 0x0001 - -/* DMA Channel Capabilities */ -#define RTE_IOAT_DMACAP_PB (1 << 0) -#define RTE_IOAT_DMACAP_DCA (1 << 4) -#define RTE_IOAT_DMACAP_BFILL (1 << 6) -#define RTE_IOAT_DMACAP_XOR (1 << 8) -#define RTE_IOAT_DMACAP_PQ (1 << 9) -#define RTE_IOAT_DMACAP_DMA_DIF (1 << 10) - -struct rte_ioat_registers { - uint8_t chancnt; - uint8_t xfercap; - uint8_t genctrl; - uint8_t intrctrl; - uint32_t attnstatus; - uint8_t cbver; /* 0x08 */ - uint8_t reserved4[0x3]; /* 0x09 */ - uint16_t intrdelay; /* 0x0C */ - uint16_t cs_status; /* 0x0E */ - uint32_t dmacapability; /* 0x10 */ - uint8_t reserved5[0x6C]; /* 0x14 */ - uint16_t chanctrl; /* 0x80 */ - uint8_t reserved6[0x2]; /* 0x82 */ - uint8_t chancmd; /* 0x84 */ - uint8_t reserved3[1]; /* 0x85 */ - uint16_t dmacount; /* 0x86 */ - uint64_t chansts; /* 0x88 */ - uint64_t chainaddr; /* 0x90 */ - uint64_t chancmp; /* 0x98 */ - uint8_t reserved2[0x8]; /* 0xA0 */ - uint32_t chanerr; /* 0xA8 */ - uint32_t chanerrmask; /* 0xAC */ -} __rte_packed; - -#define RTE_IOAT_CHANCMD_RESET 0x20 -#define RTE_IOAT_CHANCMD_SUSPEND 0x04 - -#define RTE_IOAT_CHANSTS_STATUS 0x7ULL -#define RTE_IOAT_CHANSTS_ACTIVE 0x0 -#define RTE_IOAT_CHANSTS_IDLE 0x1 -#define RTE_IOAT_CHANSTS_SUSPENDED 0x2 -#define RTE_IOAT_CHANSTS_HALTED 0x3 -#define RTE_IOAT_CHANSTS_ARMED 0x4 - -#define RTE_IOAT_CHANSTS_UNAFFILIATED_ERROR 0x8ULL -#define RTE_IOAT_CHANSTS_SOFT_ERROR 0x10ULL - -#define RTE_IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK (~0x3FULL) - -#define RTE_IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ - -struct rte_ioat_dma_hw_desc { - uint32_t size; - union { - uint32_t control_raw; - struct { - uint32_t int_enable: 1; - uint32_t src_snoop_disable: 1; - uint32_t dest_snoop_disable: 1; - uint32_t completion_update: 1; - uint32_t fence: 1; - uint32_t null: 1; - uint32_t src_page_break: 1; - uint32_t dest_page_break: 1; - uint32_t bundle: 1; - uint32_t dest_dca: 1; - uint32_t hint: 1; - uint32_t reserved: 13; -#define RTE_IOAT_OP_COPY 0x00 - uint32_t op: 8; - } control; - } u; - uint64_t src_addr; - uint64_t dest_addr; - uint64_t next; - uint64_t reserved; - uint64_t reserved2; - uint64_t user1; - uint64_t user2; -}; - -struct rte_ioat_fill_hw_desc { - uint32_t size; - union { - uint32_t control_raw; - struct { - uint32_t int_enable: 1; - uint32_t reserved: 1; - uint32_t dest_snoop_disable: 1; - uint32_t completion_update: 1; - uint32_t fence: 1; - uint32_t reserved2: 2; - uint32_t dest_page_break: 1; - uint32_t bundle: 1; - uint32_t reserved3: 15; -#define RTE_IOAT_OP_FILL 0x01 - uint32_t op: 8; - } control; - } u; - uint64_t src_data; - uint64_t dest_addr; - uint64_t next; - uint64_t reserved; - uint64_t next_dest_addr; - uint64_t user1; - uint64_t user2; -}; - -struct rte_ioat_xor_hw_desc { - uint32_t size; - union { - uint32_t control_raw; - struct { - uint32_t int_enable: 1; - uint32_t src_snoop_disable: 1; - uint32_t dest_snoop_disable: 1; - uint32_t completion_update: 1; - uint32_t fence: 1; - uint32_t src_count: 3; - uint32_t bundle: 1; - uint32_t dest_dca: 1; - uint32_t hint: 1; - uint32_t reserved: 13; -#define RTE_IOAT_OP_XOR 0x87 -#define RTE_IOAT_OP_XOR_VAL 0x88 - uint32_t op: 8; - } control; - } u; - uint64_t src_addr; - uint64_t dest_addr; - uint64_t next; - uint64_t src_addr2; - uint64_t src_addr3; - uint64_t src_addr4; - uint64_t src_addr5; -}; - -struct rte_ioat_xor_ext_hw_desc { - uint64_t src_addr6; - uint64_t src_addr7; - uint64_t src_addr8; - uint64_t next; - uint64_t reserved[4]; -}; - -struct rte_ioat_pq_hw_desc { - uint32_t size; - union { - uint32_t control_raw; - struct { - uint32_t int_enable: 1; - uint32_t src_snoop_disable: 1; - uint32_t dest_snoop_disable: 1; - uint32_t completion_update: 1; - uint32_t fence: 1; - uint32_t src_count: 3; - uint32_t bundle: 1; - uint32_t dest_dca: 1; - uint32_t hint: 1; - uint32_t p_disable: 1; - uint32_t q_disable: 1; - uint32_t reserved: 11; -#define RTE_IOAT_OP_PQ 0x89 -#define RTE_IOAT_OP_PQ_VAL 0x8a - uint32_t op: 8; - } control; - } u; - uint64_t src_addr; - uint64_t p_addr; - uint64_t next; - uint64_t src_addr2; - uint64_t src_addr3; - uint8_t coef[8]; - uint64_t q_addr; -}; - -struct rte_ioat_pq_ext_hw_desc { - uint64_t src_addr4; - uint64_t src_addr5; - uint64_t src_addr6; - uint64_t next; - uint64_t src_addr7; - uint64_t src_addr8; - uint64_t reserved[2]; -}; - -struct rte_ioat_pq_update_hw_desc { - uint32_t size; - union { - uint32_t control_raw; - struct { - uint32_t int_enable: 1; - uint32_t src_snoop_disable: 1; - uint32_t dest_snoop_disable: 1; - uint32_t completion_update: 1; - uint32_t fence: 1; - uint32_t src_cnt: 3; - uint32_t bundle: 1; - uint32_t dest_dca: 1; - uint32_t hint: 1; - uint32_t p_disable: 1; - uint32_t q_disable: 1; - uint32_t reserved: 3; - uint32_t coef: 8; -#define RTE_IOAT_OP_PQ_UP 0x8b - uint32_t op: 8; - } control; - } u; - uint64_t src_addr; - uint64_t p_addr; - uint64_t next; - uint64_t src_addr2; - uint64_t p_src; - uint64_t q_src; - uint64_t q_addr; -}; - -struct rte_ioat_raw_hw_desc { - uint64_t field[8]; -}; - -union rte_ioat_hw_desc { - struct rte_ioat_raw_hw_desc raw; - struct rte_ioat_generic_hw_desc generic; - struct rte_ioat_dma_hw_desc dma; - struct rte_ioat_fill_hw_desc fill; - struct rte_ioat_xor_hw_desc xor_desc; - struct rte_ioat_xor_ext_hw_desc xor_ext; - struct rte_ioat_pq_hw_desc pq; - struct rte_ioat_pq_ext_hw_desc pq_ext; - struct rte_ioat_pq_update_hw_desc pq_update; -}; - -/*** Definitions for Intel(R) Data Streaming Accelerator Follow ***/ - -#define IDXD_CMD_SHIFT 20 -enum rte_idxd_cmds { - idxd_enable_dev = 1, - idxd_disable_dev, - idxd_drain_all, - idxd_abort_all, - idxd_reset_device, - idxd_enable_wq, - idxd_disable_wq, - idxd_drain_wq, - idxd_abort_wq, - idxd_reset_wq, -}; - -/* General bar0 registers */ -struct rte_idxd_bar0 { - uint32_t __rte_cache_aligned version; /* offset 0x00 */ - uint64_t __rte_aligned(0x10) gencap; /* offset 0x10 */ - uint64_t __rte_aligned(0x10) wqcap; /* offset 0x20 */ - uint64_t __rte_aligned(0x10) grpcap; /* offset 0x30 */ - uint64_t __rte_aligned(0x08) engcap; /* offset 0x38 */ - uint64_t __rte_aligned(0x10) opcap; /* offset 0x40 */ - uint64_t __rte_aligned(0x20) offsets[2]; /* offset 0x60 */ - uint32_t __rte_aligned(0x20) gencfg; /* offset 0x80 */ - uint32_t __rte_aligned(0x08) genctrl; /* offset 0x88 */ - uint32_t __rte_aligned(0x10) gensts; /* offset 0x90 */ - uint32_t __rte_aligned(0x08) intcause; /* offset 0x98 */ - uint32_t __rte_aligned(0x10) cmd; /* offset 0xA0 */ - uint32_t __rte_aligned(0x08) cmdstatus; /* offset 0xA8 */ - uint64_t __rte_aligned(0x20) swerror[4]; /* offset 0xC0 */ -}; - -/* workqueue config is provided by array of uint32_t. */ -#define WQ_SIZE_IDX 0 /* size is in first 32-bit value */ -#define WQ_THRESHOLD_IDX 1 /* WQ threshold second 32-bits */ -#define WQ_MODE_IDX 2 /* WQ mode and other flags */ -#define WQ_SIZES_IDX 3 /* WQ transfer and batch sizes */ -#define WQ_OCC_INT_IDX 4 /* WQ occupancy interrupt handle */ -#define WQ_OCC_LIMIT_IDX 5 /* WQ occupancy limit */ -#define WQ_STATE_IDX 6 /* WQ state and occupancy state */ - -#define WQ_MODE_SHARED 0 -#define WQ_MODE_DEDICATED 1 -#define WQ_PRIORITY_SHIFT 4 -#define WQ_BATCH_SZ_SHIFT 5 -#define WQ_STATE_SHIFT 30 -#define WQ_STATE_MASK 0x3 - -struct rte_idxd_grpcfg { - uint64_t grpwqcfg[4] __rte_cache_aligned; /* 64-byte register set */ - uint64_t grpengcfg; /* offset 32 */ - uint32_t grpflags; /* offset 40 */ -}; - -#define GENSTS_DEV_STATE_MASK 0x03 -#define CMDSTATUS_ACTIVE_SHIFT 31 -#define CMDSTATUS_ACTIVE_MASK (1 << 31) -#define CMDSTATUS_ERR_MASK 0xFF - -#ifdef __cplusplus -} -#endif - -#endif /* RTE_IOAT_SPEC_H */ diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build deleted file mode 100644 index 1b866aab74..0000000000 --- a/drivers/raw/ioat/meson.build +++ /dev/null @@ -1,36 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright 2019 Intel Corporation - -build = dpdk_conf.has('RTE_ARCH_X86') -# only use ioat rawdev driver if we don't have the equivalent dmadev ones -if dpdk_conf.has('RTE_DMA_IDXD') and dpdk_conf.has('RTE_DMA_IOAT') - build = false - reason = 'replaced by dmadev drivers' - subdir_done() -endif - -reason = 'only supported on x86' -sources = files( - 'ioat_common.c', - 'ioat_rawdev_test.c', -) - -if not dpdk_conf.has('RTE_DMA_IDXD') - sources += files( - 'idxd_bus.c', - 'idxd_pci.c', - ) -endif - -if not dpdk_conf.has('RTE_DMA_IOAT') - sources += files ( - 'ioat_rawdev.c', - ) -endif - -deps += ['bus_pci', 'mbuf', 'rawdev'] -headers = files( - 'rte_ioat_rawdev.h', - 'rte_idxd_rawdev_fns.h', - 'rte_ioat_rawdev_fns.h', -) diff --git a/drivers/raw/ioat/rte_idxd_rawdev_fns.h b/drivers/raw/ioat/rte_idxd_rawdev_fns.h deleted file mode 100644 index a07892faa0..0000000000 --- a/drivers/raw/ioat/rte_idxd_rawdev_fns.h +++ /dev/null @@ -1,394 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2021 Intel Corporation - */ -#ifndef _RTE_IDXD_RAWDEV_FNS_H_ -#define _RTE_IDXD_RAWDEV_FNS_H_ - -/** - * @file - * This header file contains the implementation of the various ioat - * rawdev functions for DSA hardware. The API specification and key - * public structures are defined in "rte_ioat_rawdev.h". - * - * This file should not be included directly, but instead applications should - * include "rte_ioat_rawdev.h", which then includes this file - and the - * IOAT/CBDMA equivalent header - in turn. - */ - -#include -#include - -/* - * Defines used in the data path for interacting with IDXD hardware. - */ -#define IDXD_CMD_OP_SHIFT 24 -enum rte_idxd_ops { - idxd_op_nop = 0, - idxd_op_batch, - idxd_op_drain, - idxd_op_memmove, - idxd_op_fill -}; - -#define IDXD_FLAG_FENCE (1 << 0) -#define IDXD_FLAG_COMPLETION_ADDR_VALID (1 << 2) -#define IDXD_FLAG_REQUEST_COMPLETION (1 << 3) -#define IDXD_FLAG_CACHE_CONTROL (1 << 8) - -#define IOAT_COMP_UPDATE_SHIFT 3 -#define IOAT_CMD_OP_SHIFT 24 -enum rte_ioat_ops { - ioat_op_copy = 0, /* Standard DMA Operation */ - ioat_op_fill /* Block Fill */ -}; - -/** - * Hardware descriptor used by DSA hardware, for both bursts and - * for individual operations. - */ -struct rte_idxd_hw_desc { - uint32_t pasid; - uint32_t op_flags; - rte_iova_t completion; - - RTE_STD_C11 - union { - rte_iova_t src; /* source address for copy ops etc. */ - rte_iova_t desc_addr; /* descriptor pointer for batch */ - }; - rte_iova_t dst; - - uint32_t size; /* length of data for op, or batch size */ - - uint16_t intr_handle; /* completion interrupt handle */ - - /* remaining 26 bytes are reserved */ - uint16_t __reserved[13]; -} __rte_aligned(64); - -/** - * Completion record structure written back by DSA - */ -struct rte_idxd_completion { - uint8_t status; - uint8_t result; - /* 16-bits pad here */ - uint32_t completed_size; /* data length, or descriptors for batch */ - - rte_iova_t fault_address; - uint32_t invalid_flags; -} __rte_aligned(32); - -/** - * structure used to save the "handles" provided by the user to be - * returned to the user on job completion. - */ -struct rte_idxd_user_hdl { - uint64_t src; - uint64_t dst; -}; - -/** - * @internal - * Structure representing an IDXD device instance - */ -struct rte_idxd_rawdev { - enum rte_ioat_dev_type type; - struct rte_ioat_xstats xstats; - - void *portal; /* address to write the batch descriptor */ - - struct rte_ioat_rawdev_config cfg; - rte_iova_t desc_iova; /* base address of desc ring, needed for completions */ - - /* counters to track the batches */ - unsigned short max_batches; - unsigned short batch_idx_read; - unsigned short batch_idx_write; - unsigned short *batch_idx_ring; /* store where each batch ends */ - - /* track descriptors and handles */ - unsigned short desc_ring_mask; - unsigned short hdls_avail; /* handles for ops completed */ - unsigned short hdls_read; /* the read pointer for hdls/desc rings */ - unsigned short batch_start; /* start+size == write pointer for hdls/desc */ - unsigned short batch_size; - - struct rte_idxd_hw_desc *desc_ring; - struct rte_idxd_user_hdl *hdl_ring; - /* flags to indicate handle validity. Kept separate from ring, to avoid - * using 8 bytes per flag. Upper 8 bits holds error code if any. - */ - uint16_t *hdl_ring_flags; -}; - -#define RTE_IDXD_HDL_NORMAL 0 -#define RTE_IDXD_HDL_INVALID (1 << 0) /* no handle stored for this element */ -#define RTE_IDXD_HDL_OP_FAILED (1 << 1) /* return failure for this one */ -#define RTE_IDXD_HDL_OP_SKIPPED (1 << 2) /* this op was skipped */ - -static __rte_always_inline uint16_t -__idxd_burst_capacity(int dev_id) -{ - struct rte_idxd_rawdev *idxd = - (struct rte_idxd_rawdev *)rte_rawdevs[dev_id].dev_private; - uint16_t write_idx = idxd->batch_start + idxd->batch_size; - uint16_t used_space, free_space; - - /* Check for space in the batch ring */ - if ((idxd->batch_idx_read == 0 && idxd->batch_idx_write == idxd->max_batches) || - idxd->batch_idx_write + 1 == idxd->batch_idx_read) - return 0; - - /* for descriptors, check for wrap-around on write but not read */ - if (idxd->hdls_read > write_idx) - write_idx += idxd->desc_ring_mask + 1; - used_space = write_idx - idxd->hdls_read; - - /* Return amount of free space in the descriptor ring - * subtract 1 for space for batch descriptor and 1 for possible null desc - */ - free_space = idxd->desc_ring_mask - used_space; - if (free_space < 2) - return 0; - return free_space - 2; -} - -static __rte_always_inline rte_iova_t -__desc_idx_to_iova(struct rte_idxd_rawdev *idxd, uint16_t n) -{ - return idxd->desc_iova + (n * sizeof(struct rte_idxd_hw_desc)); -} - -static __rte_always_inline int -__idxd_write_desc(int dev_id, - const uint32_t op_flags, - const rte_iova_t src, - const rte_iova_t dst, - const uint32_t size, - const struct rte_idxd_user_hdl *hdl) -{ - struct rte_idxd_rawdev *idxd = - (struct rte_idxd_rawdev *)rte_rawdevs[dev_id].dev_private; - uint16_t write_idx = idxd->batch_start + idxd->batch_size; - uint16_t mask = idxd->desc_ring_mask; - - /* first check batch ring space then desc ring space */ - if ((idxd->batch_idx_read == 0 && idxd->batch_idx_write == idxd->max_batches) || - idxd->batch_idx_write + 1 == idxd->batch_idx_read) - goto failed; - /* for descriptor ring, we always need a slot for batch completion */ - if (((write_idx + 2) & mask) == idxd->hdls_read || - ((write_idx + 1) & mask) == idxd->hdls_read) - goto failed; - - /* write desc and handle. Note, descriptors don't wrap */ - idxd->desc_ring[write_idx].pasid = 0; - idxd->desc_ring[write_idx].op_flags = op_flags | IDXD_FLAG_COMPLETION_ADDR_VALID; - idxd->desc_ring[write_idx].completion = __desc_idx_to_iova(idxd, write_idx & mask); - idxd->desc_ring[write_idx].src = src; - idxd->desc_ring[write_idx].dst = dst; - idxd->desc_ring[write_idx].size = size; - - if (hdl == NULL) - idxd->hdl_ring_flags[write_idx & mask] = RTE_IDXD_HDL_INVALID; - else - idxd->hdl_ring[write_idx & mask] = *hdl; - idxd->batch_size++; - - idxd->xstats.enqueued++; - - rte_prefetch0_write(&idxd->desc_ring[write_idx + 1]); - return 1; - -failed: - idxd->xstats.enqueue_failed++; - rte_errno = ENOSPC; - return 0; -} - -static __rte_always_inline int -__idxd_enqueue_fill(int dev_id, uint64_t pattern, rte_iova_t dst, - unsigned int length, uintptr_t dst_hdl) -{ - const struct rte_idxd_user_hdl hdl = { - .dst = dst_hdl - }; - return __idxd_write_desc(dev_id, - (idxd_op_fill << IDXD_CMD_OP_SHIFT) | IDXD_FLAG_CACHE_CONTROL, - pattern, dst, length, &hdl); -} - -static __rte_always_inline int -__idxd_enqueue_copy(int dev_id, rte_iova_t src, rte_iova_t dst, - unsigned int length, uintptr_t src_hdl, uintptr_t dst_hdl) -{ - const struct rte_idxd_user_hdl hdl = { - .src = src_hdl, - .dst = dst_hdl - }; - return __idxd_write_desc(dev_id, - (idxd_op_memmove << IDXD_CMD_OP_SHIFT) | IDXD_FLAG_CACHE_CONTROL, - src, dst, length, &hdl); -} - -static __rte_always_inline int -__idxd_enqueue_nop(int dev_id) -{ - /* only op field needs filling - zero src, dst and length */ - return __idxd_write_desc(dev_id, idxd_op_nop << IDXD_CMD_OP_SHIFT, - 0, 0, 0, NULL); -} - -static __rte_always_inline int -__idxd_fence(int dev_id) -{ - /* only op field needs filling - zero src, dst and length */ - return __idxd_write_desc(dev_id, IDXD_FLAG_FENCE, 0, 0, 0, NULL); -} - -static __rte_always_inline void -__idxd_movdir64b(volatile void *dst, const struct rte_idxd_hw_desc *src) -{ - asm volatile (".byte 0x66, 0x0f, 0x38, 0xf8, 0x02" - : - : "a" (dst), "d" (src) - : "memory"); -} - -static __rte_always_inline int -__idxd_perform_ops(int dev_id) -{ - struct rte_idxd_rawdev *idxd = - (struct rte_idxd_rawdev *)rte_rawdevs[dev_id].dev_private; - - if (!idxd->cfg.no_prefetch_completions) - rte_prefetch1(&idxd->desc_ring[idxd->batch_idx_ring[idxd->batch_idx_read]]); - - if (idxd->batch_size == 0) - return 0; - - if (idxd->batch_size == 1) - /* use a NOP as a null descriptor, so batch_size >= 2 */ - if (__idxd_enqueue_nop(dev_id) != 1) - return -1; - - /* write completion beyond last desc in the batch */ - uint16_t comp_idx = (idxd->batch_start + idxd->batch_size) & idxd->desc_ring_mask; - *((uint64_t *)&idxd->desc_ring[comp_idx]) = 0; /* zero start of desc */ - idxd->hdl_ring_flags[comp_idx] = RTE_IDXD_HDL_INVALID; - - const struct rte_idxd_hw_desc batch_desc = { - .op_flags = (idxd_op_batch << IDXD_CMD_OP_SHIFT) | - IDXD_FLAG_COMPLETION_ADDR_VALID | - IDXD_FLAG_REQUEST_COMPLETION, - .desc_addr = __desc_idx_to_iova(idxd, idxd->batch_start), - .completion = __desc_idx_to_iova(idxd, comp_idx), - .size = idxd->batch_size, - }; - - _mm_sfence(); /* fence before writing desc to device */ - __idxd_movdir64b(idxd->portal, &batch_desc); - idxd->xstats.started += idxd->batch_size; - - idxd->batch_start += idxd->batch_size + 1; - idxd->batch_start &= idxd->desc_ring_mask; - idxd->batch_size = 0; - - idxd->batch_idx_ring[idxd->batch_idx_write++] = comp_idx; - if (idxd->batch_idx_write > idxd->max_batches) - idxd->batch_idx_write = 0; - - return 0; -} - -static __rte_always_inline int -__idxd_completed_ops(int dev_id, uint8_t max_ops, uint32_t *status, uint8_t *num_unsuccessful, - uintptr_t *src_hdls, uintptr_t *dst_hdls) -{ - struct rte_idxd_rawdev *idxd = - (struct rte_idxd_rawdev *)rte_rawdevs[dev_id].dev_private; - unsigned short n, h_idx; - - while (idxd->batch_idx_read != idxd->batch_idx_write) { - uint16_t idx_to_chk = idxd->batch_idx_ring[idxd->batch_idx_read]; - volatile struct rte_idxd_completion *comp_to_chk = - (struct rte_idxd_completion *)&idxd->desc_ring[idx_to_chk]; - uint8_t batch_status = comp_to_chk->status; - if (batch_status == 0) - break; - comp_to_chk->status = 0; - if (unlikely(batch_status > 1)) { - /* error occurred somewhere in batch, start where last checked */ - uint16_t desc_count = comp_to_chk->completed_size; - uint16_t batch_start = idxd->hdls_avail; - uint16_t batch_end = idx_to_chk; - - if (batch_start > batch_end) - batch_end += idxd->desc_ring_mask + 1; - /* go through each batch entry and see status */ - for (n = 0; n < desc_count; n++) { - uint16_t idx = (batch_start + n) & idxd->desc_ring_mask; - volatile struct rte_idxd_completion *comp = - (struct rte_idxd_completion *)&idxd->desc_ring[idx]; - if (comp->status != 0 && - idxd->hdl_ring_flags[idx] == RTE_IDXD_HDL_NORMAL) { - idxd->hdl_ring_flags[idx] = RTE_IDXD_HDL_OP_FAILED; - idxd->hdl_ring_flags[idx] |= (comp->status << 8); - comp->status = 0; /* clear error for next time */ - } - } - /* if batch is incomplete, mark rest as skipped */ - for ( ; n < batch_end - batch_start; n++) { - uint16_t idx = (batch_start + n) & idxd->desc_ring_mask; - if (idxd->hdl_ring_flags[idx] == RTE_IDXD_HDL_NORMAL) - idxd->hdl_ring_flags[idx] = RTE_IDXD_HDL_OP_SKIPPED; - } - } - /* avail points to one after the last one written */ - idxd->hdls_avail = (idx_to_chk + 1) & idxd->desc_ring_mask; - idxd->batch_idx_read++; - if (idxd->batch_idx_read > idxd->max_batches) - idxd->batch_idx_read = 0; - } - - n = 0; - h_idx = idxd->hdls_read; - while (h_idx != idxd->hdls_avail) { - uint16_t flag = idxd->hdl_ring_flags[h_idx]; - if (flag != RTE_IDXD_HDL_INVALID) { - if (!idxd->cfg.hdls_disable) { - src_hdls[n] = idxd->hdl_ring[h_idx].src; - dst_hdls[n] = idxd->hdl_ring[h_idx].dst; - } - if (unlikely(flag != RTE_IDXD_HDL_NORMAL)) { - if (status != NULL) - status[n] = flag == RTE_IDXD_HDL_OP_SKIPPED ? - RTE_IOAT_OP_SKIPPED : - /* failure case, return err code */ - idxd->hdl_ring_flags[h_idx] >> 8; - if (num_unsuccessful != NULL) - *num_unsuccessful += 1; - } - n++; - } - idxd->hdl_ring_flags[h_idx] = RTE_IDXD_HDL_NORMAL; - if (++h_idx > idxd->desc_ring_mask) - h_idx = 0; - if (n >= max_ops) - break; - } - - /* skip over any remaining blank elements, e.g. batch completion */ - while (idxd->hdl_ring_flags[h_idx] == RTE_IDXD_HDL_INVALID && h_idx != idxd->hdls_avail) { - idxd->hdl_ring_flags[h_idx] = RTE_IDXD_HDL_NORMAL; - if (++h_idx > idxd->desc_ring_mask) - h_idx = 0; - } - idxd->hdls_read = h_idx; - - idxd->xstats.completed += n; - return n; -} - -#endif diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h deleted file mode 100644 index 0e79f75472..0000000000 --- a/drivers/raw/ioat/rte_ioat_rawdev.h +++ /dev/null @@ -1,214 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2019 Intel Corporation - */ - -#ifndef _RTE_IOAT_RAWDEV_H_ -#define _RTE_IOAT_RAWDEV_H_ - -#ifdef __cplusplus -extern "C" { -#endif - -/** - * @file rte_ioat_rawdev.h - * - * Definitions for using the ioat rawdev device driver - * - * @warning - * @b EXPERIMENTAL: these structures and APIs may change without prior notice - */ - -#include - -/** Name of the device driver */ -#define IOAT_PMD_RAWDEV_NAME rawdev_ioat -/** String reported as the device driver name by rte_rawdev_info_get() */ -#define IOAT_PMD_RAWDEV_NAME_STR "rawdev_ioat" - -/** - * Configuration structure for an ioat rawdev instance - * - * This structure is to be passed as the ".dev_private" parameter when - * calling the rte_rawdev_get_info() and rte_rawdev_configure() APIs on - * an ioat rawdev instance. - */ -struct rte_ioat_rawdev_config { - unsigned short ring_size; /**< size of job submission descriptor ring */ - bool hdls_disable; /**< if set, ignore user-supplied handle params */ - /** set "no_prefetch_completions", if polling completions on separate core - * from the core submitting the jobs - */ - bool no_prefetch_completions; -}; - -/** - * Enqueue a fill operation onto the ioat device - * - * This queues up a fill operation to be performed by hardware, but does not - * trigger hardware to begin that operation. - * - * @param dev_id - * The rawdev device id of the ioat instance - * @param pattern - * The pattern to populate the destination buffer with - * @param dst - * The physical address of the destination buffer - * @param length - * The length of the destination buffer - * @param dst_hdl - * An opaque handle for the destination data, to be returned when this - * operation has been completed and the user polls for the completion details. - * NOTE: If hdls_disable configuration option for the device is set, this - * parameter is ignored. - * @return - * Number of operations enqueued, either 0 or 1 - */ -static inline int -__rte_experimental -rte_ioat_enqueue_fill(int dev_id, uint64_t pattern, phys_addr_t dst, - unsigned int length, uintptr_t dst_hdl); - -/** - * Enqueue a copy operation onto the ioat device - * - * This queues up a copy operation to be performed by hardware, but does not - * trigger hardware to begin that operation. - * - * @param dev_id - * The rawdev device id of the ioat instance - * @param src - * The physical address of the source buffer - * @param dst - * The physical address of the destination buffer - * @param length - * The length of the data to be copied - * @param src_hdl - * An opaque handle for the source data, to be returned when this operation - * has been completed and the user polls for the completion details. - * NOTE: If hdls_disable configuration option for the device is set, this - * parameter is ignored. - * @param dst_hdl - * An opaque handle for the destination data, to be returned when this - * operation has been completed and the user polls for the completion details. - * NOTE: If hdls_disable configuration option for the device is set, this - * parameter is ignored. - * @return - * Number of operations enqueued, either 0 or 1 - */ -static inline int -__rte_experimental -rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst, - unsigned int length, uintptr_t src_hdl, uintptr_t dst_hdl); - -/** - * Add a fence to force ordering between operations - * - * This adds a fence to a sequence of operations to enforce ordering, such that - * all operations enqueued before the fence must be completed before operations - * after the fence. - * NOTE: Since this fence may be added as a flag to the last operation enqueued, - * this API may not function correctly when called immediately after an - * "rte_ioat_perform_ops" call i.e. before any new operations are enqueued. - * - * @param dev_id - * The rawdev device id of the ioat instance - * @return - * Number of fences enqueued, either 0 or 1 - */ -static inline int -__rte_experimental -rte_ioat_fence(int dev_id); - - -/** - * Trigger hardware to begin performing enqueued operations - * - * Writes the "doorbell" to the hardware to trigger it - * to begin the operations previously enqueued by rte_ioat_enqueue_copy() - * - * @param dev_id - * The rawdev device id of the ioat instance - * @return - * 0 on success. Non-zero return on error. - */ -static inline int -__rte_experimental -rte_ioat_perform_ops(int dev_id); - -/* - * Status codes for operations. - */ -#define RTE_IOAT_OP_SUCCESS 0 /**< Operation completed successfully */ -#define RTE_IOAT_OP_SKIPPED 1 /**< Operation was not attempted (Earlier fenced op failed) */ -/* Values >1 indicate a failure condition */ -/* Error codes taken from Intel(R) Data Streaming Accelerator Architecture - * Specification, section 5.7 - */ -#define RTE_IOAT_OP_ADDRESS_ERR 0x03 /**< Page fault or invalid address */ -#define RTE_IOAT_OP_INVALID_LEN 0x13 /**< Invalid/too big length field passed */ -#define RTE_IOAT_OP_OVERLAPPING_BUFS 0x16 /**< Overlapping buffers error */ - - -/** - * Returns details of operations that have been completed - * - * The status of each operation is returned in the status array parameter. - * If the hdls_disable option was not set when the device was configured, - * the function will return to the caller the user-provided "handles" for - * the copy operations which have been completed by the hardware, and not - * already returned by a previous call to this API. - * If the hdls_disable option for the device was set on configure, the - * src_hdls and dst_hdls parameters will be ignored, and the - * function returns the number of newly-completed operations. - * If status is also NULL, then max_copies parameter is also ignored and the - * function returns a count of the number of newly-completed operations. - * - * @param dev_id - * The rawdev device id of the ioat instance - * @param max_copies - * The number of entries which can fit in the status, src_hdls and dst_hdls - * arrays, i.e. max number of completed operations to report. - * NOTE: If hdls_disable configuration option for the device is set, this - * parameter applies only to the "status" array if specified - * @param status - * Array to hold the status of each completed operation. Array should be - * set to zeros on input, as the driver will only write error status values. - * A value of 1 implies an operation was not attempted, and any other non-zero - * value indicates operation failure. - * Parameter may be NULL if no status value checking is required. - * @param num_unsuccessful - * Returns the number of elements in status where the value is non-zero, - * i.e. the operation either failed or was not attempted due to an earlier - * failure. If this value is returned as zero (the expected case), the - * status array will not have been modified by the function and need not be - * checked by software - * @param src_hdls - * Array to hold the source handle parameters of the completed ops. - * NOTE: If hdls_disable configuration option for the device is set, this - * parameter is ignored, and may be NULL - * @param dst_hdls - * Array to hold the destination handle parameters of the completed ops. - * NOTE: If hdls_disable configuration option for the device is set, this - * parameter is ignored, and may be NULL - * @return - * -1 on device error, with rte_errno set appropriately and parameters - * unmodified. - * Otherwise number of returned operations i.e. number of valid entries - * in the status, src_hdls and dst_hdls array parameters. If status is NULL, - * and the hdls_disable config option is set, this value may be greater than - * max_copies parameter. - */ -static inline int -__rte_experimental -rte_ioat_completed_ops(int dev_id, uint8_t max_copies, - uint32_t *status, uint8_t *num_unsuccessful, - uintptr_t *src_hdls, uintptr_t *dst_hdls); - -/* include the implementation details from a separate file */ -#include "rte_ioat_rawdev_fns.h" - -#ifdef __cplusplus -} -#endif - -#endif /* _RTE_IOAT_RAWDEV_H_ */ diff --git a/drivers/raw/ioat/rte_ioat_rawdev_fns.h b/drivers/raw/ioat/rte_ioat_rawdev_fns.h deleted file mode 100644 index 6049e3bd8b..0000000000 --- a/drivers/raw/ioat/rte_ioat_rawdev_fns.h +++ /dev/null @@ -1,379 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2019-2020 Intel Corporation - */ -#ifndef _RTE_IOAT_RAWDEV_FNS_H_ -#define _RTE_IOAT_RAWDEV_FNS_H_ - -/** - * @file - * This header file contains the implementation of the various ioat - * rawdev functions for IOAT/CBDMA hardware. The API specification and key - * public structures are defined in "rte_ioat_rawdev.h". - * - * This file should not be included directly, but instead applications should - * include "rte_ioat_rawdev.h", which then includes this file - and the IDXD/DSA - * equivalent header - in turn. - */ - -#include -#include -#include -#include - -/** - * @internal - * Identify the data path to use. - * Must be first field of rte_ioat_rawdev and rte_idxd_rawdev structs - */ -enum rte_ioat_dev_type { - RTE_IOAT_DEV, - RTE_IDXD_DEV, -}; - -/** - * @internal - * some statistics for tracking, if added/changed update xstats fns - */ -struct rte_ioat_xstats { - uint64_t enqueue_failed; - uint64_t enqueued; - uint64_t started; - uint64_t completed; -}; - -#include "rte_idxd_rawdev_fns.h" - -/** - * @internal - * Structure representing a device descriptor - */ -struct rte_ioat_generic_hw_desc { - uint32_t size; - union { - uint32_t control_raw; - struct { - uint32_t int_enable: 1; - uint32_t src_snoop_disable: 1; - uint32_t dest_snoop_disable: 1; - uint32_t completion_update: 1; - uint32_t fence: 1; - uint32_t reserved2: 1; - uint32_t src_page_break: 1; - uint32_t dest_page_break: 1; - uint32_t bundle: 1; - uint32_t dest_dca: 1; - uint32_t hint: 1; - uint32_t reserved: 13; - uint32_t op: 8; - } control; - } u; - uint64_t src_addr; - uint64_t dest_addr; - uint64_t next; - uint64_t op_specific[4]; -}; - -/** - * @internal - * Structure representing an IOAT device instance - */ -struct rte_ioat_rawdev { - /* common fields at the top - match those in rte_idxd_rawdev */ - enum rte_ioat_dev_type type; - struct rte_ioat_xstats xstats; - - struct rte_rawdev *rawdev; - const struct rte_memzone *mz; - const struct rte_memzone *desc_mz; - - volatile uint16_t *doorbell __rte_cache_aligned; - phys_addr_t status_addr; - phys_addr_t ring_addr; - - unsigned short ring_size; - bool hdls_disable; - struct rte_ioat_generic_hw_desc *desc_ring; - __m128i *hdls; /* completion handles for returning to user */ - - - unsigned short next_read; - unsigned short next_write; - - /* to report completions, the device will write status back here */ - volatile uint64_t status __rte_cache_aligned; - - /* pointer to the register bar */ - volatile struct rte_ioat_registers *regs; -}; - -#define RTE_IOAT_CHANSTS_IDLE 0x1 -#define RTE_IOAT_CHANSTS_SUSPENDED 0x2 -#define RTE_IOAT_CHANSTS_HALTED 0x3 -#define RTE_IOAT_CHANSTS_ARMED 0x4 - -static __rte_always_inline uint16_t -__ioat_burst_capacity(int dev_id) -{ - struct rte_ioat_rawdev *ioat = - (struct rte_ioat_rawdev *)rte_rawdevs[dev_id].dev_private; - unsigned short size = ioat->ring_size - 1; - unsigned short read = ioat->next_read; - unsigned short write = ioat->next_write; - unsigned short space = size - (write - read); - - return space; -} - -static __rte_always_inline int -__ioat_write_desc(int dev_id, uint32_t op, uint64_t src, phys_addr_t dst, - unsigned int length, uintptr_t src_hdl, uintptr_t dst_hdl) -{ - struct rte_ioat_rawdev *ioat = - (struct rte_ioat_rawdev *)rte_rawdevs[dev_id].dev_private; - unsigned short read = ioat->next_read; - unsigned short write = ioat->next_write; - unsigned short mask = ioat->ring_size - 1; - unsigned short space = mask + read - write; - struct rte_ioat_generic_hw_desc *desc; - - if (space == 0) { - ioat->xstats.enqueue_failed++; - return 0; - } - - ioat->next_write = write + 1; - write &= mask; - - desc = &ioat->desc_ring[write]; - desc->size = length; - /* set descriptor write-back every 16th descriptor */ - desc->u.control_raw = (uint32_t)((op << IOAT_CMD_OP_SHIFT) | - (!(write & 0xF) << IOAT_COMP_UPDATE_SHIFT)); - desc->src_addr = src; - desc->dest_addr = dst; - - if (!ioat->hdls_disable) - ioat->hdls[write] = _mm_set_epi64x((int64_t)dst_hdl, - (int64_t)src_hdl); - rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]); - - ioat->xstats.enqueued++; - return 1; -} - -static __rte_always_inline int -__ioat_enqueue_fill(int dev_id, uint64_t pattern, phys_addr_t dst, - unsigned int length, uintptr_t dst_hdl) -{ - static const uintptr_t null_hdl; - - return __ioat_write_desc(dev_id, ioat_op_fill, pattern, dst, length, - null_hdl, dst_hdl); -} - -/* - * Enqueue a copy operation onto the ioat device - */ -static __rte_always_inline int -__ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst, - unsigned int length, uintptr_t src_hdl, uintptr_t dst_hdl) -{ - return __ioat_write_desc(dev_id, ioat_op_copy, src, dst, length, - src_hdl, dst_hdl); -} - -/* add fence to last written descriptor */ -static __rte_always_inline int -__ioat_fence(int dev_id) -{ - struct rte_ioat_rawdev *ioat = - (struct rte_ioat_rawdev *)rte_rawdevs[dev_id].dev_private; - unsigned short write = ioat->next_write; - unsigned short mask = ioat->ring_size - 1; - struct rte_ioat_generic_hw_desc *desc; - - write = (write - 1) & mask; - desc = &ioat->desc_ring[write]; - - desc->u.control.fence = 1; - return 0; -} - -/* - * Trigger hardware to begin performing enqueued operations - */ -static __rte_always_inline int -__ioat_perform_ops(int dev_id) -{ - struct rte_ioat_rawdev *ioat = - (struct rte_ioat_rawdev *)rte_rawdevs[dev_id].dev_private; - ioat->desc_ring[(ioat->next_write - 1) & (ioat->ring_size - 1)].u - .control.completion_update = 1; - rte_compiler_barrier(); - *ioat->doorbell = ioat->next_write; - ioat->xstats.started = ioat->xstats.enqueued; - - return 0; -} - -/** - * @internal - * Returns the index of the last completed operation. - */ -static __rte_always_inline int -__ioat_get_last_completed(struct rte_ioat_rawdev *ioat, int *error) -{ - uint64_t status = ioat->status; - - /* lower 3 bits indicate "transfer status" : active, idle, halted. - * We can ignore bit 0. - */ - *error = status & (RTE_IOAT_CHANSTS_SUSPENDED | RTE_IOAT_CHANSTS_ARMED); - return (status - ioat->ring_addr) >> 6; -} - -/* - * Returns details of operations that have been completed - */ -static __rte_always_inline int -__ioat_completed_ops(int dev_id, uint8_t max_copies, - uintptr_t *src_hdls, uintptr_t *dst_hdls) -{ - struct rte_ioat_rawdev *ioat = - (struct rte_ioat_rawdev *)rte_rawdevs[dev_id].dev_private; - unsigned short mask = (ioat->ring_size - 1); - unsigned short read = ioat->next_read; - unsigned short end_read, count; - int error; - int i = 0; - - end_read = (__ioat_get_last_completed(ioat, &error) + 1) & mask; - count = (end_read - (read & mask)) & mask; - - if (error) { - rte_errno = EIO; - return -1; - } - - if (ioat->hdls_disable) { - read += count; - goto end; - } - - if (count > max_copies) - count = max_copies; - - for (; i < count - 1; i += 2, read += 2) { - __m128i hdls0 = _mm_load_si128(&ioat->hdls[read & mask]); - __m128i hdls1 = _mm_load_si128(&ioat->hdls[(read + 1) & mask]); - - _mm_storeu_si128((__m128i *)&src_hdls[i], - _mm_unpacklo_epi64(hdls0, hdls1)); - _mm_storeu_si128((__m128i *)&dst_hdls[i], - _mm_unpackhi_epi64(hdls0, hdls1)); - } - for (; i < count; i++, read++) { - uintptr_t *hdls = (uintptr_t *)&ioat->hdls[read & mask]; - src_hdls[i] = hdls[0]; - dst_hdls[i] = hdls[1]; - } - -end: - ioat->next_read = read; - ioat->xstats.completed += count; - return count; -} - -static inline uint16_t -rte_ioat_burst_capacity(int dev_id) -{ - enum rte_ioat_dev_type *type = - (enum rte_ioat_dev_type *)rte_rawdevs[dev_id].dev_private; - if (*type == RTE_IDXD_DEV) - return __idxd_burst_capacity(dev_id); - else - return __ioat_burst_capacity(dev_id); -} - -static inline int -rte_ioat_enqueue_fill(int dev_id, uint64_t pattern, phys_addr_t dst, - unsigned int len, uintptr_t dst_hdl) -{ - enum rte_ioat_dev_type *type = - (enum rte_ioat_dev_type *)rte_rawdevs[dev_id].dev_private; - if (*type == RTE_IDXD_DEV) - return __idxd_enqueue_fill(dev_id, pattern, dst, len, dst_hdl); - else - return __ioat_enqueue_fill(dev_id, pattern, dst, len, dst_hdl); -} - -static inline int -rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst, - unsigned int length, uintptr_t src_hdl, uintptr_t dst_hdl) -{ - enum rte_ioat_dev_type *type = - (enum rte_ioat_dev_type *)rte_rawdevs[dev_id].dev_private; - if (*type == RTE_IDXD_DEV) - return __idxd_enqueue_copy(dev_id, src, dst, length, - src_hdl, dst_hdl); - else - return __ioat_enqueue_copy(dev_id, src, dst, length, - src_hdl, dst_hdl); -} - -static inline int -rte_ioat_fence(int dev_id) -{ - enum rte_ioat_dev_type *type = - (enum rte_ioat_dev_type *)rte_rawdevs[dev_id].dev_private; - if (*type == RTE_IDXD_DEV) - return __idxd_fence(dev_id); - else - return __ioat_fence(dev_id); -} - -static inline int -rte_ioat_perform_ops(int dev_id) -{ - enum rte_ioat_dev_type *type = - (enum rte_ioat_dev_type *)rte_rawdevs[dev_id].dev_private; - if (*type == RTE_IDXD_DEV) - return __idxd_perform_ops(dev_id); - else - return __ioat_perform_ops(dev_id); -} - -static inline int -rte_ioat_completed_ops(int dev_id, uint8_t max_copies, - uint32_t *status, uint8_t *num_unsuccessful, - uintptr_t *src_hdls, uintptr_t *dst_hdls) -{ - enum rte_ioat_dev_type *type = - (enum rte_ioat_dev_type *)rte_rawdevs[dev_id].dev_private; - uint8_t tmp; /* used so functions don't need to check for null parameter */ - - if (num_unsuccessful == NULL) - num_unsuccessful = &tmp; - - *num_unsuccessful = 0; - if (*type == RTE_IDXD_DEV) - return __idxd_completed_ops(dev_id, max_copies, status, num_unsuccessful, - src_hdls, dst_hdls); - else - return __ioat_completed_ops(dev_id, max_copies, src_hdls, dst_hdls); -} - -static inline void -__rte_deprecated_msg("use rte_ioat_perform_ops() instead") -rte_ioat_do_copies(int dev_id) { rte_ioat_perform_ops(dev_id); } - -static inline int -__rte_deprecated_msg("use rte_ioat_completed_ops() instead") -rte_ioat_completed_copies(int dev_id, uint8_t max_copies, - uintptr_t *src_hdls, uintptr_t *dst_hdls) -{ - return rte_ioat_completed_ops(dev_id, max_copies, NULL, NULL, - src_hdls, dst_hdls); -} - -#endif /* _RTE_IOAT_RAWDEV_FNS_H_ */ diff --git a/drivers/raw/ioat/version.map b/drivers/raw/ioat/version.map deleted file mode 100644 index 78c3585d7c..0000000000 --- a/drivers/raw/ioat/version.map +++ /dev/null @@ -1,3 +0,0 @@ -DPDK_23 { - local: *; -}; diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build index 48e84d586e..05cad143fe 100644 --- a/drivers/raw/meson.build +++ b/drivers/raw/meson.build @@ -10,7 +10,6 @@ drivers = [ 'cnxk_gpio', 'dpaa2_cmdif', 'ifpga', - 'ioat', 'ntb', 'skeleton', ] -- 2.34.1