* [dpdk-dev] [PATCH v2 1/5] bbdev: librte_bbdev library @ 2017-10-18 2:14 Amr Mokhtar 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 2/5] bbdev: PMD drivers (null/turbo_sw) Amr Mokhtar ` (4 more replies) 0 siblings, 5 replies; 6+ messages in thread From: Amr Mokhtar @ 2017-10-18 2:14 UTC (permalink / raw) To: dev Cc: thomas, anatoly.burakov, pablo.de.lara.guarch, niall.power, chris.macnamara, Amr Mokhtar - BBDEV library files - BBDEV is tagged as EXPERIMENTAL - Makefiles and configuration macros definition - The bbdev framework and the 'null' driver are enabled by default - The bbdev test framework is enabled by default - Release Notes of the initial version and MAINTAINERS Signed-off-by: Amr Mokhtar <amr.mokhtar@intel.com> --- MAINTAINERS | 10 + config/common_base | 23 + doc/guides/rel_notes/release_17_11.rst | 10 + lib/Makefile | 3 + lib/librte_bbdev/Makefile | 56 ++ lib/librte_bbdev/rte_bbdev.c | 1095 ++++++++++++++++++++++++++++++++ lib/librte_bbdev/rte_bbdev.h | 741 +++++++++++++++++++++ lib/librte_bbdev/rte_bbdev_op.h | 514 +++++++++++++++ lib/librte_bbdev/rte_bbdev_pci.h | 288 +++++++++ lib/librte_bbdev/rte_bbdev_pmd.h | 223 +++++++ lib/librte_bbdev/rte_bbdev_vdev.h | 102 +++ lib/librte_bbdev/rte_bbdev_version.map | 37 ++ mk/rte.app.mk | 13 + 13 files changed, 3115 insertions(+) create mode 100644 lib/librte_bbdev/Makefile create mode 100644 lib/librte_bbdev/rte_bbdev.c create mode 100644 lib/librte_bbdev/rte_bbdev.h create mode 100644 lib/librte_bbdev/rte_bbdev_op.h create mode 100644 lib/librte_bbdev/rte_bbdev_pci.h create mode 100644 lib/librte_bbdev/rte_bbdev_pmd.h create mode 100644 lib/librte_bbdev/rte_bbdev_vdev.h create mode 100644 lib/librte_bbdev/rte_bbdev_version.map diff --git a/MAINTAINERS b/MAINTAINERS index 2a58378..df63f3f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -275,6 +275,16 @@ T: git://dpdk.org/next/dpdk-next-eventdev F: lib/librte_eventdev/*eth_rx_adapter* F: test/test/test_event_eth_rx_adapter.c +BBDEV API - EXPERIMENTAL +M: Amr Mokhtar <amr.mokhtar@intel.com> +F: lib/librte_bbdev/ +F: drivers/bbdev/ +F: app/test-bbdev +F: examples/bbdev_app/ +F: doc/guides/bbdevs/ +F: doc/guides/prog_guide/bbdev.rst +F: doc/guides/sample_app_ug/bbdev_app.rst +F: doc/guides/tools/testbbdev.rst Networking Drivers ------------------ diff --git a/config/common_base b/config/common_base index d9471e8..7e75701 100644 --- a/config/common_base +++ b/config/common_base @@ -573,6 +573,24 @@ CONFIG_RTE_LIBRTE_PMD_SW_EVENTDEV_DEBUG=n CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF=y CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF_DEBUG=n +# Compile generic wireless base band device library +# EXPERIMENTAL: API may change without prior notice +# +CONFIG_RTE_LIBRTE_BBDEV=y +CONFIG_RTE_LIBRTE_BBDEV_DEBUG=n +CONFIG_RTE_BBDEV_MAX_DEVS=128 +CONFIG_RTE_BBDEV_NAME_MAX_LEN=64 + +# +# Compile PMD for NULL bbdev device +# +CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y + +# +# Compile PMD for turbo software bbdev device +# +CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n + # # Compile librte_ring # @@ -780,6 +798,11 @@ CONFIG_RTE_APP_TEST=y CONFIG_RTE_APP_TEST_RESOURCE_TAR=n # +# Compile the bbdev test application +# +CONFIG_RTE_TEST_BBDEV=y + +# # Compile the PMD test application # CONFIG_RTE_TEST_PMD=y diff --git a/doc/guides/rel_notes/release_17_11.rst b/doc/guides/rel_notes/release_17_11.rst index 8db35f5..ab1c16b 100644 --- a/doc/guides/rel_notes/release_17_11.rst +++ b/doc/guides/rel_notes/release_17_11.rst @@ -41,6 +41,16 @@ New Features Also, make sure to start the actual text at the margin. ========================================================= +* **Added Wireless Base Band Device (bbdev).** + + The Wireless Baseband library provides a common programming framework that + abstracts HW accelerators based on FPGA and/or Fixed Function Accelerators that + assist with 3gpp Physical Layer processing. Furthermore, it decouples the + application from the compute-intensive wireless functions by abstracting their + optimized libraries to appear as virtual bbdev devices. + + The current release only supports Turbo Code FEC function. + * **Extended port_id range from uint8_t to uint16_t.** Increased port_id range from 8 bits to 16 bits in order to support more than diff --git a/lib/Makefile b/lib/Makefile index 86d475f..3641eb5 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -52,6 +52,9 @@ DEPDIRS-librte_cryptodev := librte_eal librte_mempool librte_ring librte_mbuf DEPDIRS-librte_cryptodev += librte_kvargs DIRS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += librte_eventdev DEPDIRS-librte_eventdev := librte_eal librte_ring librte_ether librte_hash +DIRS-$(CONFIG_RTE_LIBRTE_BBDEV) += librte_bbdev +DEPDIRS-librte_bbdev := librte_eal librte_mempool librte_ring librte_mbuf +DEPDIRS-librte_bbdev += librte_kvargs DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += librte_vhost DEPDIRS-librte_vhost := librte_eal librte_mempool librte_mbuf librte_ether DIRS-$(CONFIG_RTE_LIBRTE_HASH) += librte_hash diff --git a/lib/librte_bbdev/Makefile b/lib/librte_bbdev/Makefile new file mode 100644 index 0000000..60c47e2 --- /dev/null +++ b/lib/librte_bbdev/Makefile @@ -0,0 +1,56 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_bbdev.a + +# library version +LIBABIVER := 1 + +# build flags +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) + +# library source files +SRCS-y += rte_bbdev.c + +# export include files +SYMLINK-y-include += rte_bbdev_op.h +SYMLINK-y-include += rte_bbdev.h +SYMLINK-y-include += rte_bbdev_pmd.h +SYMLINK-y-include += rte_bbdev_pci.h +SYMLINK-y-include += rte_bbdev_vdev.h + +# versioning export map +EXPORT_MAP := rte_bbdev_version.map + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/lib/librte_bbdev/rte_bbdev.c b/lib/librte_bbdev/rte_bbdev.c new file mode 100644 index 0000000..16f2544 --- /dev/null +++ b/lib/librte_bbdev/rte_bbdev.c @@ -0,0 +1,1095 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stdint.h> +#include <string.h> +#include <stdbool.h> + +#include <rte_common.h> +#include <rte_errno.h> +#include <rte_log.h> +#include <rte_debug.h> +#include <rte_eal.h> +#include <rte_malloc.h> +#include <rte_mempool.h> +#include <rte_memzone.h> +#include <rte_lcore.h> +#include <rte_dev.h> +#include <rte_spinlock.h> +#include <rte_tailq.h> +#include <rte_interrupts.h> + +#include "rte_bbdev_op.h" +#include "rte_bbdev.h" +#include "rte_bbdev_pmd.h" + +#define DEV_NAME "BBDEV" + + +/* Helper macro to check dev_id is valid */ +#define VALID_DEV_OR_RET_ERR(dev, dev_id) do { \ + if (dev == NULL) { \ + rte_bbdev_log(ERR, "device %u is invalid", dev_id); \ + return -ENODEV; \ + } \ +} while (0) + +/* Helper macro to check dev_ops is valid */ +#define VALID_DEV_OPS_OR_RET_ERR(dev, dev_id) do { \ + if (dev->dev_ops == NULL) { \ + rte_bbdev_log(ERR, "NULL dev_ops structure in device %u", \ + dev_id); \ + return -ENODEV; \ + } \ +} while (0) + +/* Helper macro to check that driver implements required function pointer */ +#define VALID_FUNC_OR_RET_ERR(func, dev_id) do { \ + if (func == NULL) { \ + rte_bbdev_log(ERR, "device %u does not support %s", \ + dev_id, #func); \ + return -ENOTSUP; \ + } \ +} while (0) + +/* Helper macro to check that queue is valid */ +#define VALID_QUEUE_OR_RET_ERR(queue_id, dev) do { \ + if (queue_id >= dev->data->num_queues) { \ + rte_bbdev_log(ERR, "Invalid queue_id %u for device %u", \ + queue_id, dev->data->dev_id); \ + return -ERANGE; \ + } \ +} while (0) + +/* List of callback functions registered by an application */ +struct rte_bbdev_callback { + TAILQ_ENTRY(rte_bbdev_callback) next; /* Callbacks list */ + rte_bbdev_cb_fn cb_fn; /* Callback address */ + void *cb_arg; /* Parameter for callback */ + void *ret_param; /* Return parameter */ + enum rte_bbdev_event_type event; /* Interrupt event type */ + uint32_t active; /* Callback is executing */ +}; + +/* spinlock for bbdev device callbacks */ +static rte_spinlock_t rte_bbdev_cb_lock = RTE_SPINLOCK_INITIALIZER; + +/* Global array of all devices. This is not static because it's used by the + * inline enqueue and dequeue functions + */ +struct rte_bbdev rte_bbdev_devices[RTE_BBDEV_MAX_DEVS]; + +/* Global array with rte_bbdev_data structures */ +static struct rte_bbdev_data *rte_bbdev_data; + +/* Memzone name for global bbdev data pool */ +static const char *MZ_RTE_BBDEV_DATA = "rte_bbdev_data"; + +/* Number of currently valid devices */ +static uint16_t num_devs; + +/* Return pointer to device structure, with validity check */ +static struct rte_bbdev * +get_dev(uint16_t dev_id) +{ + if (rte_bbdev_is_valid(dev_id)) + return &rte_bbdev_devices[dev_id]; + return NULL; +} + +/* Allocate global data array */ +static void +rte_bbdev_data_alloc(void) +{ + const unsigned int flags = 0; + const struct rte_memzone *mz; + + if (rte_eal_process_type() == RTE_PROC_PRIMARY) { + mz = rte_memzone_reserve(MZ_RTE_BBDEV_DATA, + RTE_BBDEV_MAX_DEVS * sizeof(*rte_bbdev_data), + rte_socket_id(), flags); + } else + mz = rte_memzone_lookup(MZ_RTE_BBDEV_DATA); + if (mz == NULL) + rte_panic("Cannot allocate memzone for bbdev port data\n"); + + rte_bbdev_data = mz->addr; + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + memset(rte_bbdev_data, 0, + RTE_BBDEV_MAX_DEVS * sizeof(*rte_bbdev_data)); +} + +/* Find lowest device id with no attached device */ +static uint16_t +find_free_dev_id(void) +{ + uint16_t i; + for (i = 0; i < RTE_BBDEV_MAX_DEVS; i++) { + if (rte_bbdev_devices[i].state == RTE_BBDEV_UNUSED) + return i; + } + return RTE_BBDEV_MAX_DEVS; +} + +struct rte_bbdev * +rte_bbdev_allocate(const char *name) +{ + int ret; + struct rte_bbdev *bbdev; + uint16_t dev_id; + + if (name == NULL) { + rte_bbdev_log(ERR, "Invalid null device name"); + return NULL; + } + + if (rte_bbdev_get_named_dev(name) != NULL) { + rte_bbdev_log(ERR, "Device \"%s\" is already allocated", name); + return NULL; + } + + dev_id = find_free_dev_id(); + if (dev_id == RTE_BBDEV_MAX_DEVS) { + rte_bbdev_log(ERR, "Reached maximum number of devices"); + return NULL; + } + + bbdev = &rte_bbdev_devices[dev_id]; + + if (rte_bbdev_data == NULL) + rte_bbdev_data_alloc(); + + bbdev->data = &rte_bbdev_data[dev_id]; + memset(bbdev->data, 0, sizeof(*bbdev->data)); + + bbdev->data->dev_id = dev_id; + bbdev->state = RTE_BBDEV_INITALIZED; + + ret = snprintf(bbdev->data->name, RTE_BBDEV_NAME_MAX_LEN, "%s", name); + if ((ret < 0) || (ret >= RTE_BBDEV_NAME_MAX_LEN)) { + rte_bbdev_log(ERR, "Copying device name \"%s\" failed", name); + return NULL; + } + + /* init user callbacks */ + TAILQ_INIT(&(bbdev->list_cbs)); + + num_devs++; + + rte_bbdev_log_debug("Initialised device %s (id = %u). Num devices = %u", + name, dev_id, num_devs); + + return bbdev; +} + +int +rte_bbdev_release(struct rte_bbdev *bbdev) +{ + uint16_t dev_id; + struct rte_bbdev_callback *cb, *next; + + if (bbdev == NULL) { + rte_bbdev_log(ERR, "NULL bbdev"); + return -ENODEV; + } + dev_id = bbdev->data->dev_id; + + /* free all callbacks from the device's list */ + for (cb = TAILQ_FIRST(&bbdev->list_cbs); cb != NULL; cb = next) { + + next = TAILQ_NEXT(cb, next); + TAILQ_REMOVE(&(bbdev->list_cbs), cb, next); + rte_free(cb); + } + + memset(bbdev, 0, sizeof(*bbdev)); + num_devs--; + bbdev->state = RTE_BBDEV_UNUSED; + + rte_bbdev_log_debug( + "Un-initialised device id = %u. Num devices = %u", + dev_id, num_devs); + return 0; +} + +struct rte_bbdev * +rte_bbdev_get_named_dev(const char *name) +{ + unsigned int i; + + if (name == NULL) { + rte_bbdev_log(ERR, "NULL driver name"); + return NULL; + } + + for (i = 0; i < RTE_BBDEV_MAX_DEVS; i++) { + struct rte_bbdev *dev = get_dev(i); + if (dev && (strncmp(dev->data->name, + name, RTE_BBDEV_NAME_MAX_LEN) == 0)) + return dev; + } + + return NULL; +} + +uint16_t +rte_bbdev_count(void) +{ + return num_devs; +} + +bool +rte_bbdev_is_valid(uint16_t dev_id) +{ + if ((dev_id < RTE_BBDEV_MAX_DEVS) && + rte_bbdev_devices[dev_id].state == RTE_BBDEV_INITALIZED) + return true; + return false; +} + +uint16_t +rte_bbdev_find_next(uint16_t dev_id) +{ + dev_id++; + for (; dev_id < RTE_BBDEV_MAX_DEVS; dev_id++) + if (rte_bbdev_is_valid(dev_id)) + break; + return dev_id; +} + +int +rte_bbdev_setup_queues(uint16_t dev_id, uint16_t num_queues, int socket_id) +{ + unsigned int i; + int ret; + struct rte_bbdev_driver_info dev_info; + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + + VALID_DEV_OPS_OR_RET_ERR(dev, dev_id); + + if (dev->data->started) { + rte_bbdev_log(ERR, + "Device %u cannot be configured when started", + dev_id); + return -EBUSY; + } + + /* Get device driver information to get max number of queues */ + VALID_FUNC_OR_RET_ERR(dev->dev_ops->info_get, dev_id); + memset(&dev_info, 0, sizeof(dev_info)); + dev->dev_ops->info_get(dev, &dev_info); + + if ((num_queues == 0) || (num_queues > dev_info.max_num_queues)) { + rte_bbdev_log(ERR, + "Device %u supports 0 < N <= %u queues, not %u", + dev_id, dev_info.max_num_queues, num_queues); + return -EINVAL; + } + + /* If re-configuration, get driver to free existing internal memory */ + if (dev->data->queues != NULL) { + VALID_FUNC_OR_RET_ERR(dev->dev_ops->queue_release, dev_id); + for (i = 0; i < dev->data->num_queues; i++) { + int ret = dev->dev_ops->queue_release(dev, i); + if (ret < 0) { + rte_bbdev_log(ERR, + "Device %u queue %u release failed", + dev_id, i); + return ret; + } + } + /* Call optional device close */ + if (dev->dev_ops->close) { + ret = dev->dev_ops->close(dev); + if (ret < 0) { + rte_bbdev_log(ERR, + "Device %u couldn't be closed", + dev_id); + return ret; + } + } + rte_free(dev->data->queues); + } + + /* Allocate queue pointers */ + dev->data->queues = rte_calloc_socket(DEV_NAME, num_queues, + sizeof(dev->data->queues[0]), RTE_CACHE_LINE_SIZE, + dev->data->socket_id); + if (dev->data->queues == NULL) { + rte_bbdev_log(ERR, + "calloc of %u queues for device %u on socket %i failed", + num_queues, dev_id, dev->data->socket_id); + return -ENOMEM; + } + + dev->data->num_queues = num_queues; + + /* Call optional device configuration */ + if (dev->dev_ops->setup_queues) { + ret = dev->dev_ops->setup_queues(dev, num_queues, socket_id); + if (ret < 0) { + rte_bbdev_log(ERR, + "Device %u memory configuration failed", + dev_id); + goto error; + } + } + + rte_bbdev_log_debug("Device %u set up with %u queues", dev_id, + num_queues); + return 0; + +error: + dev->data->num_queues = 0; + rte_free(dev->data->queues); + dev->data->queues = NULL; + return ret; +} + +int +rte_bbdev_intr_enable(uint16_t dev_id) +{ + int ret; + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + + VALID_DEV_OPS_OR_RET_ERR(dev, dev_id); + + if (dev->data->started) { + rte_bbdev_log(ERR, + "Device %u cannot be configured when started", + dev_id); + return -EBUSY; + } + + if (dev->dev_ops->intr_enable) { + ret = dev->dev_ops->intr_enable(dev); + if (ret < 0) { + rte_bbdev_log(ERR, + "Device %u interrupts configuration failed", + dev_id); + return ret; + } + rte_bbdev_log_debug("Enabled interrupts for dev %u", dev_id); + return 0; + } + + rte_bbdev_log(ERR, "Device %u doesn't support interrupts", dev_id); + return -ENOTSUP; +} + +int +rte_bbdev_queue_configure(uint16_t dev_id, uint16_t queue_id, + const struct rte_bbdev_queue_conf *conf) +{ + int ret = 0; + struct rte_bbdev_driver_info dev_info; + struct rte_bbdev *dev = get_dev(dev_id); + const struct rte_bbdev_op_cap *p; + struct rte_bbdev_queue_conf *stored_conf; + VALID_DEV_OR_RET_ERR(dev, dev_id); + + VALID_DEV_OPS_OR_RET_ERR(dev, dev_id); + + VALID_QUEUE_OR_RET_ERR(queue_id, dev); + + if (dev->data->queues[queue_id].started || dev->data->started) { + rte_bbdev_log(ERR, + "Queue %u of device %u cannot be configured when started", + queue_id, dev_id); + return -EBUSY; + } + + VALID_FUNC_OR_RET_ERR(dev->dev_ops->queue_release, dev_id); + VALID_FUNC_OR_RET_ERR(dev->dev_ops->queue_setup, dev_id); + + /* Get device driver information to verify config is valid */ + VALID_FUNC_OR_RET_ERR(dev->dev_ops->info_get, dev_id); + memset(&dev_info, 0, sizeof(dev_info)); + dev->dev_ops->info_get(dev, &dev_info); + + /* Check configuration is valid */ + if (conf != NULL) { + if ((conf->op_type == RTE_BBDEV_OP_NONE) && + (dev_info.capabilities[0].type == + RTE_BBDEV_OP_NONE)) { + ret = 1; + } else + for (p = dev_info.capabilities; + p->type != RTE_BBDEV_OP_NONE; p++) { + if (conf->op_type == p->type) { + ret = 1; + break; + } + } + if (ret == 0) { + rte_bbdev_log(ERR, "Invalid operation type"); + return -EINVAL; + } + if (conf->queue_size > dev_info.queue_size_lim) { + rte_bbdev_log(ERR, + "Size (%u) of queue %u of device %u must be: <= %u", + conf->queue_size, queue_id, dev_id, + dev_info.queue_size_lim); + return -EINVAL; + } + if (!rte_is_power_of_2(conf->queue_size)) { + rte_bbdev_log(ERR, + "Size (%u) of queue %u of device %u must be a power of 2", + conf->queue_size, queue_id, dev_id); + return -EINVAL; + } + if (conf->priority > dev_info.max_queue_priority) { + rte_bbdev_log(ERR, + "Priority (%u) of queue %u of bdev %u must be <= %u", + conf->priority, queue_id, dev_id, + dev_info.max_queue_priority); + return -EINVAL; + } + } + + /* Release existing queue (in case of queue reconfiguration) */ + if (dev->data->queues[queue_id].queue_private != NULL) { + ret = dev->dev_ops->queue_release(dev, queue_id); + if (ret < 0) { + rte_bbdev_log(ERR, "Device %u queue %u release failed", + dev_id, queue_id); + return ret; + } + } + + /* Get driver to setup the queue */ + ret = dev->dev_ops->queue_setup(dev, queue_id, (conf != NULL) ? + conf : &dev_info.default_queue_conf); + if (ret < 0) { + rte_bbdev_log(ERR, + "Device %u queue %u setup failed", dev_id, + queue_id); + return ret; + } + + /* Store configuration */ + stored_conf = &dev->data->queues[queue_id].conf; + memcpy(stored_conf, + (conf != NULL) ? conf : &dev_info.default_queue_conf, + sizeof(*stored_conf)); + + rte_bbdev_log_debug("Configured dev%uq%u (size=%u, type=%s, prio=%u)", + dev_id, queue_id, stored_conf->queue_size, + rte_bbdev_op_type_str(stored_conf->op_type), + stored_conf->priority); + + return 0; +} + +int +rte_bbdev_start(uint16_t dev_id) +{ + int i; + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + + VALID_DEV_OPS_OR_RET_ERR(dev, dev_id); + + if (dev->data->started) { + rte_bbdev_log_debug("Device %u is already started", dev_id); + return 0; + } + + if (dev->dev_ops->start) { + int ret = dev->dev_ops->start(dev); + if (ret < 0) { + rte_bbdev_log(ERR, "Device %u start failed", dev_id); + return ret; + } + } + + /* Store new state */ + for (i = 0; i < dev->data->num_queues; i++) + if (!dev->data->queues[i].conf.deferred_start) + dev->data->queues[i].started = true; + dev->data->started = true; + + rte_bbdev_log_debug("Started device %u", dev_id); + return 0; +} + +int +rte_bbdev_stop(uint16_t dev_id) +{ + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + + VALID_DEV_OPS_OR_RET_ERR(dev, dev_id); + + if (!dev->data->started) { + rte_bbdev_log_debug("Device %u is already stopped", dev_id); + return 0; + } + + if (dev->dev_ops->stop) + dev->dev_ops->stop(dev); + dev->data->started = false; + + rte_bbdev_log_debug("Stopped device %u", dev_id); + return 0; +} + +int +rte_bbdev_close(uint16_t dev_id) +{ + int ret; + uint16_t i; + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + + VALID_DEV_OPS_OR_RET_ERR(dev, dev_id); + + if (dev->data->started) { + ret = rte_bbdev_stop(dev_id); + if (ret < 0) { + rte_bbdev_log(ERR, "Device %u stop failed", dev_id); + return ret; + } + } + + /* Free memory used by queues */ + for (i = 0; i < dev->data->num_queues; i++) { + ret = dev->dev_ops->queue_release(dev, i); + if (ret < 0) { + rte_bbdev_log(ERR, "Device %u queue %u release failed", + dev_id, i); + return ret; + } + } + rte_free(dev->data->queues); + + if (dev->dev_ops->close) { + ret = dev->dev_ops->close(dev); + if (ret < 0) { + rte_bbdev_log(ERR, "Device %u close failed", dev_id); + return ret; + } + } + + /* Clear configuration */ + dev->data->queues = NULL; + dev->data->num_queues = 0; + + rte_bbdev_log_debug("Closed device %u", dev_id); + return 0; +} + +int +rte_bbdev_queue_start(uint16_t dev_id, uint16_t queue_id) +{ + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + + VALID_DEV_OPS_OR_RET_ERR(dev, dev_id); + + VALID_QUEUE_OR_RET_ERR(queue_id, dev); + + if (dev->data->queues[queue_id].started) { + rte_bbdev_log_debug("Queue %u of device %u already started", + queue_id, dev_id); + return 0; + } + + if (dev->dev_ops->queue_start) { + int ret = dev->dev_ops->queue_start(dev, queue_id); + if (ret < 0) { + rte_bbdev_log(ERR, "Device %u queue %u start failed", + dev_id, queue_id); + return ret; + } + } + dev->data->queues[queue_id].started = true; + + rte_bbdev_log_debug("Started queue %u of device %u", queue_id, dev_id); + return 0; +} + +int +rte_bbdev_queue_stop(uint16_t dev_id, uint16_t queue_id) +{ + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + + VALID_DEV_OPS_OR_RET_ERR(dev, dev_id); + + VALID_QUEUE_OR_RET_ERR(queue_id, dev); + + if (!dev->data->queues[queue_id].started) { + rte_bbdev_log_debug("Queue %u of device %u already stopped", + queue_id, dev_id); + return 0; + } + + if (dev->dev_ops->queue_stop) { + int ret = dev->dev_ops->queue_stop(dev, queue_id); + if (ret < 0) { + rte_bbdev_log(ERR, "Device %u queue %u stop failed", + dev_id, queue_id); + return ret; + } + } + dev->data->queues[queue_id].started = false; + + rte_bbdev_log_debug("Stopped queue %u of device %u", queue_id, dev_id); + return 0; +} + +/* Get device statistics */ +static void +get_stats_from_queues(struct rte_bbdev *dev, struct rte_bbdev_stats *stats) +{ + unsigned int q_id; + for (q_id = 0; q_id < dev->data->num_queues; q_id++) { + struct rte_bbdev_stats *q_stats = + &dev->data->queues[q_id].queue_stats; + + stats->enqueued_count += q_stats->enqueued_count; + stats->dequeued_count += q_stats->dequeued_count; + stats->enqueue_err_count += q_stats->enqueue_err_count; + stats->dequeue_err_count += q_stats->dequeue_err_count; + } + rte_bbdev_log_debug("Got stats on %u", dev->data->dev_id); +} + +static void +reset_stats_in_queues(struct rte_bbdev *dev) +{ + unsigned int q_id; + for (q_id = 0; q_id < dev->data->num_queues; q_id++) { + struct rte_bbdev_stats *q_stats = + &dev->data->queues[q_id].queue_stats; + + memset(q_stats, 0, sizeof(*q_stats)); + } + rte_bbdev_log_debug("Reset stats on %u", dev->data->dev_id); +} + +int +rte_bbdev_stats_get(uint16_t dev_id, struct rte_bbdev_stats *stats) +{ + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + + VALID_DEV_OPS_OR_RET_ERR(dev, dev_id); + + if (stats == NULL) { + rte_bbdev_log(ERR, "NULL stats structure"); + return -EINVAL; + } + + memset(stats, 0, sizeof(*stats)); + if (dev->dev_ops->stats_get != NULL) + dev->dev_ops->stats_get(dev, stats); + else + get_stats_from_queues(dev, stats); + + rte_bbdev_log_debug("Retrieved stats of device %u", dev_id); + return 0; +} + +int +rte_bbdev_stats_reset(uint16_t dev_id) +{ + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + + VALID_DEV_OPS_OR_RET_ERR(dev, dev_id); + + if (dev->dev_ops->stats_reset != NULL) + dev->dev_ops->stats_reset(dev); + else + reset_stats_in_queues(dev); + + rte_bbdev_log_debug("Reset stats of device %u", dev_id); + return 0; +} + +int +rte_bbdev_info_get(uint16_t dev_id, struct rte_bbdev_info *dev_info) +{ + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + + VALID_FUNC_OR_RET_ERR(dev->dev_ops->info_get, dev_id); + + if (dev_info == NULL) { + rte_bbdev_log(ERR, "NULL dev info structure"); + return -EINVAL; + } + + /* Copy data maintained by device interface layer */ + memset(dev_info, 0, sizeof(*dev_info)); + dev_info->dev_name = dev->data->name; + dev_info->num_queues = dev->data->num_queues; + dev_info->bus = rte_bus_find_by_device(dev->device); + dev_info->socket_id = dev->data->socket_id; + dev_info->started = dev->data->started; + + /* Copy data maintained by device driver layer */ + dev->dev_ops->info_get(dev, &dev_info->drv); + + rte_bbdev_log_debug("Retrieved info of device %u", dev_id); + return 0; +} + +int +rte_bbdev_queue_info_get(uint16_t dev_id, uint16_t queue_id, + struct rte_bbdev_queue_info *dev_info) +{ + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + + VALID_QUEUE_OR_RET_ERR(queue_id, dev); + + if (dev_info == NULL) { + rte_bbdev_log(ERR, "NULL queue info structure"); + return -EINVAL; + } + + /* Copy data to output */ + memset(dev_info, 0, sizeof(*dev_info)); + dev_info->conf = dev->data->queues[queue_id].conf; + dev_info->started = dev->data->queues[queue_id].started; + + rte_bbdev_log_debug("Retrieved info of queue %u of device %u", + queue_id, dev_id); + return 0; +} + +/* Calculate size needed to store bbdev_op, depending on type */ +static unsigned int +get_bbdev_op_size(enum rte_bbdev_op_type type) +{ + unsigned int result = 0; + switch (type) { + case RTE_BBDEV_OP_NONE: + result = RTE_MAX(sizeof(struct rte_bbdev_dec_op), + sizeof(struct rte_bbdev_enc_op)); + break; + case RTE_BBDEV_OP_TURBO_DEC: + result = sizeof(struct rte_bbdev_dec_op); + break; + case RTE_BBDEV_OP_TURBO_ENC: + result = sizeof(struct rte_bbdev_enc_op); + break; + default: + break; + } + + return result; +} + +/* Initialise a bbdev_op structure */ +static void +bbdev_op_init(struct rte_mempool *mempool, void *arg, void *element, + __rte_unused unsigned int n) +{ + enum rte_bbdev_op_type type = *(enum rte_bbdev_op_type *)arg; + + if (type == RTE_BBDEV_OP_TURBO_DEC) { + struct rte_bbdev_dec_op *op = element; + memset(op, 0, mempool->elt_size); + op->mempool = mempool; + } else if (type == RTE_BBDEV_OP_TURBO_ENC) { + struct rte_bbdev_enc_op *op = element; + memset(op, 0, mempool->elt_size); + op->mempool = mempool; + } +} + +struct rte_mempool * +rte_bbdev_op_pool_create(const char *name, enum rte_bbdev_op_type type, + unsigned int num_elements, unsigned int cache_size, + int socket_id) +{ + struct rte_bbdev_op_pool_private *priv; + struct rte_mempool *mp; + + if (name == NULL) { + rte_bbdev_log(ERR, "NULL name for op pool"); + return NULL; + } + + if (type >= RTE_BBDEV_OP_TYPE_COUNT) { + rte_bbdev_log(ERR, + "Invalid op type (%u), should be less than %u", + type, RTE_BBDEV_OP_TYPE_COUNT); + return NULL; + } + + mp = rte_mempool_create(name, num_elements, get_bbdev_op_size(type), + cache_size, sizeof(struct rte_bbdev_op_pool_private), + NULL, NULL, bbdev_op_init, &type, socket_id, 0); + if (mp == NULL) { + rte_bbdev_log(ERR, + "Failed to create op pool %s (num ops=%u, op size=%u) with error: %s", + name, num_elements, get_bbdev_op_size(type), + rte_strerror(rte_errno)); + return NULL; + } + + rte_bbdev_log_debug( + "Op pool %s created for %u ops (type=%s, cache=%u, socket=%u, size=%u)", + name, num_elements, rte_bbdev_op_type_str(type), + cache_size, socket_id, get_bbdev_op_size(type)); + + priv = (struct rte_bbdev_op_pool_private *)rte_mempool_get_priv(mp); + priv->type = type; + + return mp; +} + +int +rte_bbdev_callback_register(uint16_t dev_id, enum rte_bbdev_event_type event, + rte_bbdev_cb_fn cb_fn, void *cb_arg) +{ + struct rte_bbdev_callback *user_cb; + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + + if (event >= RTE_BBDEV_EVENT_MAX) { + rte_bbdev_log(ERR, + "Invalid event type (%u), should be less than %u", + event, RTE_BBDEV_EVENT_MAX); + return -EINVAL; + } + + if (cb_fn == NULL) { + rte_bbdev_log(ERR, "NULL callback function"); + return -EINVAL; + } + + rte_spinlock_lock(&rte_bbdev_cb_lock); + + TAILQ_FOREACH(user_cb, &(dev->list_cbs), next) { + if (user_cb->cb_fn == cb_fn && + user_cb->cb_arg == cb_arg && + user_cb->event == event) + break; + } + + /* create a new callback. */ + if (user_cb == NULL) { + user_cb = rte_zmalloc("INTR_USER_CALLBACK", + sizeof(struct rte_bbdev_callback), 0); + if (user_cb != NULL) { + user_cb->cb_fn = cb_fn; + user_cb->cb_arg = cb_arg; + user_cb->event = event; + TAILQ_INSERT_TAIL(&(dev->list_cbs), user_cb, next); + } + } + + rte_spinlock_unlock(&rte_bbdev_cb_lock); + return (user_cb == NULL) ? -ENOMEM : 0; +} + +int +rte_bbdev_callback_unregister(uint16_t dev_id, enum rte_bbdev_event_type event, + rte_bbdev_cb_fn cb_fn, void *cb_arg) +{ + int ret = 0; + struct rte_bbdev_callback *cb, *next; + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + + if (event >= RTE_BBDEV_EVENT_MAX) { + rte_bbdev_log(ERR, + "Invalid event type (%u), should be less than %u", + event, RTE_BBDEV_EVENT_MAX); + return -EINVAL; + } + + if (cb_fn == NULL) { + rte_bbdev_log(ERR, + "NULL callback function cannot be unregistered"); + return -EINVAL; + } + + dev = &rte_bbdev_devices[dev_id]; + rte_spinlock_lock(&rte_bbdev_cb_lock); + + for (cb = TAILQ_FIRST(&dev->list_cbs); cb != NULL; cb = next) { + + next = TAILQ_NEXT(cb, next); + + if (cb->cb_fn != cb_fn || cb->event != event || + (cb_arg != (void *)-1 && cb->cb_arg != cb_arg)) + continue; + + /* If this callback is not executing right now, remove it. */ + if (cb->active == 0) { + TAILQ_REMOVE(&(dev->list_cbs), cb, next); + rte_free(cb); + } else + ret = -EAGAIN; + } + + rte_spinlock_unlock(&rte_bbdev_cb_lock); + return ret; +} + +void +rte_bbdev_pmd_callback_process(struct rte_bbdev *dev, + enum rte_bbdev_event_type event, void *ret_param) +{ + struct rte_bbdev_callback *cb_lst; + struct rte_bbdev_callback dev_cb; + + if (dev == NULL) { + rte_bbdev_log(ERR, "NULL device"); + return; + } + + if (dev->data == NULL) { + rte_bbdev_log(ERR, "NULL data structure"); + return; + } + + if (event >= RTE_BBDEV_EVENT_MAX) { + rte_bbdev_log(ERR, + "Invalid event type (%u), should be less than %u", + event, RTE_BBDEV_EVENT_MAX); + return; + } + + rte_spinlock_lock(&rte_bbdev_cb_lock); + TAILQ_FOREACH(cb_lst, &(dev->list_cbs), next) { + if (cb_lst->cb_fn == NULL || cb_lst->event != event) + continue; + dev_cb = *cb_lst; + cb_lst->active = 1; + if (ret_param != NULL) + dev_cb.ret_param = ret_param; + + rte_spinlock_unlock(&rte_bbdev_cb_lock); + dev_cb.cb_fn(dev->data->dev_id, dev_cb.event, + dev_cb.cb_arg, dev_cb.ret_param); + rte_spinlock_lock(&rte_bbdev_cb_lock); + cb_lst->active = 0; + } + rte_spinlock_unlock(&rte_bbdev_cb_lock); +} + +int +rte_bbdev_queue_intr_enable(uint16_t dev_id, uint16_t queue_id) +{ + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + VALID_QUEUE_OR_RET_ERR(queue_id, dev); + VALID_DEV_OPS_OR_RET_ERR(dev, dev_id); + VALID_FUNC_OR_RET_ERR(dev->dev_ops->queue_intr_enable, dev_id); + return dev->dev_ops->queue_intr_enable(dev, queue_id); +} + +int +rte_bbdev_queue_intr_disable(uint16_t dev_id, uint16_t queue_id) +{ + struct rte_bbdev *dev = get_dev(dev_id); + VALID_DEV_OR_RET_ERR(dev, dev_id); + VALID_QUEUE_OR_RET_ERR(queue_id, dev); + VALID_DEV_OPS_OR_RET_ERR(dev, dev_id); + VALID_FUNC_OR_RET_ERR(dev->dev_ops->queue_intr_disable, dev_id); + return dev->dev_ops->queue_intr_disable(dev, queue_id); +} + +int +rte_bbdev_queue_intr_ctl(uint16_t dev_id, uint16_t queue_id, int epfd, int op, + void *data) +{ + uint32_t vec; + struct rte_bbdev *dev = get_dev(dev_id); + struct rte_intr_handle *intr_handle; + int ret; + + VALID_DEV_OR_RET_ERR(dev, dev_id); + VALID_QUEUE_OR_RET_ERR(queue_id, dev); + + intr_handle = dev->intr_handle; + if (!intr_handle || !intr_handle->intr_vec) { + rte_bbdev_log(ERR, "Device %u intr handle unset\n", dev_id); + return -ENOTSUP; + } + + if (queue_id >= RTE_MAX_RXTX_INTR_VEC_ID) { + rte_bbdev_log(ERR, "Device %u queue_id %u is too big\n", + dev_id, queue_id); + return -ENOTSUP; + } + + vec = intr_handle->intr_vec[queue_id]; + ret = rte_intr_rx_ctl(intr_handle, epfd, op, vec, data); + if (ret && (ret != -EEXIST)) { + rte_bbdev_log(ERR, + "dev %u q %u int ctl error op %d epfd %d vec %u\n", + dev_id, queue_id, op, epfd, vec); + return ret; + } + + return 0; +} + + +const char * +rte_bbdev_op_type_str(enum rte_bbdev_op_type op_type) +{ + static const char * const op_types[] = { + "RTE_BBDEV_OP_NONE", + "RTE_BBDEV_OP_TURBO_DEC", + "RTE_BBDEV_OP_TURBO_ENC", + }; + + if (op_type < RTE_BBDEV_OP_TYPE_COUNT) + return op_types[op_type]; + + rte_bbdev_log(ERR, "Invalid operation type"); + return ""; +} + + +int bbdev_logtype; + +RTE_INIT(rte_bbdev_init_log); +static void +rte_bbdev_init_log(void) +{ + bbdev_logtype = rte_log_register("lib.bbdev"); + if (bbdev_logtype >= 0) + rte_log_set_level(bbdev_logtype, RTE_LOG_NOTICE); +} diff --git a/lib/librte_bbdev/rte_bbdev.h b/lib/librte_bbdev/rte_bbdev.h new file mode 100644 index 0000000..9e6d283 --- /dev/null +++ b/lib/librte_bbdev/rte_bbdev.h @@ -0,0 +1,741 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef _RTE_BBDEV_H_ +#define _RTE_BBDEV_H_ + +/** + * @file rte_bbdev.h + * + * Wireless base band device application APIs. + * + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * This API allows an application to discover, configure and use a device to + * process operations. An asynchronous API (enqueue, followed by later dequeue) + * is used for processing operations. + * + * The functions in this API are not thread-safe when called on the same + * target object (a device, or a queue on a device), with the exception that + * one thread can enqueue operations to a queue while another thread dequeues + * from the same queue. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> +#include <stdbool.h> +#include <string.h> + +#include <rte_bus.h> +#include <rte_cpuflags.h> +#include <rte_memory.h> + +#include "rte_bbdev_op.h" + +#ifndef RTE_BBDEV_MAX_DEVS +#define RTE_BBDEV_MAX_DEVS 128 /**< Max number of devices */ +#endif + +/** Flags indiciate current state of BBDEV device */ +enum rte_bbdev_state { + RTE_BBDEV_UNUSED, + RTE_BBDEV_INITALIZED +}; + +/** + * Get the total number of devices that have been successfully initialised. + * + * @return + * The total number of usable devices. + */ +uint16_t +rte_bbdev_count(void); + +/** + * Check if a device is valid. + * + * @param dev_id + * The identifier of the device. + * + * @return + * true if device ID is valid and device is attached, false otherwise. + */ +bool +rte_bbdev_is_valid(uint16_t dev_id); + +/** + * Get the next enabled device. + * + * @param dev_id + * The current device + * + * @return + * - The next device, or + * - RTE_BBDEV_MAX_DEVS if none found + */ +uint16_t +rte_bbdev_find_next(uint16_t dev_id); + +/** Iterate through all enabled devices */ +#define RTE_BBDEV_FOREACH(i) for (i = rte_bbdev_find_next(-1); \ + i < RTE_BBDEV_MAX_DEVS; \ + i = rte_bbdev_find_next(i)) + +/** + * Setup up device queues. + * This function must be called on a device before setting up the queues and + * starting the device. It can also be called when a device is in the stopped + * state. If any device queues have been configured their configuration will be + * cleared by a call to this function. + * + * @param dev_id + * The identifier of the device. + * @param num_queues + * Number of queues to configure on device. + * @param socket_id + * ID of a socket which will be used to allocate memory. + * + * @return + * - 0 on success + * - -ENODEV if dev_id is invalid or the device is corrupted + * - -EINVAL if num_queues is invalid, 0 or greater than maximum + * - -EBUSY if the identified device has already started + * - -ENOMEM if unable to allocate memory + */ +int +rte_bbdev_setup_queues(uint16_t dev_id, uint16_t num_queues, int socket_id); + +/** + * Enable interrupts. + * This function may be called before starting the device to enable the + * interrupts if they are available. + * + * @param dev_id + * The identifier of the device. + * + * @return + * - 0 on success + * - -ENODEV if dev_id is invalid or the device is corrupted + * - -EBUSY if the identified device has already started + * - -ENOTSUP if the interrupts are not supported by the device + */ +int +rte_bbdev_intr_enable(uint16_t dev_id); + +/** Device queue configuration structure */ +struct rte_bbdev_queue_conf { + int socket; /**< NUMA socket used for memory allocation */ + uint32_t queue_size; /**< Size of queue */ + uint8_t priority; /**< Queue priority */ + bool deferred_start; /**< Do not start queue when device is started. */ + enum rte_bbdev_op_type op_type; /**< Operation type */ +}; + +/** + * Configure a queue on a device. + * This function can be called after device configuration, and before starting. + * It can also be called when the device or the queue is in the stopped state. + * + * @param dev_id + * The identifier of the device. + * @param queue_id + * The index of the queue. + * @param conf + * The queue configuration. If NULL, a default configuration will be used. + * + * @return + * - 0 on success + * - EINVAL if the identified queue size or priority are invalid + * - EBUSY if the identified queue or its device have already started + */ +int +rte_bbdev_queue_configure(uint16_t dev_id, uint16_t queue_id, + const struct rte_bbdev_queue_conf *conf); + +/** + * Start a device. + * This is the last step needed before enqueueing operations is possible. + * + * @param dev_id + * The identifier of the device. + * + * @return + * - 0 on success + * - negative value on failure - as returned from PMD driver + */ +int +rte_bbdev_start(uint16_t dev_id); + +/** + * Stop a device. + * The device can be reconfigured, and restarted after being stopped. + * + * @param dev_id + * The identifier of the device. + * + * @return + * - 0 on success + */ +int +rte_bbdev_stop(uint16_t dev_id); + +/** + * Close a device. + * The device cannot be restarted without reconfiguration! + * + * @param dev_id + * The identifier of the device. + * + * @return + * - 0 on success + */ +int +rte_bbdev_close(uint16_t dev_id); + +/** + * Start a specified queue on a device. + * This is only needed if the queue has been stopped, or if the deferred_start + * flag has been set when configuring the queue. + * + * @param dev_id + * The identifier of the device. + * @param queue_id + * The index of the queue. + * + * @return + * - 0 on success + * - negative value on failure - as returned from PMD driver + */ +int +rte_bbdev_queue_start(uint16_t dev_id, uint16_t queue_id); + +/** + * Stop a specified queue on a device, to allow re configuration. + * + * @param dev_id + * The identifier of the device. + * @param queue_id + * The index of the queue. + * + * @return + * - 0 on success + * - negative value on failure - as returned from PMD driver + */ +int +rte_bbdev_queue_stop(uint16_t dev_id, uint16_t queue_id); + +/** Device statistics. */ +struct rte_bbdev_stats { + uint64_t enqueued_count; /**< Count of all operations enqueued */ + uint64_t dequeued_count; /**< Count of all operations dequeued */ + /** Total error count on operations enqueued */ + uint64_t enqueue_err_count; + /** Total error count on operations dequeued */ + uint64_t dequeue_err_count; +}; + +/** + * Retrieve the general I/O statistics of a device. + * + * @param dev_id + * The identifier of the device. + * @param stats + * Pointer to structure to where statistics will be copied. On error, this + * location may or may not have been modified. + * + * @return + * - 0 on success + * - EINVAL if invalid parameter pointer is provided + */ +int +rte_bbdev_stats_get(uint16_t dev_id, struct rte_bbdev_stats *stats); + +/** + * Reset the statistics of a device. + * + * @param dev_id + * The identifier of the device. + * @return + * - 0 on success + */ +int +rte_bbdev_stats_reset(uint16_t dev_id); + +/** Device information supplied by the device's driver */ +struct rte_bbdev_driver_info { + /** Driver name */ + const char *driver_name; + + /** Maximum number of queues supported by the device */ + unsigned int max_num_queues; + /** Queue size limit (queue size must also be power of 2) */ + uint32_t queue_size_lim; + /** Set if device off-loads operation to hardware */ + bool hardware_accelerated; + /** Max value supported by queue priority */ + uint8_t max_queue_priority; + /** Set if device supports per-queue interrupts */ + bool queue_intr_supported; + /** Minimum alignment of buffers, in bytes */ + uint16_t min_alignment; + /** Default queue configuration used if none is supplied */ + struct rte_bbdev_queue_conf default_queue_conf; + /** Device operation capabilities */ + const struct rte_bbdev_op_cap *capabilities; + /** Device cpu_flag requirements */ + const enum rte_cpu_flag_t *cpu_flag_reqs; +}; + +/** Macro used at end of bbdev PMD list */ +#define RTE_BBDEV_END_OF_CAPABILITIES_LIST() \ + { RTE_BBDEV_OP_NONE } + +/** Device information structure used by an application to discover a devices + * capabilities and current configuration + */ +struct rte_bbdev_info { + int socket_id; /**< NUMA socket that device is on */ + const char *dev_name; /**< Unique device name */ + const struct rte_bus *bus; /**< Bus information */ + uint16_t num_queues; /**< Number of queues currently configured */ + bool started; /**< Set if device is currently started */ + struct rte_bbdev_driver_info drv; /**< Info from device driver */ +}; + +/** + * Retrieve information about a device. + * + * @param dev_id + * The identifier of the device. + * @param dev_info + * Pointer to structure to where information will be copied. On error, this + * location may or may not have been modified. + * + * @return + * - 0 on success + * - EINVAL if invalid parameter pointer is provided + */ +int +rte_bbdev_info_get(uint16_t dev_id, struct rte_bbdev_info *dev_info); + +/** Queue information */ +struct rte_bbdev_queue_info { + /** Current device configuration */ + struct rte_bbdev_queue_conf conf; + /** Set if queue is currently started */ + bool started; +}; + +/** + * Retrieve information about a specific queue on a device. + * + * @param dev_id + * The identifier of the device. + * @param queue_id + * The index of the queue. + * @param dev_info + * Pointer to structure to where information will be copied. On error, this + * location may or may not have been modified. + * + * @return + * - 0 on success + * - EINVAL if invalid parameter pointer is provided + */ +int +rte_bbdev_queue_info_get(uint16_t dev_id, uint16_t queue_id, + struct rte_bbdev_queue_info *dev_info); + +/** @internal The data structure associated with each queue of a device. */ +struct rte_bbdev_queue_data { + void *queue_private; /**< Driver-specific per-queue data */ + struct rte_bbdev_queue_conf conf; /**< Current configuration */ + struct rte_bbdev_stats queue_stats; /**< Queue statistics */ + bool started; /**< Queue state */ +}; + +/** @internal Enqueue encode operations for processing on queue of a device. */ +typedef uint16_t (*rte_bbdev_enqueue_enc_ops_t)( + struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, + uint16_t num); + +/** @internal Enqueue decode operations for processing on queue of a device. */ +typedef uint16_t (*rte_bbdev_enqueue_dec_ops_t)( + struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, + uint16_t num); + +/** @internal Dequeue encode operations from a queue of a device. */ +typedef uint16_t (*rte_bbdev_dequeue_enc_ops_t)( + struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num); + +/** @internal Dequeue decode operations from a queue of a device. */ +typedef uint16_t (*rte_bbdev_dequeue_dec_ops_t)( + struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num); + +#ifndef RTE_BBDEV_NAME_MAX_LEN +#define RTE_BBDEV_NAME_MAX_LEN 64 /**< Max length of device name */ +#endif + +/** + * @internal The data associated with a device, with no function pointers. + * This structure is safe to place in shared memory to be common among + * different processes in a multi-process configuration. Drivers can access + * these fields, but should never write to them! + */ +struct rte_bbdev_data { + char name[RTE_BBDEV_NAME_MAX_LEN]; /**< Unique identifier name */ + void *dev_private; /**< Driver-specific private data */ + uint16_t num_queues; /**< Number of currently configured queues */ + struct rte_bbdev_queue_data *queues; /**< Queue structures */ + uint16_t dev_id; /**< Device ID */ + int socket_id; /**< NUMA socket that device is on */ + bool started; /**< Device run-time state */ +}; + +/* Forward declarations */ +struct rte_bbdev_ops; +struct rte_bbdev_callback; +struct rte_intr_handle; + +/** Structure to keep track of registered callbacks */ +TAILQ_HEAD(rte_bbdev_cb_list, rte_bbdev_callback); + +/** + * @internal The data structure associated with a device. Drivers can access + * these fields, but should only write to the *_ops fields. + */ +struct __rte_cache_aligned rte_bbdev { + /**< Enqueue encode function */ + rte_bbdev_enqueue_enc_ops_t enqueue_enc_ops; + /**< Enqueue decode function */ + rte_bbdev_enqueue_dec_ops_t enqueue_dec_ops; + /**< Dequeue encode function */ + rte_bbdev_dequeue_enc_ops_t dequeue_enc_ops; + /**< Dequeue decode function */ + rte_bbdev_dequeue_dec_ops_t dequeue_dec_ops; + const struct rte_bbdev_ops *dev_ops; /**< Functions exported by PMD */ + struct rte_bbdev_data *data; /**< Pointer to device data */ + enum rte_bbdev_state state; /**< If device is currently used or not */ + struct rte_device *device; /**< Backing device */ + /** User application callback for interrupts if present */ + struct rte_bbdev_cb_list list_cbs; + struct rte_intr_handle *intr_handle; /**< Device interrupt handle */ +}; + +/** @internal array of all devices */ +extern struct rte_bbdev rte_bbdev_devices[]; + +/** + * Enqueue a burst of processed encode operations to a queue of the device. + * This functions only enqueues as many operations as currently possible and + * does not block until @p num_ops entries in the queue are available. + * This function does not provide any error notification to avoid the + * corresponding overhead. + * + * @param dev_id + * The identifier of the device. + * @param queue_id + * The index of the queue. + * @param ops + * Pointer array containing operations to be enqueued Must have at least + * @p num_ops entries + * @param num_ops + * The maximum number of operations to enqueue. + * + * @return + * The number of operations actually enqueued (this is the number of processed + * entries in the @p ops array). + */ +static inline uint16_t +rte_bbdev_enqueue_enc_ops(uint16_t dev_id, uint16_t queue_id, + struct rte_bbdev_enc_op **ops, uint16_t num_ops) +{ + struct rte_bbdev *dev = &rte_bbdev_devices[dev_id]; + struct rte_bbdev_queue_data *q_data = &dev->data->queues[queue_id]; + uint16_t n = dev->enqueue_enc_ops(q_data, ops, num_ops); + + rte_bbdev_log_verbose("%u encode ops enqueued to dev%u,q%u.\n", + num_ops, dev_id, queue_id); + + return n; +} + +/** + * Enqueue a burst of processed decode operations to a queue of the device. + * This functions only enqueues as many operations as currently possible and + * does not block until @p num_ops entries in the queue are available. + * This function does not provide any error notification to avoid the + * corresponding overhead. + * + * @param dev_id + * The identifier of the device. + * @param queue_id + * The index of the queue. + * @param ops + * Pointer array containing operations to be enqueued Must have at least + * @p num_ops entries + * @param num_ops + * The maximum number of operations to enqueue. + * + * @return + * The number of operations actually enqueued (this is the number of processed + * entries in the @p ops array). + */ +static inline uint16_t +rte_bbdev_enqueue_dec_ops(uint16_t dev_id, uint16_t queue_id, + struct rte_bbdev_dec_op **ops, uint16_t num_ops) +{ + struct rte_bbdev *dev = &rte_bbdev_devices[dev_id]; + struct rte_bbdev_queue_data *q_data = &dev->data->queues[queue_id]; + uint16_t n = dev->enqueue_dec_ops(q_data, ops, num_ops); + + rte_bbdev_log_verbose("%u decode ops enqueued to dev%u,q%u.\n", + num_ops, dev_id, queue_id); + + return n; +} + +/** + * Dequeue a burst of processed encode operations from a queue of the device. + * This functions returns only the current contents of the queue, and does not + * block until @ num_ops is available. + * This function does not provide any error notification to avoid the + * corresponding overhead. + * + * @param dev_id + * The identifier of the device. + * @param queue_id + * The index of the queue. + * @param ops + * Pointer array where operations will be dequeued to. Must have at least + * @p num_ops entries + * @param num_ops + * The maximum number of operations to dequeue. + * + * @return + * The number of operations actually dequeued (this is the number of entries + * copied into the @p ops array). + */ +static inline uint16_t +rte_bbdev_dequeue_enc_ops(uint16_t dev_id, uint16_t queue_id, + struct rte_bbdev_enc_op **ops, uint16_t num_ops) +{ + struct rte_bbdev *dev = &rte_bbdev_devices[dev_id]; + struct rte_bbdev_queue_data *q_data = &dev->data->queues[queue_id]; + uint16_t n = dev->dequeue_enc_ops(q_data, ops, num_ops); + + rte_bbdev_log_verbose("%u encode ops dequeued to dev%u,q%u\n", + n, dev_id, queue_id); + + return n; +} + +/** + * Dequeue a burst of processed decode operations from a queue of the device. + * This functions returns only the current contents of the queue, and does not + * block until @ num_ops is available. + * This function does not provide any error notification to avoid the + * corresponding overhead. + * + * @param dev_id + * The identifier of the device. + * @param queue_id + * The index of the queue. + * @param ops + * Pointer array where operations will be dequeued to. Must have at least + * @p num_ops entries + * @param num_ops + * The maximum number of operations to dequeue. + * + * @return + * The number of operations actually dequeued (this is the number of entries + * copied into the @p ops array). + */ + +static inline uint16_t +rte_bbdev_dequeue_dec_ops(uint16_t dev_id, uint16_t queue_id, + struct rte_bbdev_dec_op **ops, uint16_t num_ops) +{ + struct rte_bbdev *dev = &rte_bbdev_devices[dev_id]; + struct rte_bbdev_queue_data *q_data = &dev->data->queues[queue_id]; + uint16_t n = dev->dequeue_dec_ops(q_data, ops, num_ops); + + rte_bbdev_log_verbose("%u decode ops dequeued to dev%u,q%u\n", + n, dev_id, queue_id); + + return n; +} + +/** Definitions of device event types */ +enum rte_bbdev_event_type { + RTE_BBDEV_EVENT_UNKNOWN, /**< unknown event type */ + RTE_BBDEV_EVENT_ERROR, /**< error interrupt event */ + RTE_BBDEV_EVENT_DEQUEUE, /**< dequeue event */ + RTE_BBDEV_EVENT_MAX /**< max value of this enum */ +}; + +/** + * Typedef for application callback function registered by application + * software for notification of device events + * + * @param dev_id + * Device identifier + * @param event + * Device event to register for notification of. + * @param cb_arg + * User specified parameter to be passed to user's callback function. + * @param ret_param + * To pass data back to user application. + */ +typedef void (*rte_bbdev_cb_fn)(uint16_t dev_id, + enum rte_bbdev_event_type event, void *cb_arg, + void *ret_param); + +/** + * Register a callback function for specific device id. Multiple callbacks can + * be added and will be called in the order they are added when an event is + * triggered. Callbacks are called in a separate thread created by the DPDK EAL. + * + * @param dev_id + * Device id. + * @param event + * The event that the callback will be registered for. + * @param cb_fn + * User supplied callback function to be called. + * @param cb_arg + * Pointer to parameter that will be passed to the callback. + * + * @return + * Zero on success, negative value on failure. + */ +int +rte_bbdev_callback_register(uint16_t dev_id, enum rte_bbdev_event_type event, + rte_bbdev_cb_fn cb_fn, void *cb_arg); + +/** + * Unregister a callback function for specific device id. + * + * @param dev_id + * The device identifier. + * @param event + * The event that the callback will be unregistered for. + * @param cb_fn + * User supplied callback function to be unregistered. + * @param cb_arg + * Pointer to the parameter supplied when registering the callback. + * (void *)-1 means to remove all registered callbacks with the specified + * function address. + * + * @return + * - 0 on success + * - EINVAL if invalid parameter pointer is provided + * - EAGAIN if the provided callback pointer does not exist + */ +int +rte_bbdev_callback_unregister(uint16_t dev_id, enum rte_bbdev_event_type event, + rte_bbdev_cb_fn cb_fn, void *cb_arg); + +/** + * Enable a one-shot interrupt on the next operation enqueued to a particular + * queue. The interrupt will be triggered when the operation is ready to be + * dequeued. To handle the interrupt, an epoll file descriptor must be + * registered using rte_bbdev_queue_intr_ctl(), and then an application + * thread/lcore can wait for the interrupt using rte_epoll_wait(). + * + * @param dev_id + * The device identifier. + * @param queue_id + * The index of the queue. + * + * @return + * - 0 on success + * - negative value on failure - as returned from PMD driver + */ +int +rte_bbdev_queue_intr_enable(uint16_t dev_id, uint16_t queue_id); + +/** + * Disable a one-shot interrupt on the next operation enqueued to a particular + * queue (if it has been enabled). + * + * @param dev_id + * The device identifier. + * @param queue_id + * The index of the queue. + * + * @return + * - 0 on success + * - negative value on failure - as returned from PMD driver + */ +int +rte_bbdev_queue_intr_disable(uint16_t dev_id, uint16_t queue_id); + +/** + * Control interface for per-queue interrupts. + * + * @param dev_id + * The device identifier. + * @param queue_id + * The index of the queue. + * @param epfd + * Epoll file descriptor that will be associated with the interrupt source. + * If the special value RTE_EPOLL_PER_THREAD is provided, a per thread epoll + * file descriptor created by the EAL is used (RTE_EPOLL_PER_THREAD can also + * be used when calling rte_epoll_wait()). + * @param op + * The operation be performed for the vector.RTE_INTR_EVENT_ADD or + * RTE_INTR_EVENT_DEL. + * @param data + * User context, that will be returned in the epdata.data field of the + * rte_epoll_event structure filled in by rte_epoll_wait(). + * + * @return + * - 0 on success + * - ENOTSUP if interrupts are not supported by the identified device + * - negative value on failure - as returned from PMD driver + */ +int +rte_bbdev_queue_intr_ctl(uint16_t dev_id, uint16_t queue_id, int epfd, int op, + void *data); + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_BBDEV_H_ */ diff --git a/lib/librte_bbdev/rte_bbdev_op.h b/lib/librte_bbdev/rte_bbdev_op.h new file mode 100644 index 0000000..99ad899 --- /dev/null +++ b/lib/librte_bbdev/rte_bbdev_op.h @@ -0,0 +1,514 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef _RTE_BBDEV_OP_H_ +#define _RTE_BBDEV_OP_H_ + +/** + * @file rte_bbdev_op.h + * + * Defines wireless base band layer 1 operations and capabilities + * + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> + +#include <rte_common.h> +#include <rte_mbuf.h> +#include <rte_memory.h> +#include <rte_mempool.h> + +#define RTE_BBDEV_MAX_CODE_BLOCKS 64 + +extern int bbdev_logtype; + +/** + * Helper macro for logging + * + * @param level + * Log level: EMERG, ALERT, CRIT, ERR, WARNING, NOTICE, INFO, or DEBUG + * @param fmt + * The format string, as in printf(3). + * @param ... + * The variable arguments required by the format string. + * + * @return + * - 0 on success + * - Negative on error + */ +#define rte_bbdev_log(level, fmt, ...) \ + rte_log(RTE_LOG_ ## level, bbdev_logtype, fmt "\n", ##__VA_ARGS__) + +/** + * Helper macro for debug logging with extra source info + * + * @param fmt + * The format string, as in printf(3). + * @param ... + * The variable arguments required by the format string. + * + * @return + * - 0 on success + * - Negative on error + */ +#define rte_bbdev_log_debug(fmt, ...) \ + rte_bbdev_log(DEBUG, RTE_STR(__LINE__) ":%s() " fmt, __func__, \ + ##__VA_ARGS__) + +/** + * Helper macro for extra conditional logging from datapath + * + * @param fmt + * The format string, as in printf(3). + * @param ... + * The variable arguments required by the format string. + * + * @return + * - 0 on success + * - Negative on error + */ +#define rte_bbdev_log_verbose(fmt, ...) \ + (void)((RTE_LOG_DEBUG <= RTE_LOG_DP_LEVEL) ? \ + rte_log(RTE_LOG_DEBUG, \ + bbdev_logtype, ": " fmt "\n", ##__VA_ARGS__) : 0) + +/** Flags for turbo decoder operation and capability structure */ +enum rte_bbdev_op_td_flag_bitmasks { + /** If sub block de-interleaving is to be performed. */ + RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE = (1ULL << 0), + /** To use CRC Type 24B (otherwise use CRC Type 24A). */ + RTE_BBDEV_TURBO_CRC_TYPE_24B = (1ULL << 1), + /** If turbo equalization is to be performed. */ + RTE_BBDEV_TURBO_EQUALIZER = (1ULL << 2), + /** If set, saturate soft output to +/-127 */ + RTE_BBDEV_TURBO_SOFT_OUT_SATURATE = (1ULL << 3), + /** + * Set to 1 to start iteration from even, else odd; one iteration = + * max_iteration + 0.5 + */ + RTE_BBDEV_TURBO_HALF_ITERATION_EVEN = (1ULL << 4), + /** + * If 0, TD stops after CRC matches; else if 1, runs to end of next + * odd iteration after CRC matches + */ + RTE_BBDEV_TURBO_CONTINUE_CRC_MATCH = (1ULL << 5), + /** Set if soft output is required to be output */ + RTE_BBDEV_TURBO_SOFT_OUTPUT = (1ULL << 6), + /** Set to enable early termination mode */ + RTE_BBDEV_TURBO_EARLY_TERMINATION = (1ULL << 7), + /** + * Set if the input is a raw data (E bytes, no NULL bytes). If not set + * the input is a full circular buffer with data (Kw bytes) as decribed + * in spec. 36.212, chapter 5.1.4.1.2. + */ + RTE_BBDEV_TURBO_RAW_INPUT_DATA = (1ULL << 8), + /** Set if a device supports decoder dequeue interrupts */ + RTE_BBDEV_TURBO_DEC_INTERRUPTS = (1ULL << 9), +}; + +/** Flags for turbo encoder operation and capability structure */ +enum rte_bbdev_op_te_flag_bitmasks { + /** Ignore rv_index and set K0 = 0 */ + RTE_BBDEV_TURBO_RV_INDEX_BYPASS = (1ULL << 0), + /** If rate matching is to be performed */ + RTE_BBDEV_TURBO_RATE_MATCH = (1ULL << 1), + /** This bit must be set to enable CRC-24B generation */ + RTE_BBDEV_TURBO_CRC_24B_ATTACH = (1ULL << 2), + /** This bit must be set to enable CRC-24A generation */ + RTE_BBDEV_TURBO_CRC_24A_ATTACH = (1ULL << 3), + /** Set if a device supports encoder dequeue interrupts */ + RTE_BBDEV_TURBO_ENC_INTERRUPTS = (1ULL << 4) +}; + +/** Data input and output buffer for BBDEV operations */ +struct rte_bbdev_op_data { + struct rte_mbuf *data; + /**< First mbuf segment with input/output data. Each segment represents + * one Code Block. For the input operation a mbuf needs to contain + * all Code Blocks. For the output operation the mbuf should consist of + * only one segment and the driver will take care of allocating and + * chaining another segments for the consecutive Code Blocks if needed. + */ + uint32_t offset; + /**< The starting point for the Turbo input/output, in bytes, from the + * start of the first segment's data buffer. It must be smaller than the + * first segment's data_len! + */ + uint32_t length; + /**< Length of Transport Block - number of bytes for Turbo Encode/Decode + * operation for input; length of the output for output operation. + */ +}; + +struct rte_bbdev_op_dec_cb_params { + uint16_t k; /**< size of the input code block in bits (40 - 6144) */ + uint32_t e; /**< length in bits of the rate match output (17 bits) */ +}; + +struct rte_bbdev_op_dec_tb_params { + /**< size of the input code block in bits (40 - 6144). Used when + * code block index r < c_neg + */ + uint16_t k_neg; + /**< size of the input code block in bits (40 - 6144). Used when + * code block index r >= c_neg + */ + uint16_t k_pos; + uint8_t c_neg; /**< number of code block using k_neg (0 - 63) */ + uint8_t c; /**< total number of code blocks (1 - 64) */ + uint8_t cab; /**< number of code blocks using ea before switch to eb */ + /**< length in bits of the rate match output (17 bits). Used when + * code block index r < cab + */ + uint32_t ea; + /**< length in bits of the rate match output (17 bits). Used when + * code block index r >= cab + */ + uint32_t eb; + uint8_t cb_idx; /**< Code block index */ +}; + +/** Operation structure for the Turbo Decoder */ +struct rte_bbdev_op_turbo_dec { + struct rte_bbdev_op_data input; /**< input src data */ + struct rte_bbdev_op_data hard_output; /**< hard output buffer */ + struct rte_bbdev_op_data soft_output; /**< soft output buffer */ + + uint32_t op_flags; /**< Flags from rte_bbdev_op_td_flag_bitmasks */ + uint8_t rv_index; /**< Rv index for rate matching (0 - 3) */ + uint8_t iter_min:4; /**< min number of iterations */ + uint8_t iter_max:4; /**< max number of iterations */ + uint8_t iter_count; /**< Actual num. of iterations performed */ + /** 5 bit extrinsic scale (scale factor on extrinsic info) */ + uint8_t ext_scale; + /** Number of MAP engines, must be power of 2 (or 0 to auto-select) */ + uint8_t num_maps; + uint8_t code_block_mode; /**< 0 - transpot block, 1 - code block */ + union { + /** Struct which stores Code Block specific parameters */ + struct rte_bbdev_op_dec_cb_params cb_params; + /** Struct which stores Transport Block specific parameters */ + struct rte_bbdev_op_dec_tb_params tb_params; + }; +}; + +struct rte_bbdev_op_enc_cb_params { + uint16_t k; /**< size of the input code block in bits (40 - 6144) */ + uint32_t e; /**< length in bits of the rate match output (17 bits) */ + uint16_t ncb; /**< Ncb parameter for rate matching, range [k:3(k+4)] */ +}; + +struct rte_bbdev_op_enc_tb_params { + /**< size of the input code block in bits (40 - 6144). Used when + * code block index r < c_neg + */ + uint16_t k_neg; + /**< size of the input code block in bits (40 - 6144). Used when + * code block index r >= c_neg + */ + uint16_t k_pos; + uint8_t c_neg; /**< number of code block using k_neg (0 - 63) */ + uint8_t c; /**< total number of code blocks (1 - 64) */ + uint8_t cab; /**< number of code blocks using ea before switch to eb */ + /**< length in bits of the rate match output (17 bits). Used when + * code block index r < cab + */ + uint32_t ea; + /**< length in bits of the rate match output (17 bits). Used when + * code block index r >= cab + */ + uint32_t eb; + /**< Ncb parameter for rate matching, range [k : 3(k+4)]. Used when + * code block index r < c_neg + */ + uint16_t ncb_neg; + /**< Ncb parameter for rate matching, range [k : 3(k+4)]. Used when + * code block index r >= c_neg + */ + uint16_t ncb_pos; + uint8_t cb_idx; /**< Code block index */ +}; + +/** Operation structure for the Turbo Encoder */ +struct rte_bbdev_op_turbo_enc { + struct rte_bbdev_op_data input; /**< input src data */ + struct rte_bbdev_op_data output; /**< output buffer */ + + uint32_t op_flags; /**< Flags from rte_bbdev_op_te_flag_bitmasks */ + int32_t n_soft; /**< total number of soft bits according to UE cat. */ + int32_t k_mimo; /**< MIMO type */ + int32_t mdl_harq; /**< the maximum number of DL HARQ processes */ + /**< total number of bits available for transmission of one TB */ + int32_t g; + int32_t nl; /**< number of layer */ + int32_t qm; /**< modulation type */ + uint8_t rv_index; /**< Rv index for rate matching (0 - 3) */ + uint8_t code_block_mode; /**< 0 - transpot block, 1 - code block */ + union { + /** Struct which stores Code Block specific parameters */ + struct rte_bbdev_op_enc_cb_params cb_params; + /** Struct which stores Transport Block specific parameters */ + struct rte_bbdev_op_enc_tb_params tb_params; + }; +}; + +/** List of the capabilities for the Turbo Decoder */ +struct rte_bbdev_op_cap_turbo_dec { + /** Flags from rte_bbdev_op_td_flag_bitmasks */ + uint32_t capability_flags; + uint8_t num_buffers_src; /**< Num input code block buffers */ + /** Num hard output code block buffers */ + uint8_t num_buffers_hard_out; + /** Num soft output code block buffers if supported by the driver */ + uint8_t num_buffers_soft_out; +}; + +/** List of the capabilities for the Turbo Encoder */ +struct rte_bbdev_op_cap_turbo_enc { + /** Flags from rte_bbdev_op_te_flag_bitmasks */ + uint32_t capability_flags; + uint8_t num_buffers_src; /**< Num input code block buffers */ + uint8_t num_buffers_dst; /**< Num output code block buffers */ +}; + +/** Different operation types supported by the device */ +enum rte_bbdev_op_type { + RTE_BBDEV_OP_NONE, /**< Dummy operation that does nothing */ + RTE_BBDEV_OP_TURBO_DEC, /**< Turbo decode */ + RTE_BBDEV_OP_TURBO_ENC, /**< Turbo encode */ + RTE_BBDEV_OP_TYPE_COUNT, /**< Count of different op types */ +}; + +/** Bit indexes of possible errors reported through status field */ +enum { + RTE_BBDEV_DRV_ERROR, + RTE_BBDEV_DATA_ERROR, + RTE_BBDEV_CRC_ERROR, +}; + +/** Structure specifying a single encode operation */ +struct rte_bbdev_enc_op { + int status; /**< Status of operation that was performed */ + struct rte_mempool *mempool; /**< Mempool which op instance is in */ + void *opaque_data; /**< Opaque pointer for user data */ + /** Contains encoder specific parameters */ + struct rte_bbdev_op_turbo_enc turbo_enc; +}; + +/** Structure specifying a single decode operation */ +struct rte_bbdev_dec_op { + int status; /**< Status of operation that was performed */ + struct rte_mempool *mempool; /**< Mempool which op instance is in */ + void *opaque_data; /**< Opaque pointer for user data */ + /** Contains decoder specific parameters */ + struct rte_bbdev_op_turbo_dec turbo_dec; +}; + +/** Operation capabilities supported by a device */ +struct rte_bbdev_op_cap { + enum rte_bbdev_op_type type; /**< Type of operation */ + union { + struct rte_bbdev_op_cap_turbo_dec turbo_dec; + struct rte_bbdev_op_cap_turbo_enc turbo_enc; + } cap; /**< Operation-type specific capabilities */ +}; + +/** @internal Private data structure stored with operation pool. */ +struct rte_bbdev_op_pool_private { + enum rte_bbdev_op_type type; /**< Type of operations in a pool */ +}; + +/** + * Converts queue operation type from enum to string + * + * @param op_type + * Operation type as enum + * + * @returns + * Operation type as string + * + */ +const char* +rte_bbdev_op_type_str(enum rte_bbdev_op_type op_type); + +/** + * Creates a bbdev operation mempool + * + * @param name + * Pool name. + * @param type + * Operation type, use RTE_BBDEV_OP_NONE for a pool which supports all + * operation types. + * @param num_elements + * Number of elements in the pool. + * @param cache_size + * Number of elements to cache on an lcore, see rte_mempool_create() for + * further details about cache size. + * @param socket_id + * Socket to allocate memory on. + * + * @return + * - Pointer to a mempool on success, + * - NULL pointer on failure. + */ +struct rte_mempool * +rte_bbdev_op_pool_create(const char *name, enum rte_bbdev_op_type type, + unsigned int num_elements, unsigned int cache_size, + int socket_id); + +/** + * Bulk allocate encode operations from a mempool with parameter defaults reset. + * + * @param mempool + * Operation mempool, created by rte_bbdev_op_pool_create(). + * @param ops + * Output array to place allocated operations + * @param num_ops + * Number of operations to allocate + * + * @returns + * - 0 on success + * - EINVAL if invalid mempool is provided + */ +static inline int +rte_bbdev_enc_op_alloc_bulk(struct rte_mempool *mempool, + struct rte_bbdev_enc_op **ops, uint16_t num_ops) +{ + struct rte_bbdev_op_pool_private *priv; + int ret; + + /* Check type */ + priv = (struct rte_bbdev_op_pool_private *) + rte_mempool_get_priv(mempool); + if (unlikely(priv->type != RTE_BBDEV_OP_TURBO_ENC)) + return -EINVAL; + + /* Get elements */ + ret = rte_mempool_get_bulk(mempool, (void **)ops, num_ops); + if (unlikely(ret < 0)) + return ret; + + rte_bbdev_log_verbose("%u encode ops allocated from %s\n", + num_ops, mempool->name); + + return 0; +} + +/** + * Bulk allocate decode operations from a mempool with parameter defaults reset. + * + * @param mempool + * Operation mempool, created by rte_bbdev_op_pool_create(). + * @param ops + * Output array to place allocated operations + * @param num_ops + * Number of operations to allocate + * + * @returns + * - 0 on success + * - EINVAL if invalid mempool is provided + */ +static inline int +rte_bbdev_dec_op_alloc_bulk(struct rte_mempool *mempool, + struct rte_bbdev_dec_op **ops, uint16_t num_ops) +{ + struct rte_bbdev_op_pool_private *priv; + int ret; + + /* Check type */ + priv = (struct rte_bbdev_op_pool_private *) + rte_mempool_get_priv(mempool); + if (unlikely(priv->type != RTE_BBDEV_OP_TURBO_DEC)) + return -EINVAL; + + /* Get elements */ + ret = rte_mempool_get_bulk(mempool, (void **)ops, num_ops); + if (unlikely(ret < 0)) + return ret; + + rte_bbdev_log_verbose("%u encode ops allocated from %s\n", + num_ops, mempool->name); + + return 0; +} + +/** + * Free decode operation structures that were allocated by + * rte_bbdev_dec_op_alloc_bulk(). + * All structures must belong to the same mempool. + * + * @param ops + * Operation structures + * @param num_ops + * Number of structures + */ +static inline void +rte_bbdev_dec_op_free_bulk(struct rte_bbdev_dec_op **ops, unsigned int num_ops) +{ + if (num_ops > 0) { + rte_mempool_put_bulk(ops[0]->mempool, (void **)ops, num_ops); + rte_bbdev_log_verbose("%u decode ops freed to %s\n", num_ops, + ops[0]->mempool->name); + } +} + +/** + * Free encode operation structures that were allocated by + * rte_bbdev_enc_op_alloc_bulk(). + * All structures must belong to the same mempool. + * + * @param ops + * Operation structures + * @param num_ops + * Number of structures + */ +static inline void +rte_bbdev_enc_op_free_bulk(struct rte_bbdev_enc_op **ops, unsigned int num_ops) +{ + if (num_ops > 0) { + rte_mempool_put_bulk(ops[0]->mempool, (void **)ops, num_ops); + rte_bbdev_log_verbose("%u encode ops freed to %s\n", num_ops, + ops[0]->mempool->name); + } +} + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_BBDEV_OP_H_ */ diff --git a/lib/librte_bbdev/rte_bbdev_pci.h b/lib/librte_bbdev/rte_bbdev_pci.h new file mode 100644 index 0000000..1a32132 --- /dev/null +++ b/lib/librte_bbdev/rte_bbdev_pci.h @@ -0,0 +1,288 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef _RTE_BBDEV_PCI_H_ +#define _RTE_BBDEV_PCI_H_ + +/** + * @file rte_bbdev_pci.h + * + * Wireless base band PCI-driver-facing APIs. + * + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * This API provides the helper functions for device PCI drivers to register + * with the bbdev interface. User applications should not use this API. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> +#include <rte_log.h> +#include <rte_pci.h> + +#include "rte_bbdev_pmd.h" + +/** + * @internal + * Initialisation function of a HW driver invoked for each matching HW device + * detected during the EAL initialisation phase, or when a new device is + * attached. The driver should initialise the device and its own software + * context. + * + * @param dev + * This is a new device structure instance that is associated with the + * matching device. + * The driver *must* populate the following fields: + * - dev_ops + * - enqueue_ops + * - dequeue_ops + * + * @return + * - 0 on success + */ +typedef int (*rte_bbdev_init_t)(struct rte_bbdev *dev); + +/** + * @internal + * Finalization function of a HW driver invoked for each matching HW device + * detected during the closing phase, or when a device is detached. + * + * @param dev + * The device structure instance that is associated with the matching device. + * + * @return + * - 0 on success + */ +typedef int (*rte_bbdev_uninit_t)(struct rte_bbdev *dev); + +/** + * @internal + * Allocates a new bbdev slot for an PCI device and returns the pointer to that + * slot for the driver to use. + * + * @param dev + * Pointer to the PCI device + * + * @param private_data_size + * Size of private data structure + * + * @return + * A pointer to a rte_bbdev or NULL if allocation failed. + */ +static inline struct rte_bbdev * +rte_bbdev_pci_allocate(struct rte_pci_device *dev, size_t private_data_size) +{ + const char *name; + struct rte_bbdev *bbdev = NULL; + + if (dev == NULL) { + rte_bbdev_log(ERR, "NULL PCI device"); + return NULL; + } + + name = dev->device.name; + + /* Allocate memory to be used privately by drivers */ + if (rte_eal_process_type() == RTE_PROC_PRIMARY) { + bbdev = rte_bbdev_allocate(name); + if (!bbdev) + return NULL; + + if (private_data_size) { + bbdev->data->dev_private = rte_zmalloc_socket(name, + private_data_size, RTE_CACHE_LINE_SIZE, + dev->device.numa_node); + if (bbdev->data->dev_private == NULL) { + rte_bbdev_log(CRIT, + "Allocate of %zu bytes for device \"%s\"failed", + private_data_size, name); + rte_bbdev_release(bbdev); + return NULL; + } + } + } + + bbdev->data->socket_id = dev->device.numa_node; + return bbdev; +} + +static inline void +rte_bbdev_pci_release(struct rte_bbdev *bbdev) +{ + int ret; + uint16_t dev_id = bbdev->data->dev_id; + + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + rte_free(bbdev->data->dev_private); + + ret = rte_bbdev_release(bbdev); + if (ret) + rte_bbdev_log(ERR, "Device %i failed to uninit: %i", dev_id, + ret); + + rte_bbdev_log_debug("Un-initialised HW device id = %u", dev_id); +} + +/** + * @internal + * Wrapper for use by pci drivers as a .probe function to attach to a bbdev + * interface. + */ +static inline int +rte_bbdev_pci_generic_probe(struct rte_pci_device *pci_dev, + size_t private_data_size, + rte_bbdev_init_t dev_init) +{ + struct rte_bbdev *bbdev = NULL; + char dev_name[RTE_BBDEV_NAME_MAX_LEN]; + int ret; + + rte_pci_device_name(&pci_dev->addr, dev_name, sizeof(dev_name)); + + bbdev = rte_bbdev_pci_allocate(pci_dev, private_data_size); + if (bbdev == NULL) + return -ENOMEM; + + /* Fill HW specific part of device structure */ + bbdev->device = &pci_dev->device; + bbdev->intr_handle = &pci_dev->intr_handle; + + /* Invoke PMD device initialization function */ + if (dev_init) { + ret = dev_init(bbdev); + + if (bbdev->dev_ops == NULL) { + rte_bbdev_log(ERR, "NULL dev_ops structure in device %u", + bbdev->data->dev_id); + return -ENODEV; + } + + if (bbdev->enqueue_enc_ops == NULL) { + rte_bbdev_log(ERR, + "NULL enqueue_enc_ops structure in device %u", + bbdev->data->dev_id); + return -ENODEV; + } + if (bbdev->enqueue_dec_ops == NULL) { + rte_bbdev_log(ERR, + "NULL enqueue_dec_ops structure in device %u", + bbdev->data->dev_id); + return -ENODEV; + } + if (bbdev->dequeue_enc_ops == NULL) { + rte_bbdev_log(ERR, + "NULL dequeue_enc_ops structure in device %u", + bbdev->data->dev_id); + return -ENODEV; + } + if (bbdev->dequeue_dec_ops == NULL) { + rte_bbdev_log(ERR, + "NULL dequeue_dec_ops structure in device %u", + bbdev->data->dev_id); + return -ENODEV; + } + + if (ret < 0) { + rte_bbdev_log(ERR, + "Driver %s(vendor_id=0x%x device_id=0x%x): failed: %i", + pci_dev->driver->driver.name, + pci_dev->id.vendor_id, + pci_dev->id.device_id, ret); + rte_bbdev_pci_release(bbdev); + return -ENXIO; + } + } else { + rte_bbdev_log(ERR, + "Device init function doesn't exist for driver %s", + pci_dev->driver->driver.name); + return -ENODEV; + } + + rte_bbdev_log_debug("Initialised HW device %s (id = %u)", + dev_name, bbdev->data->dev_id); + return 0; +} + +/** + * @internal + * Wrapper for use by pci drivers as a .remove function to detach a bbdev + * interface. + */ +static inline int +rte_bbdev_pci_generic_remove(struct rte_pci_device *pci_dev, + rte_bbdev_uninit_t dev_uninit) +{ + struct rte_bbdev *bbdev; + int ret; + uint8_t dev_id; + + if (pci_dev == NULL) + return -EINVAL; + + /* Find device */ + bbdev = rte_bbdev_get_named_dev(pci_dev->device.name); + if (bbdev == NULL) { + rte_bbdev_log(CRIT, + "Couldn't find HW dev \"%s\" to uninitialise it", + pci_dev->device.name); + return -ENODEV; + } + dev_id = bbdev->data->dev_id; + + /* Close device before uninit */ + ret = rte_bbdev_close(dev_id); + if (ret < 0) + rte_bbdev_log(ERR, + "Device %i failed to close during uninit: %i", + dev_id, ret); + + /* Invoke PMD device uninit function */ + if (dev_uninit) { + ret = dev_uninit(bbdev); + if (ret) + rte_bbdev_log(ERR, "Device %i failed to uninit: %i", + dev_id, ret); + } + + rte_bbdev_pci_release(bbdev); + return 0; +} + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_BBDEV_PCI_H_ */ diff --git a/lib/librte_bbdev/rte_bbdev_pmd.h b/lib/librte_bbdev/rte_bbdev_pmd.h new file mode 100644 index 0000000..cf65de0 --- /dev/null +++ b/lib/librte_bbdev/rte_bbdev_pmd.h @@ -0,0 +1,223 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef _RTE_BBDEV_PMD_H_ +#define _RTE_BBDEV_PMD_H_ + +/** + * @file rte_bbdev_pmd.h + * + * Wireless base band driver-facing APIs. + * + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * This API provides the mechanism for device drivers to register with the + * bbdev interface. User applications should not use this API. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> +#include <rte_log.h> + +#include "rte_bbdev.h" + +/** Suggested value for SW based devices */ +#define RTE_BBDEV_DEFAULT_MAX_NB_QUEUES RTE_MAX_LCORE + +/** Suggested value for SW based devices */ +#define RTE_BBDEV_QUEUE_SIZE_LIMIT 16384 + +/** + * @internal + * Allocates a new slot for a bbdev and returns the pointer to that slot + * for the driver to use. + * + * @param name + * Unique identifier name for each bbdev device + * + * @return + * - Slot in the rte_bbdev array for a new device; + */ +struct rte_bbdev * +rte_bbdev_allocate(const char *name); + +/** + * @internal + * Release the specified bbdev. + * + * @param bbdev + * The *bbdev* pointer is the address of the *rte_bbdev* structure. + * @return + * - 0 on success, negative on error + */ +int +rte_bbdev_release(struct rte_bbdev *bbdev); + +/** + * Get the device structure for a named device. + * + * @param name + * Name of the device + * + * @return + * - The device structure pointer, or + * - NULL otherwise + * + */ +struct rte_bbdev * +rte_bbdev_get_named_dev(const char *name); + +/** + * Definitions of all functions exported by a driver through the the generic + * structure of type *rte_bbdev_ops* supplied in the *rte_bbdev* structure + * associated with a device. + */ + +/** @internal Function used to configure device memory. */ +typedef int (*rte_bbdev_setup_queues_t)(struct rte_bbdev *dev, + uint16_t num_queues, int socket_id); + +/** @internal Function used to configure interrupts for a device. */ +typedef int (*rte_bbdev_intr_enable_t)(struct rte_bbdev *dev); + +/** @internal Function to allocate and configure a device queue. */ +typedef int (*rte_bbdev_queue_setup_t)(struct rte_bbdev *dev, + uint16_t queue_id, const struct rte_bbdev_queue_conf *conf); + +/* @internal + * Function to release memory resources allocated for a device queue. + */ +typedef int (*rte_bbdev_queue_release_t)(struct rte_bbdev *dev, + uint16_t queue_id); + +/** @internal Function to start a configured device. */ +typedef int (*rte_bbdev_start_t)(struct rte_bbdev *dev); + +/** @internal Function to stop a device. */ +typedef void (*rte_bbdev_stop_t)(struct rte_bbdev *dev); + +/** @internal Function to close a device. */ +typedef int (*rte_bbdev_close_t)(struct rte_bbdev *dev); + +/** @internal Function to start a device queue. */ +typedef int (*rte_bbdev_queue_start_t)(struct rte_bbdev *dev, + uint16_t queue_id); + +/** @internal Function to stop a device queue. */ +typedef int (*rte_bbdev_queue_stop_t)(struct rte_bbdev *dev, uint16_t queue_id); + +/** @internal Function to read stats from a device. */ +typedef void (*rte_bbdev_stats_get_t)(struct rte_bbdev *dev, + struct rte_bbdev_stats *stats); + +/** @internal Function to reset stats on a device. */ +typedef void (*rte_bbdev_stats_reset_t)(struct rte_bbdev *dev); + +/** @internal Function to retrieve specific information of a device. */ +typedef void (*rte_bbdev_info_get_t)(struct rte_bbdev *dev, + struct rte_bbdev_driver_info *dev_info); + +/* @internal + * Function to enable interrupt for next op on a queue of a device. + */ +typedef int (*rte_bbdev_queue_intr_enable_t)(struct rte_bbdev *dev, + uint16_t queue_id); + +/* @internal + * Function to disable interrupt for next op on a queue of a device. + */ +typedef int (*rte_bbdev_queue_intr_disable_t)(struct rte_bbdev *dev, + uint16_t queue_id); + +/** + * Operations implemented by drivers. Fields marked as "Required" must be + * provided by a driver for a device to have basic functionality. "Optional" + * fields are for non-vital operations + */ +struct rte_bbdev_ops { + /**< Allocate and configure device memory. Optional. */ + rte_bbdev_setup_queues_t setup_queues; + /**< Configure interrupts. Optional. */ + rte_bbdev_intr_enable_t intr_enable; + /**< Start device. Optional. */ + rte_bbdev_start_t start; + /**< Stop device. Optional. */ + rte_bbdev_stop_t stop; + /**< Close device. Optional. */ + rte_bbdev_close_t close; + + /**< Get device info. Required. */ + rte_bbdev_info_get_t info_get; + /** Get device statistics. Optional. */ + rte_bbdev_stats_get_t stats_get; + /** Reset device statistics. Optional. */ + rte_bbdev_stats_reset_t stats_reset; + + /** Set up a device queue. Required. */ + rte_bbdev_queue_setup_t queue_setup; + /** Release a queue. Required. */ + rte_bbdev_queue_release_t queue_release; + /** Start a queue. Optional. */ + rte_bbdev_queue_start_t queue_start; + /**< Stop a queue pair. Optional. */ + rte_bbdev_queue_stop_t queue_stop; + + /** Enable queue interrupt. Optional */ + rte_bbdev_queue_intr_enable_t queue_intr_enable; + /** Disable queue interrupt. Optional */ + rte_bbdev_queue_intr_disable_t queue_intr_disable; +}; + +/** + * Executes all the user application registered callbacks for the specific + * device and event type. + * + * @param dev + * Pointer to the device structure. + * @param event + * Event type. + * @param ret_param + * To pass data back to user application. + */ +void +rte_bbdev_pmd_callback_process(struct rte_bbdev *dev, + enum rte_bbdev_event_type event, void *ret_param); + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_BBDEV_PMD_H_ */ diff --git a/lib/librte_bbdev/rte_bbdev_vdev.h b/lib/librte_bbdev/rte_bbdev_vdev.h new file mode 100644 index 0000000..fbaef2e --- /dev/null +++ b/lib/librte_bbdev/rte_bbdev_vdev.h @@ -0,0 +1,102 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef _RTE_BBDEV_VDEV_H_ +#define _RTE_BBDEV_VDEV_H_ + +/** + * @file rte_bbdev_vdev.h + * + * Wireless base band virtual device driver-facing APIs. + * + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * This API provides the helper functions for virtual device drivers to register + * with the bbdev interface. User applications should not use this API. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> +#include <rte_vdev.h> + +#include "rte_bbdev_pmd.h" + +/** + * @internal + * Allocates a new slot for a virtual bbdev and returns the pointer to that slot + * for the driver to use. + * + * @param dev + * Pointer to virtual device + * + * @param private_data_size + * Size of private data structure + * + * @return + * A pointer to a rte_bbdev or NULL if allocation failed. + */ +static inline struct rte_bbdev * +rte_bbdev_vdev_allocate(struct rte_vdev_device *dev, size_t private_data_size) +{ + struct rte_bbdev *bbdev; + const char *name = rte_vdev_device_name(dev); + + bbdev = rte_bbdev_allocate(name); + if (!bbdev) + return NULL; + + if (private_data_size) { + bbdev->data->dev_private = rte_zmalloc_socket(name, + private_data_size, RTE_CACHE_LINE_SIZE, + dev->device.numa_node); + if (!bbdev->data->dev_private) { + rte_bbdev_release(bbdev); + return NULL; + } + } + + bbdev->data->socket_id = dev->device.numa_node; + bbdev->device = &dev->device; + bbdev->intr_handle = NULL; + + return bbdev; +} + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_BBDEV_VDEV_H_ */ diff --git a/lib/librte_bbdev/rte_bbdev_version.map b/lib/librte_bbdev/rte_bbdev_version.map new file mode 100644 index 0000000..316a275 --- /dev/null +++ b/lib/librte_bbdev/rte_bbdev_version.map @@ -0,0 +1,37 @@ +EXPERIMENTAL { + global: + + rte_bbdev_allocate; + rte_bbdev_callback_register; + rte_bbdev_callback_unregister; + rte_bbdev_close; + rte_bbdev_setup_queues; + rte_bbdev_intr_enable; + rte_bbdev_count; + rte_bbdev_dequeue_dec_ops; + rte_bbdev_dequeue_enc_ops; + rte_bbdev_devices; + rte_bbdev_enqueue_dec_ops; + rte_bbdev_enqueue_enc_ops; + rte_bbdev_find_next; + rte_bbdev_get_named_dev; + rte_bbdev_info_get; + rte_bbdev_is_valid; + rte_bbdev_op_pool_create; + rte_bbdev_op_type_str; + rte_bbdev_pmd_callback_process; + rte_bbdev_queue_configure; + rte_bbdev_queue_info_get; + rte_bbdev_queue_intr_ctl; + rte_bbdev_queue_intr_disable; + rte_bbdev_queue_intr_enable; + rte_bbdev_queue_start; + rte_bbdev_queue_stop; + rte_bbdev_release; + rte_bbdev_start; + rte_bbdev_stats_get; + rte_bbdev_stats_reset; + rte_bbdev_stop; + + local: *; +}; \ No newline at end of file diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 8192b98..e0f9d13 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -94,6 +94,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_NET) += -lrte_net _LDLIBS-$(CONFIG_RTE_LIBRTE_ETHER) += -lrte_ethdev _LDLIBS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += -lrte_cryptodev _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += -lrte_eventdev +_LDLIBS-$(CONFIG_RTE_LIBRTE_BBDEV) += -lrte_bbdev _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL) += -lrte_mempool _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring @@ -156,6 +157,18 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_VHOST) += -lrte_pmd_vhost endif # $(CONFIG_RTE_LIBRTE_VHOST) _LDLIBS-$(CONFIG_RTE_LIBRTE_VMXNET3_PMD) += -lrte_pmd_vmxnet3_uio +ifeq ($(CONFIG_RTE_LIBRTE_BBDEV),y) +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null + +# TURBO SOFTWARE PMD is dependent on the FLEXRAN library +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_common -lcommon +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_crc -lcrc +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_turbo -lturbo +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lirc -limf -lstdc++ -lipps +endif # CONFIG_RTE_LIBRTE_BBDEV + ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_AESNI_MB) += -lrte_pmd_aesni_mb _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_AESNI_MB) += -L$(AESNI_MULTI_BUFFER_LIB_PATH) -lIPSec_MB -- 2.7.4 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [dpdk-dev] [PATCH v2 2/5] bbdev: PMD drivers (null/turbo_sw) 2017-10-18 2:14 [dpdk-dev] [PATCH v2 1/5] bbdev: librte_bbdev library Amr Mokhtar @ 2017-10-18 2:14 ` Amr Mokhtar 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 3/5] bbdev: test applications Amr Mokhtar ` (3 subsequent siblings) 4 siblings, 0 replies; 6+ messages in thread From: Amr Mokhtar @ 2017-10-18 2:14 UTC (permalink / raw) To: dev Cc: thomas, anatoly.burakov, pablo.de.lara.guarch, niall.power, chris.macnamara, Amr Mokhtar - bbdev 'null' PMD enabled by default - bbdev 'turbo_sw' PMD disabled by default - 'turbo_sw' requires the external FLEXRAN SDK libraries Signed-off-by: Amr Mokhtar <amr.mokhtar@intel.com> --- drivers/Makefile | 2 + drivers/bbdev/Makefile | 41 + drivers/bbdev/null/Makefile | 49 + drivers/bbdev/null/bbdev_null.c | 377 ++++++ drivers/bbdev/null/rte_pmd_bbdev_null_version.map | 3 + drivers/bbdev/turbo_sw/Makefile | 59 + drivers/bbdev/turbo_sw/bbdev_turbo_software.c | 1235 ++++++++++++++++++ .../bbdev/turbo_sw/bbdev_turbo_software_tables.h | 1344 ++++++++++++++++++++ .../turbo_sw/rte_pmd_bbdev_turbo_sw_version.map | 3 + 9 files changed, 3113 insertions(+) create mode 100644 drivers/bbdev/Makefile create mode 100644 drivers/bbdev/null/Makefile create mode 100644 drivers/bbdev/null/bbdev_null.c create mode 100644 drivers/bbdev/null/rte_pmd_bbdev_null_version.map create mode 100644 drivers/bbdev/turbo_sw/Makefile create mode 100644 drivers/bbdev/turbo_sw/bbdev_turbo_software.c create mode 100644 drivers/bbdev/turbo_sw/bbdev_turbo_software_tables.h create mode 100644 drivers/bbdev/turbo_sw/rte_pmd_bbdev_turbo_sw_version.map diff --git a/drivers/Makefile b/drivers/Makefile index 3a5b223..16f5d78 100644 --- a/drivers/Makefile +++ b/drivers/Makefile @@ -40,5 +40,7 @@ DIRS-y += net DEPDIRS-net := bus mempool DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += crypto DEPDIRS-crypto := bus mempool +DIRS-$(CONFIG_RTE_LIBRTE_BBDEV) += bbdev +DEPDIRS-bbdev := mempool include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/drivers/bbdev/Makefile b/drivers/bbdev/Makefile new file mode 100644 index 0000000..c70cd06 --- /dev/null +++ b/drivers/bbdev/Makefile @@ -0,0 +1,41 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +include $(RTE_SDK)/mk/rte.vars.mk + +core-libs := librte_eal librte_mbuf librte_mempool librte_ring +core-libs += librte_bbdev librte_kvargs librte_cfgfile + +DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += null +DEPDIRS-null = $(core-libs) +DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += turbo_sw +DEPDIRS-turbo_sw = $(core-libs) + +include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/drivers/bbdev/null/Makefile b/drivers/bbdev/null/Makefile new file mode 100644 index 0000000..7719eb6 --- /dev/null +++ b/drivers/bbdev/null/Makefile @@ -0,0 +1,49 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +include $(RTE_SDK)/mk/rte.vars.mk +# library name +LIB = librte_pmd_bbdev_null.a + +# build flags +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) + +# versioning export map +EXPORT_MAP := rte_pmd_bbdev_null_version.map + +# library version +LIBABIVER := 1 + +# library source files +SRCS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += bbdev_null.c + +include $(RTE_SDK)/mk/rte.lib.mk + diff --git a/drivers/bbdev/null/bbdev_null.c b/drivers/bbdev/null/bbdev_null.c new file mode 100644 index 0000000..e63904a --- /dev/null +++ b/drivers/bbdev/null/bbdev_null.c @@ -0,0 +1,377 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ +#include <string.h> + +#include <rte_common.h> +#include <rte_vdev.h> +#include <rte_malloc.h> +#include <rte_ring.h> +#include <rte_bbdev_vdev.h> +#include <rte_kvargs.h> + +#define DRIVER_NAME bbdev_null + +/* Initialisation params structure that can be used by null BBDEV driver */ +struct bbdev_null_params { + int socket_id; /*< Null BBDEV socket */ + uint16_t queues_num; /*< Null BBDEV queues number */ +}; + +/* Accecptable params for null BBDEV devices */ +#define BBDEV_NULL_MAX_NB_QUEUES_ARG "max_nb_queues" +#define BBDEV_NULL_SOCKET_ID_ARG "socket_id" + +static const char * const bbdev_null_valid_params[] = { + BBDEV_NULL_MAX_NB_QUEUES_ARG, + BBDEV_NULL_SOCKET_ID_ARG +}; + +/* private data structure */ +struct bbdev_private { + unsigned int max_nb_queues; /**< Max number of queues */ +}; + +/* queue */ +struct bbdev_queue { + struct rte_ring *processed_pkts; /* Ring for processed packets */ +} __rte_cache_aligned; + +/* Get device info */ +static void +info_get(struct rte_bbdev *dev, struct rte_bbdev_driver_info *dev_info) +{ + struct bbdev_private *internals = dev->data->dev_private; + + static const struct rte_bbdev_op_cap bbdev_capabilities[] = { + RTE_BBDEV_END_OF_CAPABILITIES_LIST(), + }; + + static struct rte_bbdev_queue_conf default_queue_conf = { + .queue_size = RTE_BBDEV_QUEUE_SIZE_LIMIT, + }; + + default_queue_conf.socket = dev->data->socket_id; + + dev_info->driver_name = RTE_STR(DRIVER_NAME); + dev_info->max_num_queues = internals->max_nb_queues; + dev_info->queue_size_lim = RTE_BBDEV_QUEUE_SIZE_LIMIT; + dev_info->hardware_accelerated = false; + dev_info->max_queue_priority = 0; + dev_info->default_queue_conf = default_queue_conf; + dev_info->capabilities = bbdev_capabilities; + dev_info->cpu_flag_reqs = NULL; + dev_info->min_alignment = 0; + + rte_bbdev_log_debug("got device info from %u", dev->data->dev_id); +} + +/* Release queue */ +static int +q_release(struct rte_bbdev *dev, uint16_t q_id) +{ + struct bbdev_queue *q = dev->data->queues[q_id].queue_private; + + if (q != NULL) { + rte_ring_free(q->processed_pkts); + rte_free(q); + dev->data->queues[q_id].queue_private = NULL; + } + + rte_bbdev_log_debug("released device queue %u:%u", + dev->data->dev_id, q_id); + return 0; +} + +/* Setup a queue */ +static int +q_setup(struct rte_bbdev *dev, uint16_t q_id, + const struct rte_bbdev_queue_conf *queue_conf) +{ + struct bbdev_queue *q; + char ring_name[RTE_RING_NAMESIZE]; + snprintf(ring_name, RTE_RING_NAMESIZE, RTE_STR(DRIVER_NAME) "%u:%u", + dev->data->dev_id, q_id); + + /* Allocate the queue data structure. */ + q = rte_zmalloc_socket(RTE_STR(DRIVER_NAME), sizeof(*q), + RTE_CACHE_LINE_SIZE, queue_conf->socket); + if (q == NULL) { + rte_bbdev_log(ERR, "Failed to allocate queue memory"); + return -ENOMEM; + } + + q->processed_pkts = rte_ring_create(ring_name, queue_conf->queue_size, + queue_conf->socket, RING_F_SP_ENQ | RING_F_SC_DEQ); + if (q->processed_pkts == NULL) { + rte_bbdev_log(ERR, "Failed to create ring"); + goto free_q; + } + + dev->data->queues[q_id].queue_private = q; + rte_bbdev_log_debug("setup device queue %s", ring_name); + return 0; + +free_q: + rte_free(q); + return -EFAULT; +} + +static const struct rte_bbdev_ops pmd_ops = { + .setup_queues = NULL, + .intr_enable = NULL, + .start = NULL, + .stop = NULL, + .close = NULL, + .info_get = info_get, + .stats_get = NULL, + .stats_reset = NULL, + .queue_setup = q_setup, + .queue_release = q_release, + .queue_start = NULL, + .queue_stop = NULL, + .queue_intr_enable = NULL, + .queue_intr_disable = NULL +}; + +/* Enqueue decode burst */ +static uint16_t +enqueue_dec_ops(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t nb_ops) +{ + struct bbdev_queue *q = q_data->queue_private; + uint16_t nb_enqueued = rte_ring_enqueue_burst(q->processed_pkts, + (void **)ops, nb_ops, NULL); + + q_data->queue_stats.enqueue_err_count += nb_ops - nb_enqueued; + q_data->queue_stats.enqueued_count += nb_enqueued; + + return nb_enqueued; +} + +/* Enqueue encode burst */ +static uint16_t +enqueue_enc_ops(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t nb_ops) +{ + struct bbdev_queue *q = q_data->queue_private; + uint16_t nb_enqueued = rte_ring_enqueue_burst(q->processed_pkts, + (void **)ops, nb_ops, NULL); + + q_data->queue_stats.enqueue_err_count += nb_ops - nb_enqueued; + q_data->queue_stats.enqueued_count += nb_enqueued; + + return nb_enqueued; +} + +/* Dequeue decode burst */ +static uint16_t +dequeue_dec_ops(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t nb_ops) +{ + struct bbdev_queue *q = q_data->queue_private; + uint16_t nb_dequeued = rte_ring_dequeue_burst(q->processed_pkts, + (void **)ops, nb_ops, NULL); + q_data->queue_stats.dequeued_count += nb_dequeued; + + return nb_dequeued; +} + +/* Dequeue encode burst */ +static uint16_t +dequeue_enc_ops(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t nb_ops) +{ + struct bbdev_queue *q = q_data->queue_private; + uint16_t nb_dequeued = rte_ring_dequeue_burst(q->processed_pkts, + (void **)ops, nb_ops, NULL); + q_data->queue_stats.dequeued_count += nb_dequeued; + + return nb_dequeued; +} + +/* Parse 16bit integer from string argument */ +static inline int +parse_u16_arg(const char *key, const char *value, void *extra_args) +{ + uint16_t *u16 = extra_args; + unsigned int long result; + + if ((value == NULL) || (extra_args == NULL)) + return -EINVAL; + errno = 0; + result = strtoul(value, NULL, 0); + if ((result >= (1 << 16)) || (errno != 0)) { + rte_bbdev_log(ERR, "Invalid value %lu for %s", result, key); + return -ERANGE; + } + *u16 = (uint16_t)result; + return 0; +} + +/* Parse parameters used to create device */ +static int +parse_bbdev_null_params(struct bbdev_null_params *params, + const char *input_args) +{ + struct rte_kvargs *kvlist = NULL; + int ret = 0; + + if (params == NULL) + return -EINVAL; + if (input_args) { + kvlist = rte_kvargs_parse(input_args, bbdev_null_valid_params); + if (kvlist == NULL) + return -EFAULT; + + ret = rte_kvargs_process(kvlist, bbdev_null_valid_params[0], + &parse_u16_arg, ¶ms->queues_num); + if (ret < 0) + goto exit; + + ret = rte_kvargs_process(kvlist, bbdev_null_valid_params[1], + &parse_u16_arg, ¶ms->socket_id); + if (ret < 0) + goto exit; + + if (params->socket_id >= RTE_MAX_NUMA_NODES) { + rte_bbdev_log(ERR, "Invalid socket, must be < %u", + RTE_MAX_NUMA_NODES); + goto exit; + } + } + +exit: + if (kvlist) + rte_kvargs_free(kvlist); + return ret; +} + +/* Create device */ +static int +null_bbdev_create(struct rte_vdev_device *vdev, + struct bbdev_null_params *init_params) +{ + struct rte_bbdev *bbdev; + size_t dev_private_size = sizeof(struct bbdev_private); + struct bbdev_private *internals; + + vdev->device.numa_node = init_params->socket_id; + + bbdev = rte_bbdev_vdev_allocate(vdev, dev_private_size); + if (bbdev == NULL) { + rte_bbdev_log(ERR, "Failed to create %s", + rte_vdev_device_name(vdev)); + return -EFAULT; + } + + bbdev->dev_ops = &pmd_ops; + + /* register rx/tx burst functions for data path */ + bbdev->dequeue_enc_ops = dequeue_enc_ops; + bbdev->dequeue_dec_ops = dequeue_dec_ops; + bbdev->enqueue_enc_ops = enqueue_enc_ops; + bbdev->enqueue_dec_ops = enqueue_dec_ops; + internals = bbdev->data->dev_private; + internals->max_nb_queues = init_params->queues_num; + + return 0; +} + +/* Initialise device */ +static int +null_bbdev_probe(struct rte_vdev_device *vdev) +{ + struct bbdev_null_params init_params = { + rte_socket_id(), + RTE_BBDEV_DEFAULT_MAX_NB_QUEUES + }; + const char *name; + const char *input_args; + + if (!vdev) + return -EINVAL; + + name = rte_vdev_device_name(vdev); + if (name == NULL) + return -EINVAL; + input_args = rte_vdev_device_args(vdev); + parse_bbdev_null_params(&init_params, input_args); + + rte_bbdev_log_debug("Init %s on NUMA node %d with max queues: %d", + name, init_params.socket_id, init_params.queues_num); + + return null_bbdev_create(vdev, &init_params); +} + +/* Uninitialise device */ +static int +null_bbdev_remove(struct rte_vdev_device *vdev) +{ + struct rte_bbdev *bbdev; + const char *name; + + if (!vdev) + return -EINVAL; + + name = rte_vdev_device_name(vdev); + if (name == NULL) + return -EINVAL; + + bbdev = rte_bbdev_get_named_dev(name); + if (bbdev == NULL) + return -EINVAL; + + rte_free(bbdev->data->dev_private); + + return rte_bbdev_release(bbdev); +} + +static struct rte_vdev_driver bbdev_null_pmd_drv = { + .probe = null_bbdev_probe, + .remove = null_bbdev_remove +}; + +RTE_PMD_REGISTER_VDEV(DRIVER_NAME, bbdev_null_pmd_drv); +RTE_PMD_REGISTER_ALIAS(DRIVER_NAME, bbdev_null_pmd); +RTE_PMD_REGISTER_PARAM_STRING(DRIVER_NAME, + BBDEV_NULL_MAX_NB_QUEUES_ARG"=<int> " + BBDEV_NULL_SOCKET_ID_ARG"=<int>"); + +int bbdev_logtype; +RTE_INIT(null_bbdev_init_log); +static void +null_bbdev_init_log(void) +{ + bbdev_logtype = rte_log_register("pmd.null_bbdev"); + if (bbdev_logtype >= 0) + rte_log_set_level(bbdev_logtype, RTE_LOG_NOTICE); +} diff --git a/drivers/bbdev/null/rte_pmd_bbdev_null_version.map b/drivers/bbdev/null/rte_pmd_bbdev_null_version.map new file mode 100644 index 0000000..a753031 --- /dev/null +++ b/drivers/bbdev/null/rte_pmd_bbdev_null_version.map @@ -0,0 +1,3 @@ +DPDK_17.11 { + local: *; +}; diff --git a/drivers/bbdev/turbo_sw/Makefile b/drivers/bbdev/turbo_sw/Makefile new file mode 100644 index 0000000..35cfac0 --- /dev/null +++ b/drivers/bbdev/turbo_sw/Makefile @@ -0,0 +1,59 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +include $(RTE_SDK)/mk/rte.vars.mk + +ifeq ($(FLEXRAN_SDK),) +$(error "Please define FLEXRAN_SDK environment variable") +endif + +# library name +LIB = librte_pmd_bbdev_turbo_sw.a + +# build flags +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) + +# versioning export map +EXPORT_MAP := rte_pmd_bbdev_turbo_sw_version.map + +# external library dependencies +CFLAGS += -I$(FLEXRAN_SDK)/lib_common +CFLAGS += -I$(FLEXRAN_SDK)/lib_turbo +CFLAGS += -I$(FLEXRAN_SDK)/lib_crc +CFLAGS += -I$(FLEXRAN_SDK)/lib_rate_matching + +# library version +LIBABIVER := 1 + +# library source files +SRCS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += bbdev_turbo_software.c + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/bbdev/turbo_sw/bbdev_turbo_software.c b/drivers/bbdev/turbo_sw/bbdev_turbo_software.c new file mode 100644 index 0000000..477318b --- /dev/null +++ b/drivers/bbdev/turbo_sw/bbdev_turbo_software.c @@ -0,0 +1,1235 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ +#include <string.h> + +#include <rte_common.h> +#include <rte_vdev.h> +#include <rte_malloc.h> +#include <rte_ring.h> +#include <rte_bbdev_vdev.h> +#include <rte_kvargs.h> + +#include <phy_turbo.h> +#include <phy_crc.h> +#include <phy_rate_match.h> +#include <divide.h> + +#include "bbdev_turbo_software_tables.h" + +#define DRIVER_NAME turbo_sw + +#define MAX_CB_SIZE (6144) +#define MAX_NCB ((MAX_CB_SIZE + 4) * 3) + +/* private data structure */ +struct bbdev_private { + unsigned int max_nb_queues; /**< Max number of queues */ +}; + +/* Initialisation params structure that can be used by Turbo SW driver */ +struct turbo_sw_params { + int socket_id; /*< Turbo SW device socket */ + uint16_t queues_num; /*< Turbo SW device queues number */ +}; + +/* Accecptable params for Turbo SW devices */ +#define TURBO_SW_MAX_NB_QUEUES_ARG "max_nb_queues" +#define TURBO_SW_SOCKET_ID_ARG "socket_id" + +static const char * const turbo_sw_valid_params[] = { + TURBO_SW_MAX_NB_QUEUES_ARG, + TURBO_SW_SOCKET_ID_ARG +}; + +/* queue */ +struct turbo_sw_queue { + /* Ring for processed (encoded/decoded) operations which are ready to + * be dequeued. + */ + struct rte_ring *processed_pkts; + /* Stores output from turbo encoder */ + uint8_t *enc_out; + /* Stores output after rate-matching */ + uint8_t *rm_out; + /* Alpha gamma buf for bblib_turbo_decoder() function */ + int8_t *ag; + /* Temp buf for bblib_turbo_decoder() function */ + uint16_t *code_block; + /* Input buf for bblib_harqcombine_lte() function */ + uint8_t *harq_input; + /* Output buf for bblib_harqcombine_lte() function */ + uint8_t *harq_output; + /* Output buf for bblib_rate_dematching_lte() function */ + uint8_t *deint_output; + /* Output buf for bblib_turbodec_adapter_lte() function */ + uint8_t *adapter_output; + /* Operation type of this queue */ + enum rte_bbdev_op_type type; +} __rte_cache_aligned; + +/* Calculate index based on Table 5.1.3-3 from TS34.212 */ +static inline int32_t +compute_idx(uint16_t k) +{ + int32_t result = 0; + + if (k < 40 || k > MAX_CB_SIZE) + return -1; + + if (k > 2048) { + if ((k - 2048) % 64 != 0) + result = -1; + + result = 124 + (k - 2048) / 64; + } else if (k <= 512) { + if ((k - 40) % 8 != 0) + result = -1; + + result = (k - 40) / 8 + 1; + } else if (k <= 1024) { + if ((k - 512) % 16 != 0) + result = -1; + + result = 60 + (k - 512) / 16; + } else { /* 1024 < k <= 2048 */ + if ((k - 1024) % 32 != 0) + result = -1; + + result = 92 + (k - 1024) / 32; + } + + return result; +} + +/* Read flag value 0/1 from bitmap */ +static inline bool +check_bit(uint32_t bitmap, uint32_t bitmask) +{ + return bitmap & bitmask; +} + +/* Get device info */ +static void +info_get(struct rte_bbdev *dev, struct rte_bbdev_driver_info *dev_info) +{ + struct bbdev_private *internals = dev->data->dev_private; + + static const struct rte_bbdev_op_cap bbdev_capabilities[] = { + { + .type = RTE_BBDEV_OP_TURBO_DEC, + .cap.turbo_dec = { + .capability_flags = + RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE | + RTE_BBDEV_TURBO_RAW_INPUT_DATA, + .num_buffers_src = RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_hard_out = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_soft_out = 0, + } + }, + { + .type = RTE_BBDEV_OP_TURBO_ENC, + .cap.turbo_enc = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_24B_ATTACH | + RTE_BBDEV_TURBO_RATE_MATCH | + RTE_BBDEV_TURBO_RV_INDEX_BYPASS, + .num_buffers_src = RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_dst = RTE_BBDEV_MAX_CODE_BLOCKS, + } + }, + RTE_BBDEV_END_OF_CAPABILITIES_LIST() + }; + + static struct rte_bbdev_queue_conf default_queue_conf = { + .queue_size = RTE_BBDEV_QUEUE_SIZE_LIMIT, + }; + + static const enum rte_cpu_flag_t cpu_flag = RTE_CPUFLAG_SSE4_2; + + default_queue_conf.socket = dev->data->socket_id; + + dev_info->driver_name = RTE_STR(DRIVER_NAME); + dev_info->max_num_queues = internals->max_nb_queues; + dev_info->queue_size_lim = RTE_BBDEV_QUEUE_SIZE_LIMIT; + dev_info->hardware_accelerated = false; + dev_info->max_queue_priority = 0; + dev_info->default_queue_conf = default_queue_conf; + dev_info->capabilities = bbdev_capabilities; + dev_info->cpu_flag_reqs = &cpu_flag; + dev_info->min_alignment = 64; + + rte_bbdev_log_debug("got device info from %u\n", dev->data->dev_id); +} + +/* Release queue */ +static int +q_release(struct rte_bbdev *dev, uint16_t q_id) +{ + struct turbo_sw_queue *q = dev->data->queues[q_id].queue_private; + + if (q != NULL) { + rte_ring_free(q->processed_pkts); + rte_free(q->enc_out); + rte_free(q->rm_out); + rte_free(q->ag); + rte_free(q->code_block); + rte_free(q->harq_input); + rte_free(q->harq_output); + rte_free(q->deint_output); + rte_free(q->adapter_output); + rte_free(q); + dev->data->queues[q_id].queue_private = NULL; + } + + rte_bbdev_log_debug("released device queue %u:%u", + dev->data->dev_id, q_id); + return 0; +} + +/* Setup a queue */ +static int +q_setup(struct rte_bbdev *dev, uint16_t q_id, + const struct rte_bbdev_queue_conf *queue_conf) +{ + int ret; + struct turbo_sw_queue *q; + char name[RTE_RING_NAMESIZE]; + + /* Allocate the queue data structure. */ + q = rte_zmalloc_socket(RTE_STR(DRIVER_NAME), sizeof(*q), + RTE_CACHE_LINE_SIZE, queue_conf->socket); + if (q == NULL) { + rte_bbdev_log(ERR, "Failed to allocate queue memory"); + return -ENOMEM; + } + + /* Allocate memory for encoder output. */ + ret = snprintf(name, RTE_RING_NAMESIZE, RTE_STR(DRIVER_NAME)"_enc_out%u:%u", + dev->data->dev_id, q_id); + if ((ret < 0) || (ret >= (int)RTE_RING_NAMESIZE)) { + rte_bbdev_log(ERR, + "Creating queue name for device %u queue %u failed", + dev->data->dev_id, q_id); + return -ENAMETOOLONG; + } + q->enc_out = rte_zmalloc_socket(name, + ((MAX_CB_SIZE >> 3) + 3) * sizeof(*q->enc_out) * 3, + RTE_CACHE_LINE_SIZE, queue_conf->socket); + if (q->enc_out == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate queue memory for %s", name); + goto free_q; + } + + /* Allocate memory for rate matching output. */ + ret = snprintf(name, RTE_RING_NAMESIZE, RTE_STR(DRIVER_NAME)"_rm_out%u:%u", + dev->data->dev_id, q_id); + if ((ret < 0) || (ret >= (int)RTE_RING_NAMESIZE)) { + rte_bbdev_log(ERR, + "Creating queue name for device %u queue %u failed", + dev->data->dev_id, q_id); + return -ENAMETOOLONG; + } + q->rm_out = rte_zmalloc_socket(name, + ((MAX_CB_SIZE >> 3) + 3) * sizeof(*q->rm_out) * 3, + RTE_CACHE_LINE_SIZE, queue_conf->socket); + if (q->rm_out == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate queue memory for %s", name); + goto free_q; + } + + /* Allocate memory for Aplha Gamma temp buffer. */ + ret = snprintf(name, RTE_RING_NAMESIZE, RTE_STR(DRIVER_NAME)"_ag%u:%u", + dev->data->dev_id, q_id); + if ((ret < 0) || (ret >= (int)RTE_RING_NAMESIZE)) { + rte_bbdev_log(ERR, + "Creating queue name for device %u queue %u failed", + dev->data->dev_id, q_id); + return -ENAMETOOLONG; + } + q->ag = rte_zmalloc_socket(name, + MAX_CB_SIZE * 10 * sizeof(*q->ag), + RTE_CACHE_LINE_SIZE, queue_conf->socket); + if (q->ag == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate queue memory for %s", name); + goto free_q; + } + + /* Allocate memory for code block temp buffer. */ + ret = snprintf(name, RTE_RING_NAMESIZE, RTE_STR(DRIVER_NAME)"_cb%u:%u", + dev->data->dev_id, q_id); + if ((ret < 0) || (ret >= (int)RTE_RING_NAMESIZE)) { + rte_bbdev_log(ERR, + "Creating queue name for device %u queue %u failed", + dev->data->dev_id, q_id); + return -ENAMETOOLONG; + } + q->code_block = rte_zmalloc_socket(name, + (6144 >> 3) * sizeof(*q->code_block), + RTE_CACHE_LINE_SIZE, queue_conf->socket); + if (q->code_block == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate queue memory for %s", name); + goto free_q; + } + + /* Allocate memory for HARQ input. */ + ret = snprintf(name, RTE_RING_NAMESIZE, + RTE_STR(DRIVER_NAME)"_harq_input%u:%u", + dev->data->dev_id, q_id); + if ((ret < 0) || (ret >= (int)RTE_RING_NAMESIZE)) { + rte_bbdev_log(ERR, + "Creating queue name for device %u queue %u failed", + dev->data->dev_id, q_id); + return -ENAMETOOLONG; + } + q->harq_input = rte_zmalloc_socket(name, + MAX_NCB * sizeof(*q->harq_input), + RTE_CACHE_LINE_SIZE, queue_conf->socket); + if (q->harq_input == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate queue memory for %s", name); + goto free_q; + } + + /* Allocate memory for HARQ output. */ + ret = snprintf(name, RTE_RING_NAMESIZE, + RTE_STR(DRIVER_NAME)"_harq_output%u:%u", + dev->data->dev_id, q_id); + if ((ret < 0) || (ret >= (int)RTE_RING_NAMESIZE)) { + rte_bbdev_log(ERR, + "Creating queue name for device %u queue %u failed", + dev->data->dev_id, q_id); + return -ENAMETOOLONG; + } + q->harq_output = rte_zmalloc_socket(name, + MAX_NCB * sizeof(*q->harq_output), + RTE_CACHE_LINE_SIZE, queue_conf->socket); + if (q->harq_output == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate queue memory for %s", name); + goto free_q; + } + + /* Allocate memory for Deinterleaver output. */ + ret = snprintf(name, RTE_RING_NAMESIZE, + RTE_STR(DRIVER_NAME)"_deint_output%u:%u", + dev->data->dev_id, q_id); + if ((ret < 0) || (ret >= (int)RTE_RING_NAMESIZE)) { + rte_bbdev_log(ERR, + "Creating queue name for device %u queue %u failed", + dev->data->dev_id, q_id); + return -ENAMETOOLONG; + } + q->deint_output = rte_zmalloc_socket(NULL, + MAX_NCB * sizeof(*q->deint_output), + RTE_CACHE_LINE_SIZE, queue_conf->socket); + if (q->deint_output == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate queue memory for %s", name); + goto free_q; + } + + /* Allocate memory for Adapter output. */ + ret = snprintf(name, RTE_RING_NAMESIZE, + RTE_STR(DRIVER_NAME)"_adapter_output%u:%u", + dev->data->dev_id, q_id); + if ((ret < 0) || (ret >= (int)RTE_RING_NAMESIZE)) { + rte_bbdev_log(ERR, + "Creating queue name for device %u queue %u failed", + dev->data->dev_id, q_id); + return -ENAMETOOLONG; + } + q->adapter_output = rte_zmalloc_socket(NULL, + MAX_CB_SIZE * 6 * sizeof(*q->adapter_output), + RTE_CACHE_LINE_SIZE, queue_conf->socket); + if (q->adapter_output == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate queue memory for %s", name); + goto free_q; + } + + /* Create ring for packets awaiting to be dequeued. */ + ret = snprintf(name, RTE_RING_NAMESIZE, RTE_STR(DRIVER_NAME)"%u:%u", + dev->data->dev_id, q_id); + if ((ret < 0) || (ret >= (int)RTE_RING_NAMESIZE)) { + rte_bbdev_log(ERR, + "Creating queue name for device %u queue %u failed", + dev->data->dev_id, q_id); + return -ENAMETOOLONG; + } + q->processed_pkts = rte_ring_create(name, queue_conf->queue_size, + queue_conf->socket, RING_F_SP_ENQ | RING_F_SC_DEQ); + if (q->processed_pkts == NULL) { + rte_bbdev_log(ERR, "Failed to create ring for %s", name); + goto free_q; + } + + q->type = queue_conf->op_type; + + dev->data->queues[q_id].queue_private = q; + rte_bbdev_log_debug("setup device queue %s", name); + return 0; + +free_q: + rte_ring_free(q->processed_pkts); + rte_free(q->enc_out); + rte_free(q->rm_out); + rte_free(q->ag); + rte_free(q->code_block); + rte_free(q->harq_input); + rte_free(q->harq_output); + rte_free(q->deint_output); + rte_free(q->adapter_output); + rte_free(q); + return -EFAULT; +} + +static const struct rte_bbdev_ops pmd_ops = { + .setup_queues = NULL, + .intr_enable = NULL, + .start = NULL, + .stop = NULL, + .close = NULL, + .info_get = info_get, + .stats_get = NULL, + .stats_reset = NULL, + .queue_setup = q_setup, + .queue_release = q_release, + .queue_start = NULL, + .queue_stop = NULL, + .queue_intr_enable = NULL, + .queue_intr_disable = NULL +}; + +/* Checks if the encoder input buffer is correct. + * Returns 0 if it's valid, -1 otherwise. + */ +static inline int +is_enc_input_valid(uint16_t k, int32_t k_idx, uint16_t in_length) +{ + if (k_idx < 0) { + rte_bbdev_log(ERR, "K Index is invalid"); + return -1; + } + + if (in_length != (k >> 3)) { + rte_bbdev_log(ERR, + "Mismatch between input length (%u bytes) and K (%u bits)", + in_length, k); + return -1; + } + + if (in_length > (MAX_CB_SIZE >> 3)) { + rte_bbdev_log(ERR, "Input length (%u) is too big, max: %d", + in_length, (MAX_CB_SIZE >> 3)); + return -1; + } + + return 0; +} + +/* Checks if the decoder input buffer is correct. + * Returns 0 if it's valid, -1 otherwise. + */ +static inline int +is_dec_input_valid(int32_t k_idx, uint16_t in_length) +{ + if (k_idx < 0) { + rte_bbdev_log(ERR, "K index is invalid"); + return -1; + } + + if (in_length > MAX_NCB) { + rte_bbdev_log(ERR, "Input length (%u) is too big, max: %d", + in_length, MAX_NCB); + return -1; + } + + return 0; +} + +static inline void +process_enc_cb(struct turbo_sw_queue *q, struct rte_bbdev_enc_op *op, + uint8_t cb_idx, uint8_t cb_total, uint16_t k, uint32_t e, + struct rte_mbuf *m_in, struct rte_mbuf *m_out, + uint16_t in_offset, uint16_t out_offset, uint16_t *total_left) +{ + int ret; + int16_t k_idx, in_length; + uint16_t m; + uint8_t *in, *out0, *out1, *out2, *tmp_out; + struct rte_bbdev_op_turbo_enc enc = op->turbo_enc; + struct bblib_crc_request crc_req; + struct bblib_turbo_encoder_request turbo_req; + struct bblib_turbo_encoder_response turbo_resp; + struct bblib_rate_match_dl_request rm_req; + struct bblib_rate_match_dl_response rm_resp; + + k_idx = compute_idx(k); + + in_length = (*total_left >= (m_in->data_len - in_offset)) ? + m_in->data_len - in_offset : *total_left; + *total_left -= in_length; + + ret = is_enc_input_valid(k, k_idx, in_length); + if (ret != 0) { + op->status |= 1 << RTE_BBDEV_DATA_ERROR; + return; + } + + in = rte_pktmbuf_mtod_offset(m_in, uint8_t *, in_offset); + + /* CRC24B */ + if (enc.op_flags & RTE_BBDEV_TURBO_CRC_24B_ATTACH) { + /* len is shortened by 3 bytes as CRC is attached to + * last 3 bytes. + */ + crc_req.data = in; + crc_req.len = in_length - 3; + if (bblib_lte_crc24b_gen(&crc_req) == -1) { + op->status |= 1 << RTE_BBDEV_CRC_ERROR; + rte_bbdev_log(ERR, "CRC24b generation failed"); + return; + } + } + + /* Turbo encoder */ + + /* Each bit layer output from turbo encoder is (k+4) bits long, i.e. + * input length + 4 tail bits. That's (k/8) + 1 bytes after rounding up. + * So dst_data's length should be 3*(k/8) + 3 bytes. + */ + out0 = q->enc_out; + out1 = RTE_PTR_ADD(out0, in_length + 1); + out2 = RTE_PTR_ADD(out1, in_length + 1); + + turbo_req.case_id = k_idx; + turbo_req.input_win = in; + turbo_req.length = in_length; + turbo_resp.output_win_0 = out0; + turbo_resp.output_win_1 = out1; + turbo_resp.output_win_2 = out2; + if (bblib_turbo_encoder(&turbo_req, &turbo_resp) != 0) { + op->status |= 1 << RTE_BBDEV_DRV_ERROR; + rte_bbdev_log(ERR, "Turbo Encoder failed"); + return; + } + + /* Rate-matching */ + if (enc.op_flags & RTE_BBDEV_TURBO_RATE_MATCH) { + /* index of current code block */ + rm_req.r = cb_idx; + /* total number of code block */ + rm_req.C = cb_total; + /* For DL - 1, UL - 0 */ + rm_req.direction = 1; + rm_req.Nsoft = enc.n_soft; + rm_req.KMIMO = enc.k_mimo; + rm_req.MDL_HARQ = enc.mdl_harq; + rm_req.G = enc.g; + rm_req.NL = enc.nl; + rm_req.Qm = enc.qm; + rm_req.rvidx = enc.rv_index; + rm_req.Kidx = k_idx - 1; + rm_req.nLen = k + 4; + rm_req.tin0 = out0; + rm_req.tin1 = out1; + rm_req.tin2 = out2; + rm_resp.output = q->rm_out; + rm_resp.OutputLen = (e >> 3); + if (enc.op_flags & RTE_BBDEV_TURBO_RV_INDEX_BYPASS) + rm_req.bypass_rvidx = 1; + else + rm_req.bypass_rvidx = 0; + + if (bblib_rate_match_dl(&rm_req, &rm_resp) != 0) { + op->status |= 1 << RTE_BBDEV_DRV_ERROR; + rte_bbdev_log(ERR, "Rate matching failed"); + return; + } + + /* copy rate-match output to turbo_enc entity */ + out0 = (uint8_t *)rte_pktmbuf_append(m_out, + rm_resp.OutputLen); + if (out0 == NULL) { + op->status |= 1 << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log(ERR, + "Too little space in output mbuf"); + return; + } + enc.output.length += rm_resp.OutputLen; + /* rte_bbdev_op_data.offset can be different than the offset + * of the appended bytes + */ + out0 = rte_pktmbuf_mtod_offset(m_out, uint8_t *, + out_offset); + rte_memcpy(out0, q->rm_out, rm_resp.OutputLen); + } else { + /* Rate matching is bypassed */ + + /* Completing last byte of out0 (where 4 tail bits are stored) + * by moving first 4 bits from out1 + */ + tmp_out = (uint8_t *) --out1; + *tmp_out = *tmp_out | ((*(tmp_out + 1) & 0xF0) >> 4); + tmp_out++; + /* Shifting out1 data by 4 bits to the left */ + for (m = 0; m < in_length; ++m) { + uint8_t *first = tmp_out; + uint8_t second = *(tmp_out + 1); + *first = (*first << 4) | ((second & 0xF0) >> 4); + tmp_out++; + } + /* Shifting out2 data by 8 bits to the left */ + for (m = 0; m < in_length + 1; ++m) { + *tmp_out = *(tmp_out + 1); + tmp_out++; + } + *tmp_out = 0; + + /* copy shifted output to turbo_enc entity */ + out0 = (uint8_t *)rte_pktmbuf_append(m_out, + in_length * 3 + 2); + if (out0 == NULL) { + op->status |= 1 << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log(ERR, + "Too little space in output mbuf"); + return; + } + enc.output.length += in_length * 3 + 2; + /* rte_bbdev_op_data.offset can be different than the + * offset of the appended bytes + */ + out0 = rte_pktmbuf_mtod_offset(m_out, uint8_t *, + out_offset); + rte_memcpy(out0, q->enc_out, in_length * 3 + 2); + } +} + +static inline void +enqueue_enc_one_op(struct turbo_sw_queue *q, struct rte_bbdev_enc_op *op) +{ + uint8_t cb_idx = 0; + uint8_t cb_total, cab, c_neg; + uint16_t k_pos, k_neg, k; + uint32_t ea, eb, e; + struct rte_bbdev_op_turbo_enc enc = op->turbo_enc; + uint16_t in_offset = enc.input.offset; + uint16_t out_offset = enc.output.offset; + struct rte_mbuf *m_in = enc.input.data; + struct rte_mbuf *m_out = enc.output.data; + struct rte_mbuf *m_out_next; + uint16_t total_left = enc.input.length; + + if (enc.code_block_mode == 0) { /* For Transport Block mode */ + k_pos = enc.tb_params.k_pos; + k_neg = enc.tb_params.k_neg; + ea = enc.tb_params.ea; + eb = enc.tb_params.eb; + cb_total = enc.tb_params.c; + cab = enc.tb_params.cab; + c_neg = enc.tb_params.c_neg; + } else { /* For Code Block mode */ + k_pos = enc.cb_params.k; + k_neg = enc.cb_params.k; + ea = enc.cb_params.e; + eb = enc.cb_params.e; + cb_total = 1; + cab = 1; + c_neg = 1; + } + + /* Clear op status */ + op->status = 0; + + if (m_in == NULL || m_out == NULL) { + rte_bbdev_log(ERR, "Invalid mbuf pointer"); + op->status = 1 << RTE_BBDEV_DATA_ERROR; + return; + } + + k = (cb_idx < c_neg) ? k_neg : k_pos; + e = (cb_idx < cab) ? ea : eb; + process_enc_cb(q, op, cb_idx, cb_total, k, e, m_in, m_out, in_offset, + out_offset, &total_left); + + for (cb_idx = 1; cb_idx < cb_total; ++cb_idx) { + k = (cb_idx < c_neg) ? k_neg : k_pos; + e = (cb_idx < cab) ? ea : eb; + + m_in = m_in->next; + m_out_next = rte_pktmbuf_alloc(m_out->pool); + if (m_in == NULL || m_out_next == NULL) { + rte_bbdev_log(ERR, "Invalid mbuf pointer"); + op->status |= 1 << RTE_BBDEV_DATA_ERROR; + continue; + } + rte_pktmbuf_chain(m_out, m_out_next); + + process_enc_cb(q, op, cb_idx, cb_total, k, e, m_in, m_out_next, + 0, 0, &total_left); + } + + /* check if all input data was processed */ + if (total_left != 0) { + op->status |= 1 << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log(ERR, + "Mismatch between rte_bbdev_op_data.length and mbuf data"); + } +} + +static inline uint16_t +enqueue_enc_all_ops(struct turbo_sw_queue *q, struct rte_bbdev_enc_op **ops, + uint16_t nb_ops) +{ + uint16_t i; + + for (i = 0; i < nb_ops; ++i) + enqueue_enc_one_op(q, ops[i]); + + return rte_ring_enqueue_burst(q->processed_pkts, (void **)ops, nb_ops, + NULL); +} + +static inline int32_t +calc_k0_without_null(int k0, int k_idx) +{ + int32_t null_num; + int32_t i; + + k_idx--; + null_num = kidx_null_num[k_idx]; + for (i = 0; i < null_num; ++i) + if (k0_null_num[k_idx][i] > k0) + break; + return k0 - i; +} + +static inline int32_t +calc_k0(int k, int rv_idx) +{ + int16_t temp; + int32_t ncb, k0; + int32_t r_tc_subblock = ceili(k+4, 32); + int32_t kpai = r_tc_subblock * 32; + int32_t kw = 3 * kpai; + + ncb = kw; + temp = ceili(ncb, 8 * r_tc_subblock); + k0 = r_tc_subblock * (2 * temp * rv_idx + 2); + + return k0; +} + +/* Remove null bytes starting from k0, return number of non-null bytes */ +static inline uint16_t +remove_null_bytes(const uint8_t *in, uint8_t *out, uint16_t k0, uint16_t len) +{ + uint16_t in_idx, out_idx; + + for (in_idx = k0, out_idx = 0; in_idx < len; ++in_idx) + if (in[in_idx] != 0x00) + out[out_idx++] = in[in_idx]; + + for (in_idx = 0; in_idx < k0; ++in_idx) + if (in[in_idx] != 0x00) + out[out_idx++] = in[in_idx]; + + return out_idx; +} + +static inline int +process_raw_buf_dec_op(struct turbo_sw_queue *q, + struct rte_bbdev_op_turbo_dec *dec, uint8_t *in, + uint16_t in_len, int32_t k_idx, uint16_t k, int8_t **out) +{ + struct bblib_turbo_adapter_ul_request adapter_req; + struct bblib_turbo_adapter_ul_response adapter_resp; + struct bblib_harq_combine_ul_request harq_req; + struct bblib_harq_combine_ul_response harq_resp; + struct bblib_deinterleave_ul_request deint_req; + struct bblib_deinterleave_ul_response deint_resp; + int32_t ncb_without_null, k0, k0_without_null; + + ncb_without_null = (k + 4) * 3; + k0 = calc_k0(k, dec->rv_index); + k0_without_null = calc_k0_without_null(k0, k_idx); + + /* Input contains raw RX data - E bytes. */ + harq_req.e = in_len; + /* Assume that it's never a retransmission. */ + harq_req.isretx = 0; + harq_req.k0withoutnull = k0_without_null; + harq_req.ncb = ncb_without_null; + harq_req.pdmout = in; + harq_resp.pharqbuffer = q->harq_output; + bblib_harq_combine_ul(&harq_req, &harq_resp); + + deint_req.pharqbuffer = q->harq_output; + deint_req.ncb = ncb_without_null; + deint_resp.pinteleavebuffer = q->deint_output; + bblib_deinterleave_ul(&deint_req, &deint_resp); + + adapter_req.isinverted = 0; + adapter_req.ncb = ncb_without_null; + adapter_req.pinteleavebuffer = q->deint_output; + adapter_resp.pharqout = q->adapter_output; + bblib_turbo_adapter_ul(&adapter_req, &adapter_resp); + + *out = (int8_t *)q->adapter_output; + return 0; +} + +static inline int +process_circular_buf_dec_op(struct turbo_sw_queue *q, + struct rte_bbdev_op_turbo_dec *dec, uint8_t *in, + uint16_t in_len, int32_t k_idx, int16_t k, int8_t **out) +{ + struct bblib_turbo_adapter_ul_request adapter_req; + struct bblib_turbo_adapter_ul_response adapter_resp; + int32_t ncb, ncb_without_null; + uint8_t *adapter_input = in; + + /* Input contains the cyclic buffer. */ + ncb = ncb_without_null = in_len; + + if (check_bit(dec->op_flags, RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE)) { + struct bblib_harq_combine_ul_request harq_req; + struct bblib_harq_combine_ul_response harq_resp; + struct bblib_deinterleave_ul_request deint_req; + struct bblib_deinterleave_ul_response deint_resp; + int32_t e, k0, k0_without_null; + + ncb_without_null = (k + 4) * 3; + k0 = calc_k0(k, dec->rv_index); + k0_without_null = calc_k0_without_null(k0, k_idx); + + e = remove_null_bytes(in, q->harq_input, k0, ncb); + + harq_req.e = e; + /* Assume that it's never a retransmission. */ + harq_req.isretx = 0; + harq_req.k0withoutnull = k0_without_null; + harq_req.ncb = ncb_without_null; + harq_req.pdmout = q->harq_input; + harq_resp.pharqbuffer = q->harq_output; + bblib_harq_combine_ul(&harq_req, &harq_resp); + + deint_req.pharqbuffer = q->harq_output; + deint_req.ncb = ncb_without_null; + deint_resp.pinteleavebuffer = q->deint_output; + bblib_deinterleave_ul(&deint_req, &deint_resp); + adapter_input = q->deint_output; + } + + adapter_req.isinverted = 0; + adapter_req.ncb = ncb_without_null; + adapter_req.pinteleavebuffer = adapter_input; + adapter_resp.pharqout = q->adapter_output; + bblib_turbo_adapter_ul(&adapter_req, &adapter_resp); + + *out = (int8_t *)q->adapter_output; + return 0; +} + +static inline void +process_dec_cb(struct turbo_sw_queue *q, struct rte_bbdev_dec_op *op, + uint8_t cb_idx, uint16_t k, struct rte_mbuf *m_in, + struct rte_mbuf *m_out, uint16_t in_offset, uint16_t out_offset, + uint16_t *total_left) +{ + int ret; + int32_t k_idx; + uint16_t in_length; + int32_t iter_cnt; + uint8_t *in, *out; + int8_t *dec_input; + struct bblib_turbo_decoder_request turbo_req; + struct bblib_turbo_decoder_response turbo_resp; + struct rte_bbdev_op_turbo_dec dec = op->turbo_dec; + + k_idx = compute_idx(k); + + in_length = (*total_left >= (m_in->data_len - in_offset)) ? + m_in->data_len - in_offset : *total_left; + *total_left -= in_length; + + ret = is_dec_input_valid(k_idx, in_length); + if (ret != 0) { + op->status |= 1 << RTE_BBDEV_DATA_ERROR; + return; + } + + in = rte_pktmbuf_mtod_offset(m_in, uint8_t *, in_offset); + + if (check_bit(dec.op_flags, RTE_BBDEV_TURBO_RAW_INPUT_DATA)) { + ret = process_raw_buf_dec_op(q, &dec, in, in_length, + k_idx, k, &dec_input); + if (ret != 0) { + op->status |= 1 << RTE_BBDEV_DATA_ERROR; + return; + } + } else { + ret = process_circular_buf_dec_op(q, &dec, in, in_length, + k_idx, k, &dec_input); + if (ret != 0) { + op->status |= 1 << RTE_BBDEV_DATA_ERROR; + return; + } + } + + out = (uint8_t *)rte_pktmbuf_append(m_out, (k >> 3)); + if (out == NULL) { + op->status |= 1 << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log(ERR, "Too little space in output mbuf"); + return; + } + /* rte_bbdev_op_data.offset can be different than the offset of the + * appended bytes + */ + out = rte_pktmbuf_mtod_offset(m_out, uint8_t *, out_offset); + turbo_req.c = cb_idx + 1; + turbo_req.input = dec_input; + turbo_req.k = k; + turbo_req.k_idx = k_idx; + turbo_req.max_iter_num = dec.iter_max; + turbo_resp.ag_buf = q->ag; + turbo_resp.cb_buf = q->code_block; + turbo_resp.output = out; + iter_cnt = bblib_turbo_decoder(&turbo_req, &turbo_resp); + + dec.hard_output.length += (k >> 3); + + if (iter_cnt > 0) + dec.iter_count = RTE_MAX(iter_cnt, dec.iter_count); + else { + op->status |= 1 << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log(ERR, "Turbo Decoder failed"); + return; + } +} + +static inline void +enqueue_dec_one_op(struct turbo_sw_queue *q, struct rte_bbdev_dec_op *op) +{ + uint8_t cb_idx = 0; + uint8_t cb_total, c_neg; + uint16_t k_pos, k_neg, k; + struct rte_bbdev_op_turbo_dec dec = op->turbo_dec; + struct rte_mbuf *m_in = dec.input.data; + struct rte_mbuf *m_out = dec.hard_output.data; + struct rte_mbuf *m_out_next; + uint16_t total_left = dec.input.length; + uint16_t in_offset = dec.input.offset; + uint16_t out_offset = dec.hard_output.offset; + + if (dec.code_block_mode == 0) { /* For Transport Block mode */ + k_pos = dec.tb_params.k_pos; + k_neg = dec.tb_params.k_neg; + cb_total = dec.tb_params.c; + c_neg = dec.tb_params.c_neg; + } else { /* For Code Block mode */ + k_pos = dec.cb_params.k; + k_neg = dec.cb_params.k; + cb_total = 1; + c_neg = 1; + } + + /* Clear op status */ + op->status = 0; + + if (m_in == NULL || m_out == NULL) { + rte_bbdev_log(ERR, "Invalid mbuf pointer"); + op->status = 1 << RTE_BBDEV_DATA_ERROR; + return; + } + + k = (cb_idx < c_neg) ? k_neg : k_pos; + process_dec_cb(q, op, cb_idx, k, m_in, m_out, in_offset, out_offset, + &total_left); + + for (cb_idx = 1; cb_idx < cb_total; ++cb_idx) { + k = (cb_idx < c_neg) ? k_neg : k_pos; + + m_in = m_in->next; + m_out_next = rte_pktmbuf_alloc(m_out->pool); + if (m_in == NULL || m_out_next == NULL) { + rte_bbdev_log(ERR, "Invalid mbuf pointer"); + op->status |= 1 << RTE_BBDEV_DATA_ERROR; + return; + } + rte_pktmbuf_chain(m_out, m_out_next); + + process_dec_cb(q, op, cb_idx, k, m_in, m_out, in_offset, + out_offset, &total_left); + } + if (total_left != 0) { + op->status |= 1 << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log(ERR, + "Mismatch between rte_bbdev_op_data.length and mbuf data"); + } +} + +static inline uint16_t +enqueue_dec_all_ops(struct turbo_sw_queue *q, struct rte_bbdev_dec_op **ops, + uint16_t nb_ops) +{ + uint16_t i; + + for (i = 0; i < nb_ops; ++i) + enqueue_dec_one_op(q, ops[i]); + + return rte_ring_enqueue_burst(q->processed_pkts, (void **)ops, nb_ops, + NULL); +} + +/* Enqueue burst */ +static uint16_t +enqueue_enc_ops(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t nb_ops) +{ + void *queue = q_data->queue_private; + struct turbo_sw_queue *q = queue; + uint16_t nb_enqueued = 0; + + nb_enqueued = enqueue_enc_all_ops(q, ops, nb_ops); + + q_data->queue_stats.enqueue_err_count += nb_ops - nb_enqueued; + q_data->queue_stats.enqueued_count += nb_enqueued; + + return nb_enqueued; +} + +/* Enqueue burst */ +static uint16_t +enqueue_dec_ops(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t nb_ops) +{ + void *queue = q_data->queue_private; + struct turbo_sw_queue *q = queue; + uint16_t nb_enqueued = 0; + + nb_enqueued = enqueue_dec_all_ops(q, ops, nb_ops); + + q_data->queue_stats.enqueue_err_count += nb_ops - nb_enqueued; + q_data->queue_stats.enqueued_count += nb_enqueued; + + return nb_enqueued; +} + +/* Dequeue decode burst */ +static uint16_t +dequeue_dec_ops(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t nb_ops) +{ + struct turbo_sw_queue *q = q_data->queue_private; + uint16_t nb_dequeued = rte_ring_dequeue_burst(q->processed_pkts, + (void **)ops, nb_ops, NULL); + q_data->queue_stats.dequeued_count += nb_dequeued; + + return nb_dequeued; +} + +/* Dequeue encode burst */ +static uint16_t +dequeue_enc_ops(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t nb_ops) +{ + struct turbo_sw_queue *q = q_data->queue_private; + uint16_t nb_dequeued = rte_ring_dequeue_burst(q->processed_pkts, + (void **)ops, nb_ops, NULL); + q_data->queue_stats.dequeued_count += nb_dequeued; + + return nb_dequeued; +} + +/* Parse 16bit integer from string argument */ +static inline int +parse_u16_arg(const char *key, const char *value, void *extra_args) +{ + uint16_t *u16 = extra_args; + unsigned int long result; + + if ((value == NULL) || (extra_args == NULL)) + return -EINVAL; + errno = 0; + result = strtoul(value, NULL, 0); + if ((result >= (1 << 16)) || (errno != 0)) { + rte_bbdev_log(ERR, "Invalid value %lu for %s", result, key); + return -ERANGE; + } + *u16 = (uint16_t)result; + return 0; +} + +/* Parse parameters used to create device */ +static int +parse_turbo_sw_params(struct turbo_sw_params *params, const char *input_args) +{ + struct rte_kvargs *kvlist = NULL; + int ret = 0; + + if (params == NULL) + return -EINVAL; + if (input_args) { + kvlist = rte_kvargs_parse(input_args, turbo_sw_valid_params); + if (kvlist == NULL) + return -EFAULT; + + ret = rte_kvargs_process(kvlist, turbo_sw_valid_params[0], + &parse_u16_arg, ¶ms->queues_num); + if (ret < 0) + goto exit; + + ret = rte_kvargs_process(kvlist, turbo_sw_valid_params[1], + &parse_u16_arg, ¶ms->socket_id); + if (ret < 0) + goto exit; + + if (params->socket_id >= RTE_MAX_NUMA_NODES) { + rte_bbdev_log(ERR, "Invalid socket, must be < %u", + RTE_MAX_NUMA_NODES); + goto exit; + } + } + +exit: + if (kvlist) + rte_kvargs_free(kvlist); + return ret; +} + +/* Create device */ +static int +turbo_sw_bbdev_create(struct rte_vdev_device *vdev, + struct turbo_sw_params *init_params) +{ + struct rte_bbdev *bbdev; + size_t dev_private_size = sizeof(struct bbdev_private); + struct bbdev_private *internals; + + vdev->device.numa_node = init_params->socket_id; + + bbdev = rte_bbdev_vdev_allocate(vdev, dev_private_size); + if (bbdev == NULL) { + rte_bbdev_log(ERR, "Failed to create %s", + rte_vdev_device_name(vdev)); + return -EFAULT; + } + + bbdev->dev_ops = &pmd_ops; + + /* register rx/tx burst functions for data path */ + bbdev->dequeue_enc_ops = dequeue_enc_ops; + bbdev->dequeue_dec_ops = dequeue_dec_ops; + bbdev->enqueue_enc_ops = enqueue_enc_ops; + bbdev->enqueue_dec_ops = enqueue_dec_ops; + internals = bbdev->data->dev_private; + internals->max_nb_queues = init_params->queues_num; + + return 0; +} + +/* Initialise device */ +static int +turbo_sw_bbdev_probe(struct rte_vdev_device *vdev) +{ + struct turbo_sw_params init_params = { + rte_socket_id(), + RTE_BBDEV_DEFAULT_MAX_NB_QUEUES + }; + const char *name; + const char *input_args; + + if (!vdev) + return -EINVAL; + + name = rte_vdev_device_name(vdev); + if (name == NULL) + return -EINVAL; + input_args = rte_vdev_device_args(vdev); + parse_turbo_sw_params(&init_params, input_args); + + rte_bbdev_log_debug( + "Initialising %s on NUMA node %d with max queues: %d\n", + name, init_params.socket_id, init_params.queues_num); + + return turbo_sw_bbdev_create(vdev, &init_params); +} + +/* Uninitialise device */ +static int +turbo_sw_bbdev_remove(struct rte_vdev_device *vdev) +{ + struct rte_bbdev *bbdev; + const char *name; + + if (!vdev) + return -EINVAL; + + name = rte_vdev_device_name(vdev); + if (name == NULL) + return -EINVAL; + + bbdev = rte_bbdev_get_named_dev(name); + if (bbdev == NULL) + return -EINVAL; + + rte_free(bbdev->data->dev_private); + + return rte_bbdev_release(bbdev); +} + +static struct rte_vdev_driver bbdev_turbo_sw_pmd_drv = { + .probe = turbo_sw_bbdev_probe, + .remove = turbo_sw_bbdev_remove +}; + +RTE_PMD_REGISTER_VDEV(DRIVER_NAME, bbdev_turbo_sw_pmd_drv); +RTE_PMD_REGISTER_ALIAS(DRIVER_NAME, bbdev_turbo_sw_pmd); +RTE_PMD_REGISTER_PARAM_STRING(DRIVER_NAME, + TURBO_SW_MAX_NB_QUEUES_ARG"=<int> " + TURBO_SW_SOCKET_ID_ARG"=<int>"); + +int bbdev_logtype; +RTE_INIT(null_bbdev_init_log); +static void +null_bbdev_init_log(void) +{ + bbdev_logtype = rte_log_register("pmd.turbo_sw"); + if (bbdev_logtype >= 0) + rte_log_set_level(bbdev_logtype, RTE_LOG_NOTICE); +} diff --git a/drivers/bbdev/turbo_sw/bbdev_turbo_software_tables.h b/drivers/bbdev/turbo_sw/bbdev_turbo_software_tables.h new file mode 100644 index 0000000..97d9d0e --- /dev/null +++ b/drivers/bbdev/turbo_sw/bbdev_turbo_software_tables.h @@ -0,0 +1,1344 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef _BBDEV_TURBO_SW_TABLES_H +#define _BBDEV_TURBO_SW_TABLES_H + +#include <stdint.h> + +#include <rte_memory.h> +#include "bbdev_turbo_software_tables.h" + +/* Number of NULL bytes for each K index */ +const int32_t kidx_null_num[188] __rte_cache_aligned = { + 60, 36, 12, 84, 60, 36, 12, 84, 60, 36, 12, 84, 60, 36, 12, 84, 60, 36, + 12, 84, 60, 36, 12, 84, 60, 36, 12, 84, 60, 36, 12, 84, 60, 36, 12, 84, + 60, 36, 12, 84, 60, 36, 12, 84, 60, 36, 12, 84, 60, 36, 12, 84, 60, 36, + 12, 84, 60, 36, 12, 84, 36, 84, 36, 84, 36, 84, 36, 84, 36, 84, 36, 84, + 36, 84, 36, 84, 36, 84, 36, 84, 36, 84, 36, 84, 36, 84, 36, 84, 36, 84, + 36, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, + 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, + 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, + 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, + 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, + 84, 84, 84, 84, 84, 84, 84, 84 +}; + +/* Indexes of NULL bytes incremented by 1 for each K index */ +const int32_t k0_null_num[188][84] __rte_cache_aligned = { + { 1, 3, 5, 9, 13, 17, 19, 21, 25, 29, 33, 35, 37, 41, 45, 49, 51, 53, + 57, 61, 65, 66, 69, 70, 73, 74, 81, 82, 89, 90, 97, 98, 101, 102, 105, + 106, 113, 114, 121, 122, 129, 130, 133, 134, 137, 138, 145, 146, 153, + 154, 161, 162, 165, 169, 170, 177, 178, 185, 186, 192, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 5, 9, 17, 21, 25, 33, 37, 41, 49, 53, 57, 65, 66, 73, 74, 81, 82, + 97, 98, 105, 106, 113, 114, 129, 130, 137, 138, 145, 146, 161, 162, + 169, 177, 178, 192, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 17, 33, 49, 65, 66, 97, 98, 129, 130, 161, 192, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 4, 7, 10, 13, 16, 19, 25, 28, 31, 34, 37, 40, 43, 49, 52, 55, 58, + 61, 64, 67, 73, 76, 79, 82, 85, 88, 91, 97, 98, 103, 104, 109, 110, + 115, 116, 121, 122, 127, 128, 133, 134, 145, 146, 151, 152, 157, 158, + 163, 164, 169, 170, 175, 176, 181, 182, 193, 194, 199, 200, 205, 206, + 211, 212, 217, 218, 223, 224, 229, 230, 241, 242, 247, 248, 253, 254, + 259, 265, 266, 271, 272, 277, 278, 288 }, + { 1, 4, 7, 13, 19, 25, 28, 31, 37, 43, 49, 52, 55, 61, 67, 73, 76, 79, + 85, 91, 97, 98, 103, 104, 109, 110, 121, 122, 133, 134, 145, 146, 151, + 152, 157, 158, 169, 170, 181, 182, 193, 194, 199, 200, 205, 206, 217, + 218, 229, 230, 241, 242, 247, 253, 254, 265, 266, 277, 278, 288, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 7, 13, 25, 31, 37, 49, 55, 61, 73, 79, 85, 97, 98, 109, 110, 121, + 122, 145, 146, 157, 158, 169, 170, 193, 194, 205, 206, 217, 218, 241, + 242, 253, 265, 266, 288, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 25, 49, 73, 97, 98, 145, 146, 193, 194, 241, 288, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 5, 9, 13, 17, 21, 25, 33, 37, 41, 45, 49, 53, 57, 65, 69, 73, 77, + 81, 85, 89, 97, 101, 105, 109, 113, 117, 121, 129, 130, 137, 138, 145, + 146, 153, 154, 161, 162, 169, 170, 177, 178, 193, 194, 201, 202, 209, + 210, 217, 218, 225, 226, 233, 234, 241, 242, 257, 258, 265, 266, 273, + 274, 281, 282, 289, 290, 297, 298, 305, 306, 321, 322, 329, 330, 337, + 338, 345, 353, 354, 361, 362, 369, 370, 384 }, + { 1, 5, 9, 17, 25, 33, 37, 41, 49, 57, 65, 69, 73, 81, 89, 97, 101, 105, + 113, 121, 129, 130, 137, 138, 145, 146, 161, 162, 177, 178, 193, 194, + 201, 202, 209, 210, 225, 226, 241, 242, 257, 258, 265, 266, 273, 274, + 289, 290, 305, 306, 321, 322, 329, 337, 338, 353, 354, 369, 370, 384, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0 }, + { 1, 9, 17, 33, 41, 49, 65, 73, 81, 97, 105, 113, 129, 130, 145, 146, + 161, 162, 193, 194, 209, 210, 225, 226, 257, 258, 273, 274, 289, 290, + 321, 322, 337, 353, 354, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 33, 65, 97, 129, 130, 193, 194, 257, 258, 321, 384, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 6, 11, 16, 21, 26, 31, 41, 46, 51, 56, 61, 66, 71, 81, 86, 91, 96, + 101, 106, 111, 121, 126, 131, 136, 141, 146, 151, 161, 162, 171, 172, + 181, 182, 191, 192, 201, 202, 211, 212, 221, 222, 241, 242, 251, 252, + 261, 262, 271, 272, 281, 282, 291, 292, 301, 302, 321, 322, 331, 332, + 341, 342, 351, 352, 361, 362, 371, 372, 381, 382, 401, 402, 411, 412, + 421, 422, 431, 441, 442, 451, 452, 461, 462, 480 }, + { 1, 6, 11, 21, 31, 41, 46, 51, 61, 71, 81, 86, 91, 101, 111, 121, 126, + 131, 141, 151, 161, 162, 171, 172, 181, 182, 201, 202, 221, 222, 241, + 242, 251, 252, 261, 262, 281, 282, 301, 302, 321, 322, 331, 332, 341, + 342, 361, 362, 381, 382, 401, 402, 411, 421, 422, 441, 442, 461, 462, + 480, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0 }, + { 1, 11, 21, 41, 51, 61, 81, 91, 101, 121, 131, 141, 161, 162, 181, 182, + 201, 202, 241, 242, 261, 262, 281, 282, 321, 322, 341, 342, 361, 362, + 401, 402, 421, 441, 442, 480, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 41, 81, 121, 161, 162, 241, 242, 321, 322, 401, 480, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 7, 13, 19, 25, 31, 37, 49, 55, 61, 67, 73, 79, 85, 97, 103, 109, + 115, 121, 127, 133, 145, 151, 157, 163, 169, 175, 181, 193, 194, 205, + 206, 217, 218, 229, 230, 241, 242, 253, 254, 265, 266, 289, 290, 301, + 302, 313, 314, 325, 326, 337, 338, 349, 350, 361, 362, 385, 386, 397, + 398, 409, 410, 421, 422, 433, 434, 445, 446, 457, 458, 481, 482, 493, + 494, 505, 506, 517, 529, 530, 541, 542, 553, 554, 576 }, + { 1, 7, 13, 25, 37, 49, 55, 61, 73, 85, 97, 103, 109, 121, 133, 145, + 151, 157, 169, 181, 193, 194, 205, 206, 217, 218, 241, 242, 265, 266, + 289, 290, 301, 302, 313, 314, 337, 338, 361, 362, 385, 386, 397, 398, + 409, 410, 433, 434, 457, 458, 481, 482, 493, 505, 506, 529, 530, 553, + 554, 576, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0 }, + { 1, 13, 25, 49, 61, 73, 97, 109, 121, 145, 157, 169, 193, 194, 217, + 218, 241, 242, 289, 290, 313, 314, 337, 338, 385, 386, 409, 410, 433, + 434, 481, 482, 505, 529, 530, 576, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 49, 97, 145, 193, 194, 289, 290, 385, 386, 481, 576, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 8, 15, 22, 29, 36, 43, 57, 64, 71, 78, 85, 92, 99, 113, 120, 127, + 134, 141, 148, 155, 169, 176, 183, 190, 197, 204, 211, 225, 226, 239, + 240, 253, 254, 267, 268, 281, 282, 295, 296, 309, 310, 337, 338, 351, + 352, 365, 366, 379, 380, 393, 394, 407, 408, 421, 422, 449, 450, 463, + 464, 477, 478, 491, 492, 505, 506, 519, 520, 533, 534, 561, 562, 575, + 576, 589, 590, 603, 617, 618, 631, 632, 645, 646, 672 }, + { 1, 8, 15, 29, 43, 57, 64, 71, 85, 99, 113, 120, 127, 141, 155, 169, + 176, 183, 197, 211, 225, 226, 239, 240, 253, 254, 281, 282, 309, 310, + 337, 338, 351, 352, 365, 366, 393, 394, 421, 422, 449, 450, 463, 464, + 477, 478, 505, 506, 533, 534, 561, 562, 575, 589, 590, 617, 618, 645, + 646, 672, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0 }, + { 1, 15, 29, 57, 71, 85, 113, 127, 141, 169, 183, 197, 225, 226, 253, + 254, 281, 282, 337, 338, 365, 366, 393, 394, 449, 450, 477, 478, 505, + 506, 561, 562, 589, 617, 618, 672, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 57, 113, 169, 225, 226, 337, 338, 449, 450, 561, 672, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 9, 17, 25, 33, 41, 49, 65, 73, 81, 89, 97, 105, 113, 129, 137, 145, + 153, 161, 169, 177, 193, 201, 209, 217, 225, 233, 241, 257, 258, 273, + 274, 289, 290, 305, 306, 321, 322, 337, 338, 353, 354, 385, 386, 401, + 402, 417, 418, 433, 434, 449, 450, 465, 466, 481, 482, 513, 514, 529, + 530, 545, 546, 561, 562, 577, 578, 593, 594, 609, 610, 641, 642, 657, + 658, 673, 674, 689, 705, 706, 721, 722, 737, 738, 768 }, + { 1, 9, 17, 33, 49, 65, 73, 81, 97, 113, 129, 137, 145, 161, 177, 193, + 201, 209, 225, 241, 257, 258, 273, 274, 289, 290, 321, 322, 353, 354, + 385, 386, 401, 402, 417, 418, 449, 450, 481, 482, 513, 514, 529, 530, + 545, 546, 577, 578, 609, 610, 641, 642, 657, 673, 674, 705, 706, 737, + 738, 768, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0 }, + { 1, 17, 33, 65, 81, 97, 129, 145, 161, 193, 209, 225, 257, 258, 289, + 290, 321, 322, 385, 386, 417, 418, 449, 450, 513, 514, 545, 546, 577, + 578, 641, 642, 673, 705, 706, 768, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 65, 129, 193, 257, 258, 385, 386, 513, 514, 641, 768, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 10, 19, 28, 37, 46, 55, 73, 82, 91, 100, 109, 118, 127, 145, 154, + 163, 172, 181, 190, 199, 217, 226, 235, 244, 253, 262, 271, 289, 290, + 307, 308, 325, 326, 343, 344, 361, 362, 379, 380, 397, 398, 433, 434, + 451, 452, 469, 470, 487, 488, 505, 506, 523, 524, 541, 542, 577, 578, + 595, 596, 613, 614, 631, 632, 649, 650, 667, 668, 685, 686, 721, 722, + 739, 740, 757, 758, 775, 793, 794, 811, 812, 829, 830, 864 }, + { 1, 10, 19, 37, 55, 73, 82, 91, 109, 127, 145, 154, 163, 181, 199, 217, + 226, 235, 253, 271, 289, 290, 307, 308, 325, 326, 361, 362, 397, 398, + 433, 434, 451, 452, 469, 470, 505, 506, 541, 542, 577, 578, 595, 596, + 613, 614, 649, 650, 685, 686, 721, 722, 739, 757, 758, 793, 794, 829, + 830, 864, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0 }, + { 1, 19, 37, 73, 91, 109, 145, 163, 181, 217, 235, 253, 289, 290, 325, + 326, 361, 362, 433, 434, 469, 470, 505, 506, 577, 578, 613, 614, 649, + 650, 721, 722, 757, 793, 794, 864, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 73, 145, 217, 289, 290, 433, 434, 577, 578, 721, 864, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 11, 21, 31, 41, 51, 61, 81, 91, 101, 111, 121, 131, 141, 161, 171, + 181, 191, 201, 211, 221, 241, 251, 261, 271, 281, 291, 301, 321, 322, + 341, 342, 361, 362, 381, 382, 401, 402, 421, 422, 441, 442, 481, 482, + 501, 502, 521, 522, 541, 542, 561, 562, 581, 582, 601, 602, 641, 642, + 661, 662, 681, 682, 701, 702, 721, 722, 741, 742, 761, 762, 801, 802, + 821, 822, 841, 842, 861, 881, 882, 901, 902, 921, 922, 960 }, + { 1, 11, 21, 41, 61, 81, 91, 101, 121, 141, 161, 171, 181, 201, 221, + 241, 251, 261, 281, 301, 321, 322, 341, 342, 361, 362, 401, 402, 441, + 442, 481, 482, 501, 502, 521, 522, 561, 562, 601, 602, 641, 642, 661, + 662, 681, 682, 721, 722, 761, 762, 801, 802, 821, 841, 842, 881, 882, + 921, 922, 960, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0 }, + { 1, 21, 41, 81, 101, 121, 161, 181, 201, 241, 261, 281, 321, 322, 361, + 362, 401, 402, 481, 482, 521, 522, 561, 562, 641, 642, 681, 682, 721, + 722, 801, 802, 841, 881, 882, 960, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 81, 161, 241, 321, 322, 481, 482, 641, 642, 801, 960, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 12, 23, 34, 45, 56, 67, 89, 100, 111, 122, 133, 144, 155, 177, 188, + 199, 210, 221, 232, 243, 265, 276, 287, 298, 309, 320, 331, 353, 354, + 375, 376, 397, 398, 419, 420, 441, 442, 463, 464, 485, 486, 529, 530, + 551, 552, 573, 574, 595, 596, 617, 618, 639, 640, 661, 662, 705, 706, + 727, 728, 749, 750, 771, 772, 793, 794, 815, 816, 837, 838, 881, 882, + 903, 904, 925, 926, 947, 969, 970, 991, 992, 1013, 1014, 1056 }, + { 1, 12, 23, 45, 67, 89, 100, 111, 133, 155, 177, 188, 199, 221, 243, + 265, 276, 287, 309, 331, 353, 354, 375, 376, 397, 398, 441, 442, 485, + 486, 529, 530, 551, 552, 573, 574, 617, 618, 661, 662, 705, 706, 727, + 728, 749, 750, 793, 794, 837, 838, 881, 882, 903, 925, 926, 969, 970, + 1013, 1014, 1056, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0 }, + { 1, 23, 45, 89, 111, 133, 177, 199, 221, 265, 287, 309, 353, 354, 397, + 398, 441, 442, 529, 530, 573, 574, 617, 618, 705, 706, 749, 750, 793, + 794, 881, 882, 925, 969, 970, 1056, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 89, 177, 265, 353, 354, 529, 530, 705, 706, 881, 1056, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 13, 25, 37, 49, 61, 73, 97, 109, 121, 133, 145, 157, 169, 193, 205, + 217, 229, 241, 253, 265, 289, 301, 313, 325, 337, 349, 361, 385, 386, + 409, 410, 433, 434, 457, 458, 481, 482, 505, 506, 529, 530, 577, 578, + 601, 602, 625, 626, 649, 650, 673, 674, 697, 698, 721, 722, 769, 770, + 793, 794, 817, 818, 841, 842, 865, 866, 889, 890, 913, 914, 961, 962, + 985, 986, 1009, 1010, 1033, 1057, 1058, 1081, 1082, 1105, 1106, + 1152 }, + { 1, 13, 25, 49, 73, 97, 109, 121, 145, 169, 193, 205, 217, 241, 265, + 289, 301, 313, 337, 361, 385, 386, 409, 410, 433, 434, 481, 482, 529, + 530, 577, 578, 601, 602, 625, 626, 673, 674, 721, 722, 769, 770, 793, + 794, 817, 818, 865, 866, 913, 914, 961, 962, 985, 1009, 1010, 1057, + 1058, 1105, 1106, 1152, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 25, 49, 97, 121, 145, 193, 217, 241, 289, 313, 337, 385, 386, 433, + 434, 481, 482, 577, 578, 625, 626, 673, 674, 769, 770, 817, 818, 865, + 866, 961, 962, 1009, 1057, 1058, 1152, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 97, 193, 289, 385, 386, 577, 578, 769, 770, 961, 1152, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 14, 27, 40, 53, 66, 79, 105, 118, 131, 144, 157, 170, 183, 209, + 222, 235, 248, 261, 274, 287, 313, 326, 339, 352, 365, 378, 391, 417, + 418, 443, 444, 469, 470, 495, 496, 521, 522, 547, 548, 573, 574, 625, + 626, 651, 652, 677, 678, 703, 704, 729, 730, 755, 756, 781, 782, 833, + 834, 859, 860, 885, 886, 911, 912, 937, 938, 963, 964, 989, 990, 1041, + 1042, 1067, 1068, 1093, 1094, 1119, 1145, 1146, 1171, 1172, 1197, + 1198, 1248 }, + { 1, 14, 27, 53, 79, 105, 118, 131, 157, 183, 209, 222, 235, 261, 287, + 313, 326, 339, 365, 391, 417, 418, 443, 444, 469, 470, 521, 522, 573, + 574, 625, 626, 651, 652, 677, 678, 729, 730, 781, 782, 833, 834, 859, + 860, 885, 886, 937, 938, 989, 990, 1041, 1042, 1067, 1093, 1094, 1145, + 1146, 1197, 1198, 1248, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 27, 53, 105, 131, 157, 209, 235, 261, 313, 339, 365, 417, 418, 469, + 470, 521, 522, 625, 626, 677, 678, 729, 730, 833, 834, 885, 886, 937, + 938, 1041, 1042, 1093, 1145, 1146, 1248, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 105, 209, 313, 417, 418, 625, 626, 833, 834, 1041, 1248, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 15, 29, 43, 57, 71, 85, 113, 127, 141, 155, 169, 183, 197, 225, + 239, 253, 267, 281, 295, 309, 337, 351, 365, 379, 393, 407, 421, 449, + 450, 477, 478, 505, 506, 533, 534, 561, 562, 589, 590, 617, 618, 673, + 674, 701, 702, 729, 730, 757, 758, 785, 786, 813, 814, 841, 842, 897, + 898, 925, 926, 953, 954, 981, 982, 1009, 1010, 1037, 1038, 1065, 1066, + 1121, 1122, 1149, 1150, 1177, 1178, 1205, 1233, 1234, 1261, 1262, + 1289, 1290, 1344 }, + { 1, 15, 29, 57, 85, 113, 127, 141, 169, 197, 225, 239, 253, 281, 309, + 337, 351, 365, 393, 421, 449, 450, 477, 478, 505, 506, 561, 562, 617, + 618, 673, 674, 701, 702, 729, 730, 785, 786, 841, 842, 897, 898, 925, + 926, 953, 954, 1009, 1010, 1065, 1066, 1121, 1122, 1149, 1177, 1178, + 1233, 1234, 1289, 1290, 1344, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 29, 57, 113, 141, 169, 225, 253, 281, 337, 365, 393, 449, 450, 505, + 506, 561, 562, 673, 674, 729, 730, 785, 786, 897, 898, 953, 954, 1009, + 1010, 1121, 1122, 1177, 1233, 1234, 1344, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 113, 225, 337, 449, 450, 673, 674, 897, 898, 1121, 1344, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 16, 31, 46, 61, 76, 91, 121, 136, 151, 166, 181, 196, 211, 241, + 256, 271, 286, 301, 316, 331, 361, 376, 391, 406, 421, 436, 451, 481, + 482, 511, 512, 541, 542, 571, 572, 601, 602, 631, 632, 661, 662, 721, + 722, 751, 752, 781, 782, 811, 812, 841, 842, 871, 872, 901, 902, 961, + 962, 991, 992, 1021, 1022, 1051, 1052, 1081, 1082, 1111, 1112, 1141, + 1142, 1201, 1202, 1231, 1232, 1261, 1262, 1291, 1321, 1322, 1351, + 1352, 1381, 1382, 1440 }, + { 1, 16, 31, 61, 91, 121, 136, 151, 181, 211, 241, 256, 271, 301, 331, + 361, 376, 391, 421, 451, 481, 482, 511, 512, 541, 542, 601, 602, 661, + 662, 721, 722, 751, 752, 781, 782, 841, 842, 901, 902, 961, 962, 991, + 992, 1021, 1022, 1081, 1082, 1141, 1142, 1201, 1202, 1231, 1261, 1262, + 1321, 1322, 1381, 1382, 1440, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 31, 61, 121, 151, 181, 241, 271, 301, 361, 391, 421, 481, 482, 541, + 542, 601, 602, 721, 722, 781, 782, 841, 842, 961, 962, 1021, 1022, + 1081, 1082, 1201, 1202, 1261, 1321, 1322, 1440, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 121, 241, 361, 481, 482, 721, 722, 961, 962, 1201, 1440, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 17, 33, 49, 65, 81, 97, 129, 145, 161, 177, 193, 209, 225, 257, + 273, 289, 305, 321, 337, 353, 385, 401, 417, 433, 449, 465, 481, 513, + 514, 545, 546, 577, 578, 609, 610, 641, 642, 673, 674, 705, 706, 769, + 770, 801, 802, 833, 834, 865, 866, 897, 898, 929, 930, 961, 962, 1025, + 1026, 1057, 1058, 1089, 1090, 1121, 1122, 1153, 1154, 1185, 1186, + 1217, 1218, 1281, 1282, 1313, 1314, 1345, 1346, 1377, 1409, 1410, + 1441, 1442, 1473, 1474, 1536 }, + { 1, 17, 33, 65, 97, 129, 145, 161, 193, 225, 257, 273, 289, 321, 353, + 385, 401, 417, 449, 481, 513, 514, 545, 546, 577, 578, 641, 642, 705, + 706, 769, 770, 801, 802, 833, 834, 897, 898, 961, 962, 1025, 1026, + 1057, 1058, 1089, 1090, 1153, 1154, 1217, 1218, 1281, 1282, 1313, + 1345, 1346, 1409, 1410, 1473, 1474, 1536, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 33, 65, 129, 161, 193, 257, 289, 321, 385, 417, 449, 513, 514, 577, + 578, 641, 642, 769, 770, 833, 834, 897, 898, 1025, 1026, 1089, 1090, + 1153, 1154, 1281, 1282, 1345, 1409, 1410, 1536, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 129, 257, 385, 513, 514, 769, 770, 1025, 1026, 1281, 1536, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 18, 35, 52, 69, 86, 103, 137, 154, 171, 188, 205, 222, 239, 273, + 290, 307, 324, 341, 358, 375, 409, 426, 443, 460, 477, 494, 511, 545, + 546, 579, 580, 613, 614, 647, 648, 681, 682, 715, 716, 749, 750, 817, + 818, 851, 852, 885, 886, 919, 920, 953, 954, 987, 988, 1021, 1022, + 1089, 1090, 1123, 1124, 1157, 1158, 1191, 1192, 1225, 1226, 1259, + 1260, 1293, 1294, 1361, 1362, 1395, 1396, 1429, 1430, 1463, 1497, + 1498, 1531, 1532, 1565, 1566, 1632 }, + { 1, 35, 69, 137, 171, 205, 273, 307, 341, 409, 443, 477, 545, 546, 613, + 614, 681, 682, 817, 818, 885, 886, 953, 954, 1089, 1090, 1157, 1158, + 1225, 1226, 1361, 1362, 1429, 1497, 1498, 1632, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 19, 37, 55, 73, 91, 109, 145, 163, 181, 199, 217, 235, 253, 289, + 307, 325, 343, 361, 379, 397, 433, 451, 469, 487, 505, 523, 541, 577, + 578, 613, 614, 649, 650, 685, 686, 721, 722, 757, 758, 793, 794, 865, + 866, 901, 902, 937, 938, 973, 974, 1009, 1010, 1045, 1046, 1081, 1082, + 1153, 1154, 1189, 1190, 1225, 1226, 1261, 1262, 1297, 1298, 1333, + 1334, 1369, 1370, 1441, 1442, 1477, 1478, 1513, 1514, 1549, 1585, + 1586, 1621, 1622, 1657, 1658, 1728 }, + { 1, 37, 73, 145, 181, 217, 289, 325, 361, 433, 469, 505, 577, 578, 649, + 650, 721, 722, 865, 866, 937, 938, 1009, 1010, 1153, 1154, 1225, 1226, + 1297, 1298, 1441, 1442, 1513, 1585, 1586, 1728, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 20, 39, 58, 77, 96, 115, 153, 172, 191, 210, 229, 248, 267, 305, + 324, 343, 362, 381, 400, 419, 457, 476, 495, 514, 533, 552, 571, 609, + 610, 647, 648, 685, 686, 723, 724, 761, 762, 799, 800, 837, 838, 913, + 914, 951, 952, 989, 990, 1027, 1028, 1065, 1066, 1103, 1104, 1141, + 1142, 1217, 1218, 1255, 1256, 1293, 1294, 1331, 1332, 1369, 1370, + 1407, 1408, 1445, 1446, 1521, 1522, 1559, 1560, 1597, 1598, 1635, + 1673, 1674, 1711, 1712, 1749, 1750, 1824 }, + { 1, 39, 77, 153, 191, 229, 305, 343, 381, 457, 495, 533, 609, 610, 685, + 686, 761, 762, 913, 914, 989, 990, 1065, 1066, 1217, 1218, 1293, 1294, + 1369, 1370, 1521, 1522, 1597, 1673, 1674, 1824, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 21, 41, 61, 81, 101, 121, 161, 181, 201, 221, 241, 261, 281, 321, + 341, 361, 381, 401, 421, 441, 481, 501, 521, 541, 561, 581, 601, 641, + 642, 681, 682, 721, 722, 761, 762, 801, 802, 841, 842, 881, 882, 961, + 962, 1001, 1002, 1041, 1042, 1081, 1082, 1121, 1122, 1161, 1162, 1201, + 1202, 1281, 1282, 1321, 1322, 1361, 1362, 1401, 1402, 1441, 1442, + 1481, 1482, 1521, 1522, 1601, 1602, 1641, 1642, 1681, 1682, 1721, + 1761, 1762, 1801, 1802, 1841, 1842, 1920 }, + { 1, 41, 81, 161, 201, 241, 321, 361, 401, 481, 521, 561, 641, 642, 721, + 722, 801, 802, 961, 962, 1041, 1042, 1121, 1122, 1281, 1282, 1361, + 1362, 1441, 1442, 1601, 1602, 1681, 1761, 1762, 1920, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 22, 43, 64, 85, 106, 127, 169, 190, 211, 232, 253, 274, 295, 337, + 358, 379, 400, 421, 442, 463, 505, 526, 547, 568, 589, 610, 631, 673, + 674, 715, 716, 757, 758, 799, 800, 841, 842, 883, 884, 925, 926, 1009, + 1010, 1051, 1052, 1093, 1094, 1135, 1136, 1177, 1178, 1219, 1220, + 1261, 1262, 1345, 1346, 1387, 1388, 1429, 1430, 1471, 1472, 1513, + 1514, 1555, 1556, 1597, 1598, 1681, 1682, 1723, 1724, 1765, 1766, + 1807, 1849, 1850, 1891, 1892, 1933, 1934, 2016 }, + { 1, 43, 85, 169, 211, 253, 337, 379, 421, 505, 547, 589, 673, 674, 757, + 758, 841, 842, 1009, 1010, 1093, 1094, 1177, 1178, 1345, 1346, 1429, + 1430, 1513, 1514, 1681, 1682, 1765, 1849, 1850, 2016, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 23, 45, 67, 89, 111, 133, 177, 199, 221, 243, 265, 287, 309, 353, + 375, 397, 419, 441, 463, 485, 529, 551, 573, 595, 617, 639, 661, 705, + 706, 749, 750, 793, 794, 837, 838, 881, 882, 925, 926, 969, 970, 1057, + 1058, 1101, 1102, 1145, 1146, 1189, 1190, 1233, 1234, 1277, 1278, + 1321, 1322, 1409, 1410, 1453, 1454, 1497, 1498, 1541, 1542, 1585, + 1586, 1629, 1630, 1673, 1674, 1761, 1762, 1805, 1806, 1849, 1850, + 1893, 1937, 1938, 1981, 1982, 2025, 2026, 2112 }, + { 1, 45, 89, 177, 221, 265, 353, 397, 441, 529, 573, 617, 705, 706, 793, + 794, 881, 882, 1057, 1058, 1145, 1146, 1233, 1234, 1409, 1410, 1497, + 1498, 1585, 1586, 1761, 1762, 1849, 1937, 1938, 2112, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 24, 47, 70, 93, 116, 139, 185, 208, 231, 254, 277, 300, 323, 369, + 392, 415, 438, 461, 484, 507, 553, 576, 599, 622, 645, 668, 691, 737, + 738, 783, 784, 829, 830, 875, 876, 921, 922, 967, 968, 1013, 1014, + 1105, 1106, 1151, 1152, 1197, 1198, 1243, 1244, 1289, 1290, 1335, + 1336, 1381, 1382, 1473, 1474, 1519, 1520, 1565, 1566, 1611, 1612, + 1657, 1658, 1703, 1704, 1749, 1750, 1841, 1842, 1887, 1888, 1933, + 1934, 1979, 2025, 2026, 2071, 2072, 2117, 2118, 2208 }, + { 1, 47, 93, 185, 231, 277, 369, 415, 461, 553, 599, 645, 737, 738, 829, + 830, 921, 922, 1105, 1106, 1197, 1198, 1289, 1290, 1473, 1474, 1565, + 1566, 1657, 1658, 1841, 1842, 1933, 2025, 2026, 2208, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 25, 49, 73, 97, 121, 145, 193, 217, 241, 265, 289, 313, 337, 385, + 409, 433, 457, 481, 505, 529, 577, 601, 625, 649, 673, 697, 721, 769, + 770, 817, 818, 865, 866, 913, 914, 961, 962, 1009, 1010, 1057, 1058, + 1153, 1154, 1201, 1202, 1249, 1250, 1297, 1298, 1345, 1346, 1393, + 1394, 1441, 1442, 1537, 1538, 1585, 1586, 1633, 1634, 1681, 1682, + 1729, 1730, 1777, 1778, 1825, 1826, 1921, 1922, 1969, 1970, 2017, + 2018, 2065, 2113, 2114, 2161, 2162, 2209, 2210, 2304 }, + { 1, 49, 97, 193, 241, 289, 385, 433, 481, 577, 625, 673, 769, 770, 865, + 866, 961, 962, 1153, 1154, 1249, 1250, 1345, 1346, 1537, 1538, 1633, + 1634, 1729, 1730, 1921, 1922, 2017, 2113, 2114, 2304, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 26, 51, 76, 101, 126, 151, 201, 226, 251, 276, 301, 326, 351, 401, + 426, 451, 476, 501, 526, 551, 601, 626, 651, 676, 701, 726, 751, 801, + 802, 851, 852, 901, 902, 951, 952, 1001, 1002, 1051, 1052, 1101, 1102, + 1201, 1202, 1251, 1252, 1301, 1302, 1351, 1352, 1401, 1402, 1451, + 1452, 1501, 1502, 1601, 1602, 1651, 1652, 1701, 1702, 1751, 1752, + 1801, 1802, 1851, 1852, 1901, 1902, 2001, 2002, 2051, 2052, 2101, + 2102, 2151, 2201, 2202, 2251, 2252, 2301, 2302, 2400 }, + { 1, 51, 101, 201, 251, 301, 401, 451, 501, 601, 651, 701, 801, 802, + 901, 902, 1001, 1002, 1201, 1202, 1301, 1302, 1401, 1402, 1601, 1602, + 1701, 1702, 1801, 1802, 2001, 2002, 2101, 2201, 2202, 2400, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 27, 53, 79, 105, 131, 157, 209, 235, 261, 287, 313, 339, 365, 417, + 443, 469, 495, 521, 547, 573, 625, 651, 677, 703, 729, 755, 781, 833, + 834, 885, 886, 937, 938, 989, 990, 1041, 1042, 1093, 1094, 1145, 1146, + 1249, 1250, 1301, 1302, 1353, 1354, 1405, 1406, 1457, 1458, 1509, + 1510, 1561, 1562, 1665, 1666, 1717, 1718, 1769, 1770, 1821, 1822, + 1873, 1874, 1925, 1926, 1977, 1978, 2081, 2082, 2133, 2134, 2185, + 2186, 2237, 2289, 2290, 2341, 2342, 2393, 2394, 2496 }, + { 1, 53, 105, 209, 261, 313, 417, 469, 521, 625, 677, 729, 833, 834, + 937, 938, 1041, 1042, 1249, 1250, 1353, 1354, 1457, 1458, 1665, 1666, + 1769, 1770, 1873, 1874, 2081, 2082, 2185, 2289, 2290, 2496, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 28, 55, 82, 109, 136, 163, 217, 244, 271, 298, 325, 352, 379, 433, + 460, 487, 514, 541, 568, 595, 649, 676, 703, 730, 757, 784, 811, 865, + 866, 919, 920, 973, 974, 1027, 1028, 1081, 1082, 1135, 1136, 1189, + 1190, 1297, 1298, 1351, 1352, 1405, 1406, 1459, 1460, 1513, 1514, + 1567, 1568, 1621, 1622, 1729, 1730, 1783, 1784, 1837, 1838, 1891, + 1892, 1945, 1946, 1999, 2000, 2053, 2054, 2161, 2162, 2215, 2216, + 2269, 2270, 2323, 2377, 2378, 2431, 2432, 2485, 2486, 2592 }, + { 1, 55, 109, 217, 271, 325, 433, 487, 541, 649, 703, 757, 865, 866, + 973, 974, 1081, 1082, 1297, 1298, 1405, 1406, 1513, 1514, 1729, 1730, + 1837, 1838, 1945, 1946, 2161, 2162, 2269, 2377, 2378, 2592, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, + { 1, 29, 57, 85, 113, 141, 169, 225, 253, 281, 309, 337, 365, 393, 449, + 477, 505, 533, 561, 589, 617, 673, 701, 729, 757, 785, 813, 841, 897, + 898, 953, 954, 1009, 1010, 1065, 1066, 1121, 1122, 1177, 1178, 1233, + 1234, 1345, 1346, 1401, 1402, 1457, 1458, 1513, 1514, 1569, 1570, + 1625, 1626, 1681, 1682, 1793, 1794, 1849, 1850, 1905, 1906, 1961, + 1962, 2017, 2018, 2073, 2074, 2129, 2130, 2241, 2242, 2297, 2298, + 2353, 2354, 2409, 2465, 2466, 2521, 2522, 2577, 2578, 2688 }, + { 1, 57, 113, 225, 281, 337, 449, 505, 561, 673, 729, 785, 897, 898, + 1009, 1010, 1121, 1122, 1345, 1346, 1457, 1458, 1569, 1570, 1793, + 1794, 1905, 1906, 2017, 2018, 2241, 2242, 2353, 2465, 2466, 2688, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0 }, + { 1, 30, 59, 88, 117, 146, 175, 233, 262, 291, 320, 349, 378, 407, 465, + 494, 523, 552, 581, 610, 639, 697, 726, 755, 784, 813, 842, 871, 929, + 930, 987, 988, 1045, 1046, 1103, 1104, 1161, 1162, 1219, 1220, 1277, + 1278, 1393, 1394, 1451, 1452, 1509, 1510, 1567, 1568, 1625, 1626, + 1683, 1684, 1741, 1742, 1857, 1858, 1915, 1916, 1973, 1974, 2031, + 2032, 2089, 2090, 2147, 2148, 2205, 2206, 2321, 2322, 2379, 2380, + 2437, 2438, 2495, 2553, 2554, 2611, 2612, 2669, 2670, 2784 }, + { 1, 59, 117, 233, 291, 349, 465, 523, 581, 697, 755, 813, 929, 930, + 1045, 1046, 1161, 1162, 1393, 1394, 1509, 1510, 1625, 1626, 1857, + 1858, 1973, 1974, 2089, 2090, 2321, 2322, 2437, 2553, 2554, 2784, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0 }, + { 1, 31, 61, 91, 121, 151, 181, 241, 271, 301, 331, 361, 391, 421, 481, + 511, 541, 571, 601, 631, 661, 721, 751, 781, 811, 841, 871, 901, 961, + 962, 1021, 1022, 1081, 1082, 1141, 1142, 1201, 1202, 1261, 1262, 1321, + 1322, 1441, 1442, 1501, 1502, 1561, 1562, 1621, 1622, 1681, 1682, + 1741, 1742, 1801, 1802, 1921, 1922, 1981, 1982, 2041, 2042, 2101, + 2102, 2161, 2162, 2221, 2222, 2281, 2282, 2401, 2402, 2461, 2462, + 2521, 2522, 2581, 2641, 2642, 2701, 2702, 2761, 2762, 2880 }, + { 1, 61, 121, 241, 301, 361, 481, 541, 601, 721, 781, 841, 961, 962, + 1081, 1082, 1201, 1202, 1441, 1442, 1561, 1562, 1681, 1682, 1921, + 1922, 2041, 2042, 2161, 2162, 2401, 2402, 2521, 2641, 2642, 2880, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0 }, + { 1, 32, 63, 94, 125, 156, 187, 249, 280, 311, 342, 373, 404, 435, 497, + 528, 559, 590, 621, 652, 683, 745, 776, 807, 838, 869, 900, 931, 993, + 994, 1055, 1056, 1117, 1118, 1179, 1180, 1241, 1242, 1303, 1304, 1365, + 1366, 1489, 1490, 1551, 1552, 1613, 1614, 1675, 1676, 1737, 1738, + 1799, 1800, 1861, 1862, 1985, 1986, 2047, 2048, 2109, 2110, 2171, + 2172, 2233, 2234, 2295, 2296, 2357, 2358, 2481, 2482, 2543, 2544, + 2605, 2606, 2667, 2729, 2730, 2791, 2792, 2853, 2854, 2976 }, + { 1, 63, 125, 249, 311, 373, 497, 559, 621, 745, 807, 869, 993, 994, + 1117, 1118, 1241, 1242, 1489, 1490, 1613, 1614, 1737, 1738, 1985, + 1986, 2109, 2110, 2233, 2234, 2481, 2482, 2605, 2729, 2730, 2976, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0 }, + { 1, 33, 65, 97, 129, 161, 193, 257, 289, 321, 353, 385, 417, 449, 513, + 545, 577, 609, 641, 673, 705, 769, 801, 833, 865, 897, 929, 961, 1025, + 1026, 1089, 1090, 1153, 1154, 1217, 1218, 1281, 1282, 1345, 1346, + 1409, 1410, 1537, 1538, 1601, 1602, 1665, 1666, 1729, 1730, 1793, + 1794, 1857, 1858, 1921, 1922, 2049, 2050, 2113, 2114, 2177, 2178, + 2241, 2242, 2305, 2306, 2369, 2370, 2433, 2434, 2561, 2562, 2625, + 2626, 2689, 2690, 2753, 2817, 2818, 2881, 2882, 2945, 2946, 3072 }, + { 1, 65, 129, 257, 321, 385, 513, 577, 641, 769, 833, 897, 1025, 1026, + 1153, 1154, 1281, 1282, 1537, 1538, 1665, 1666, 1793, 1794, 2049, + 2050, 2177, 2178, 2305, 2306, 2561, 2562, 2689, 2817, 2818, 3072, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0 }, + { 1, 34, 67, 100, 133, 166, 199, 265, 298, 331, 364, 397, 430, 463, 529, + 562, 595, 628, 661, 694, 727, 793, 826, 859, 892, 925, 958, 991, 1057, + 1058, 1123, 1124, 1189, 1190, 1255, 1256, 1321, 1322, 1387, 1388, + 1453, 1454, 1585, 1586, 1651, 1652, 1717, 1718, 1783, 1784, 1849, + 1850, 1915, 1916, 1981, 1982, 2113, 2114, 2179, 2180, 2245, 2246, + 2311, 2312, 2377, 2378, 2443, 2444, 2509, 2510, 2641, 2642, 2707, + 2708, 2773, 2774, 2839, 2905, 2906, 2971, 2972, 3037, 3038, 3168 }, + { 1, 35, 69, 103, 137, 171, 205, 273, 307, 341, 375, 409, 443, 477, 545, + 579, 613, 647, 681, 715, 749, 817, 851, 885, 919, 953, 987, 1021, + 1089, 1090, 1157, 1158, 1225, 1226, 1293, 1294, 1361, 1362, 1429, + 1430, 1497, 1498, 1633, 1634, 1701, 1702, 1769, 1770, 1837, 1838, + 1905, 1906, 1973, 1974, 2041, 2042, 2177, 2178, 2245, 2246, 2313, + 2314, 2381, 2382, 2449, 2450, 2517, 2518, 2585, 2586, 2721, 2722, + 2789, 2790, 2857, 2858, 2925, 2993, 2994, 3061, 3062, 3129, 3130, + 3264 }, + { 1, 36, 71, 106, 141, 176, 211, 281, 316, 351, 386, 421, 456, 491, 561, + 596, 631, 666, 701, 736, 771, 841, 876, 911, 946, 981, 1016, 1051, + 1121, 1122, 1191, 1192, 1261, 1262, 1331, 1332, 1401, 1402, 1471, + 1472, 1541, 1542, 1681, 1682, 1751, 1752, 1821, 1822, 1891, 1892, + 1961, 1962, 2031, 2032, 2101, 2102, 2241, 2242, 2311, 2312, 2381, + 2382, 2451, 2452, 2521, 2522, 2591, 2592, 2661, 2662, 2801, 2802, + 2871, 2872, 2941, 2942, 3011, 3081, 3082, 3151, 3152, 3221, 3222, + 3360 }, + { 1, 37, 73, 109, 145, 181, 217, 289, 325, 361, 397, 433, 469, 505, 577, + 613, 649, 685, 721, 757, 793, 865, 901, 937, 973, 1009, 1045, 1081, + 1153, 1154, 1225, 1226, 1297, 1298, 1369, 1370, 1441, 1442, 1513, + 1514, 1585, 1586, 1729, 1730, 1801, 1802, 1873, 1874, 1945, 1946, + 2017, 2018, 2089, 2090, 2161, 2162, 2305, 2306, 2377, 2378, 2449, + 2450, 2521, 2522, 2593, 2594, 2665, 2666, 2737, 2738, 2881, 2882, + 2953, 2954, 3025, 3026, 3097, 3169, 3170, 3241, 3242, 3313, 3314, + 3456 }, + { 1, 38, 75, 112, 149, 186, 223, 297, 334, 371, 408, 445, 482, 519, 593, + 630, 667, 704, 741, 778, 815, 889, 926, 963, 1000, 1037, 1074, 1111, + 1185, 1186, 1259, 1260, 1333, 1334, 1407, 1408, 1481, 1482, 1555, + 1556, 1629, 1630, 1777, 1778, 1851, 1852, 1925, 1926, 1999, 2000, + 2073, 2074, 2147, 2148, 2221, 2222, 2369, 2370, 2443, 2444, 2517, + 2518, 2591, 2592, 2665, 2666, 2739, 2740, 2813, 2814, 2961, 2962, + 3035, 3036, 3109, 3110, 3183, 3257, 3258, 3331, 3332, 3405, 3406, + 3552 }, + { 1, 39, 77, 115, 153, 191, 229, 305, 343, 381, 419, 457, 495, 533, 609, + 647, 685, 723, 761, 799, 837, 913, 951, 989, 1027, 1065, 1103, 1141, + 1217, 1218, 1293, 1294, 1369, 1370, 1445, 1446, 1521, 1522, 1597, + 1598, 1673, 1674, 1825, 1826, 1901, 1902, 1977, 1978, 2053, 2054, + 2129, 2130, 2205, 2206, 2281, 2282, 2433, 2434, 2509, 2510, 2585, + 2586, 2661, 2662, 2737, 2738, 2813, 2814, 2889, 2890, 3041, 3042, + 3117, 3118, 3193, 3194, 3269, 3345, 3346, 3421, 3422, 3497, 3498, + 3648 }, + { 1, 40, 79, 118, 157, 196, 235, 313, 352, 391, 430, 469, 508, 547, 625, + 664, 703, 742, 781, 820, 859, 937, 976, 1015, 1054, 1093, 1132, 1171, + 1249, 1250, 1327, 1328, 1405, 1406, 1483, 1484, 1561, 1562, 1639, + 1640, 1717, 1718, 1873, 1874, 1951, 1952, 2029, 2030, 2107, 2108, + 2185, 2186, 2263, 2264, 2341, 2342, 2497, 2498, 2575, 2576, 2653, + 2654, 2731, 2732, 2809, 2810, 2887, 2888, 2965, 2966, 3121, 3122, + 3199, 3200, 3277, 3278, 3355, 3433, 3434, 3511, 3512, 3589, 3590, + 3744 }, + { 1, 41, 81, 121, 161, 201, 241, 321, 361, 401, 441, 481, 521, 561, 641, + 681, 721, 761, 801, 841, 881, 961, 1001, 1041, 1081, 1121, 1161, 1201, + 1281, 1282, 1361, 1362, 1441, 1442, 1521, 1522, 1601, 1602, 1681, + 1682, 1761, 1762, 1921, 1922, 2001, 2002, 2081, 2082, 2161, 2162, + 2241, 2242, 2321, 2322, 2401, 2402, 2561, 2562, 2641, 2642, 2721, + 2722, 2801, 2802, 2881, 2882, 2961, 2962, 3041, 3042, 3201, 3202, + 3281, 3282, 3361, 3362, 3441, 3521, 3522, 3601, 3602, 3681, 3682, + 3840 }, + { 1, 42, 83, 124, 165, 206, 247, 329, 370, 411, 452, 493, 534, 575, 657, + 698, 739, 780, 821, 862, 903, 985, 1026, 1067, 1108, 1149, 1190, 1231, + 1313, 1314, 1395, 1396, 1477, 1478, 1559, 1560, 1641, 1642, 1723, + 1724, 1805, 1806, 1969, 1970, 2051, 2052, 2133, 2134, 2215, 2216, + 2297, 2298, 2379, 2380, 2461, 2462, 2625, 2626, 2707, 2708, 2789, + 2790, 2871, 2872, 2953, 2954, 3035, 3036, 3117, 3118, 3281, 3282, + 3363, 3364, 3445, 3446, 3527, 3609, 3610, 3691, 3692, 3773, 3774, + 3936 }, + { 1, 43, 85, 127, 169, 211, 253, 337, 379, 421, 463, 505, 547, 589, 673, + 715, 757, 799, 841, 883, 925, 1009, 1051, 1093, 1135, 1177, 1219, + 1261, 1345, 1346, 1429, 1430, 1513, 1514, 1597, 1598, 1681, 1682, + 1765, 1766, 1849, 1850, 2017, 2018, 2101, 2102, 2185, 2186, 2269, + 2270, 2353, 2354, 2437, 2438, 2521, 2522, 2689, 2690, 2773, 2774, + 2857, 2858, 2941, 2942, 3025, 3026, 3109, 3110, 3193, 3194, 3361, + 3362, 3445, 3446, 3529, 3530, 3613, 3697, 3698, 3781, 3782, 3865, + 3866, 4032 }, + { 1, 44, 87, 130, 173, 216, 259, 345, 388, 431, 474, 517, 560, 603, 689, + 732, 775, 818, 861, 904, 947, 1033, 1076, 1119, 1162, 1205, 1248, + 1291, 1377, 1378, 1463, 1464, 1549, 1550, 1635, 1636, 1721, 1722, + 1807, 1808, 1893, 1894, 2065, 2066, 2151, 2152, 2237, 2238, 2323, + 2324, 2409, 2410, 2495, 2496, 2581, 2582, 2753, 2754, 2839, 2840, + 2925, 2926, 3011, 3012, 3097, 3098, 3183, 3184, 3269, 3270, 3441, + 3442, 3527, 3528, 3613, 3614, 3699, 3785, 3786, 3871, 3872, 3957, + 3958, 4128 }, + { 1, 45, 89, 133, 177, 221, 265, 353, 397, 441, 485, 529, 573, 617, 705, + 749, 793, 837, 881, 925, 969, 1057, 1101, 1145, 1189, 1233, 1277, + 1321, 1409, 1410, 1497, 1498, 1585, 1586, 1673, 1674, 1761, 1762, + 1849, 1850, 1937, 1938, 2113, 2114, 2201, 2202, 2289, 2290, 2377, + 2378, 2465, 2466, 2553, 2554, 2641, 2642, 2817, 2818, 2905, 2906, + 2993, 2994, 3081, 3082, 3169, 3170, 3257, 3258, 3345, 3346, 3521, + 3522, 3609, 3610, 3697, 3698, 3785, 3873, 3874, 3961, 3962, 4049, + 4050, 4224 }, + { 1, 46, 91, 136, 181, 226, 271, 361, 406, 451, 496, 541, 586, 631, 721, + 766, 811, 856, 901, 946, 991, 1081, 1126, 1171, 1216, 1261, 1306, + 1351, 1441, 1442, 1531, 1532, 1621, 1622, 1711, 1712, 1801, 1802, + 1891, 1892, 1981, 1982, 2161, 2162, 2251, 2252, 2341, 2342, 2431, + 2432, 2521, 2522, 2611, 2612, 2701, 2702, 2881, 2882, 2971, 2972, + 3061, 3062, 3151, 3152, 3241, 3242, 3331, 3332, 3421, 3422, 3601, + 3602, 3691, 3692, 3781, 3782, 3871, 3961, 3962, 4051, 4052, 4141, + 4142, 4320 }, + { 1, 47, 93, 139, 185, 231, 277, 369, 415, 461, 507, 553, 599, 645, 737, + 783, 829, 875, 921, 967, 1013, 1105, 1151, 1197, 1243, 1289, 1335, + 1381, 1473, 1474, 1565, 1566, 1657, 1658, 1749, 1750, 1841, 1842, + 1933, 1934, 2025, 2026, 2209, 2210, 2301, 2302, 2393, 2394, 2485, + 2486, 2577, 2578, 2669, 2670, 2761, 2762, 2945, 2946, 3037, 3038, + 3129, 3130, 3221, 3222, 3313, 3314, 3405, 3406, 3497, 3498, 3681, + 3682, 3773, 3774, 3865, 3866, 3957, 4049, 4050, 4141, 4142, 4233, + 4234, 4416 }, + { 1, 48, 95, 142, 189, 236, 283, 377, 424, 471, 518, 565, 612, 659, 753, + 800, 847, 894, 941, 988, 1035, 1129, 1176, 1223, 1270, 1317, 1364, + 1411, 1505, 1506, 1599, 1600, 1693, 1694, 1787, 1788, 1881, 1882, + 1975, 1976, 2069, 2070, 2257, 2258, 2351, 2352, 2445, 2446, 2539, + 2540, 2633, 2634, 2727, 2728, 2821, 2822, 3009, 3010, 3103, 3104, + 3197, 3198, 3291, 3292, 3385, 3386, 3479, 3480, 3573, 3574, 3761, + 3762, 3855, 3856, 3949, 3950, 4043, 4137, 4138, 4231, 4232, 4325, + 4326, 4512 }, + { 1, 49, 97, 145, 193, 241, 289, 385, 433, 481, 529, 577, 625, 673, 769, + 817, 865, 913, 961, 1009, 1057, 1153, 1201, 1249, 1297, 1345, 1393, + 1441, 1537, 1538, 1633, 1634, 1729, 1730, 1825, 1826, 1921, 1922, + 2017, 2018, 2113, 2114, 2305, 2306, 2401, 2402, 2497, 2498, 2593, + 2594, 2689, 2690, 2785, 2786, 2881, 2882, 3073, 3074, 3169, 3170, + 3265, 3266, 3361, 3362, 3457, 3458, 3553, 3554, 3649, 3650, 3841, + 3842, 3937, 3938, 4033, 4034, 4129, 4225, 4226, 4321, 4322, 4417, + 4418, 4608 }, + { 1, 50, 99, 148, 197, 246, 295, 393, 442, 491, 540, 589, 638, 687, 785, + 834, 883, 932, 981, 1030, 1079, 1177, 1226, 1275, 1324, 1373, 1422, + 1471, 1569, 1570, 1667, 1668, 1765, 1766, 1863, 1864, 1961, 1962, + 2059, 2060, 2157, 2158, 2353, 2354, 2451, 2452, 2549, 2550, 2647, + 2648, 2745, 2746, 2843, 2844, 2941, 2942, 3137, 3138, 3235, 3236, + 3333, 3334, 3431, 3432, 3529, 3530, 3627, 3628, 3725, 3726, 3921, + 3922, 4019, 4020, 4117, 4118, 4215, 4313, 4314, 4411, 4412, 4509, + 4510, 4704 }, + { 1, 51, 101, 151, 201, 251, 301, 401, 451, 501, 551, 601, 651, 701, + 801, 851, 901, 951, 1001, 1051, 1101, 1201, 1251, 1301, 1351, 1401, + 1451, 1501, 1601, 1602, 1701, 1702, 1801, 1802, 1901, 1902, 2001, + 2002, 2101, 2102, 2201, 2202, 2401, 2402, 2501, 2502, 2601, 2602, + 2701, 2702, 2801, 2802, 2901, 2902, 3001, 3002, 3201, 3202, 3301, + 3302, 3401, 3402, 3501, 3502, 3601, 3602, 3701, 3702, 3801, 3802, + 4001, 4002, 4101, 4102, 4201, 4202, 4301, 4401, 4402, 4501, 4502, + 4601, 4602, 4800 }, + { 1, 52, 103, 154, 205, 256, 307, 409, 460, 511, 562, 613, 664, 715, + 817, 868, 919, 970, 1021, 1072, 1123, 1225, 1276, 1327, 1378, 1429, + 1480, 1531, 1633, 1634, 1735, 1736, 1837, 1838, 1939, 1940, 2041, + 2042, 2143, 2144, 2245, 2246, 2449, 2450, 2551, 2552, 2653, 2654, + 2755, 2756, 2857, 2858, 2959, 2960, 3061, 3062, 3265, 3266, 3367, + 3368, 3469, 3470, 3571, 3572, 3673, 3674, 3775, 3776, 3877, 3878, + 4081, 4082, 4183, 4184, 4285, 4286, 4387, 4489, 4490, 4591, 4592, + 4693, 4694, 4896 }, + { 1, 53, 105, 157, 209, 261, 313, 417, 469, 521, 573, 625, 677, 729, + 833, 885, 937, 989, 1041, 1093, 1145, 1249, 1301, 1353, 1405, 1457, + 1509, 1561, 1665, 1666, 1769, 1770, 1873, 1874, 1977, 1978, 2081, + 2082, 2185, 2186, 2289, 2290, 2497, 2498, 2601, 2602, 2705, 2706, + 2809, 2810, 2913, 2914, 3017, 3018, 3121, 3122, 3329, 3330, 3433, + 3434, 3537, 3538, 3641, 3642, 3745, 3746, 3849, 3850, 3953, 3954, + 4161, 4162, 4265, 4266, 4369, 4370, 4473, 4577, 4578, 4681, 4682, + 4785, 4786, 4992 }, + { 1, 54, 107, 160, 213, 266, 319, 425, 478, 531, 584, 637, 690, 743, + 849, 902, 955, 1008, 1061, 1114, 1167, 1273, 1326, 1379, 1432, 1485, + 1538, 1591, 1697, 1698, 1803, 1804, 1909, 1910, 2015, 2016, 2121, + 2122, 2227, 2228, 2333, 2334, 2545, 2546, 2651, 2652, 2757, 2758, + 2863, 2864, 2969, 2970, 3075, 3076, 3181, 3182, 3393, 3394, 3499, + 3500, 3605, 3606, 3711, 3712, 3817, 3818, 3923, 3924, 4029, 4030, + 4241, 4242, 4347, 4348, 4453, 4454, 4559, 4665, 4666, 4771, 4772, + 4877, 4878, 5088 }, + { 1, 55, 109, 163, 217, 271, 325, 433, 487, 541, 595, 649, 703, 757, + 865, 919, 973, 1027, 1081, 1135, 1189, 1297, 1351, 1405, 1459, 1513, + 1567, 1621, 1729, 1730, 1837, 1838, 1945, 1946, 2053, 2054, 2161, + 2162, 2269, 2270, 2377, 2378, 2593, 2594, 2701, 2702, 2809, 2810, + 2917, 2918, 3025, 3026, 3133, 3134, 3241, 3242, 3457, 3458, 3565, + 3566, 3673, 3674, 3781, 3782, 3889, 3890, 3997, 3998, 4105, 4106, + 4321, 4322, 4429, 4430, 4537, 4538, 4645, 4753, 4754, 4861, 4862, + 4969, 4970, 5184 }, + { 1, 56, 111, 166, 221, 276, 331, 441, 496, 551, 606, 661, 716, 771, + 881, 936, 991, 1046, 1101, 1156, 1211, 1321, 1376, 1431, 1486, 1541, + 1596, 1651, 1761, 1762, 1871, 1872, 1981, 1982, 2091, 2092, 2201, + 2202, 2311, 2312, 2421, 2422, 2641, 2642, 2751, 2752, 2861, 2862, + 2971, 2972, 3081, 3082, 3191, 3192, 3301, 3302, 3521, 3522, 3631, + 3632, 3741, 3742, 3851, 3852, 3961, 3962, 4071, 4072, 4181, 4182, + 4401, 4402, 4511, 4512, 4621, 4622, 4731, 4841, 4842, 4951, 4952, + 5061, 5062, 5280 }, + { 1, 57, 113, 169, 225, 281, 337, 449, 505, 561, 617, 673, 729, 785, + 897, 953, 1009, 1065, 1121, 1177, 1233, 1345, 1401, 1457, 1513, 1569, + 1625, 1681, 1793, 1794, 1905, 1906, 2017, 2018, 2129, 2130, 2241, + 2242, 2353, 2354, 2465, 2466, 2689, 2690, 2801, 2802, 2913, 2914, + 3025, 3026, 3137, 3138, 3249, 3250, 3361, 3362, 3585, 3586, 3697, + 3698, 3809, 3810, 3921, 3922, 4033, 4034, 4145, 4146, 4257, 4258, + 4481, 4482, 4593, 4594, 4705, 4706, 4817, 4929, 4930, 5041, 5042, + 5153, 5154, 5376 }, + { 1, 58, 115, 172, 229, 286, 343, 457, 514, 571, 628, 685, 742, 799, + 913, 970, 1027, 1084, 1141, 1198, 1255, 1369, 1426, 1483, 1540, 1597, + 1654, 1711, 1825, 1826, 1939, 1940, 2053, 2054, 2167, 2168, 2281, + 2282, 2395, 2396, 2509, 2510, 2737, 2738, 2851, 2852, 2965, 2966, + 3079, 3080, 3193, 3194, 3307, 3308, 3421, 3422, 3649, 3650, 3763, + 3764, 3877, 3878, 3991, 3992, 4105, 4106, 4219, 4220, 4333, 4334, + 4561, 4562, 4675, 4676, 4789, 4790, 4903, 5017, 5018, 5131, 5132, + 5245, 5246, 5472 }, + { 1, 59, 117, 175, 233, 291, 349, 465, 523, 581, 639, 697, 755, 813, + 929, 987, 1045, 1103, 1161, 1219, 1277, 1393, 1451, 1509, 1567, 1625, + 1683, 1741, 1857, 1858, 1973, 1974, 2089, 2090, 2205, 2206, 2321, + 2322, 2437, 2438, 2553, 2554, 2785, 2786, 2901, 2902, 3017, 3018, + 3133, 3134, 3249, 3250, 3365, 3366, 3481, 3482, 3713, 3714, 3829, + 3830, 3945, 3946, 4061, 4062, 4177, 4178, 4293, 4294, 4409, 4410, + 4641, 4642, 4757, 4758, 4873, 4874, 4989, 5105, 5106, 5221, 5222, + 5337, 5338, 5568 }, + { 1, 60, 119, 178, 237, 296, 355, 473, 532, 591, 650, 709, 768, 827, + 945, 1004, 1063, 1122, 1181, 1240, 1299, 1417, 1476, 1535, 1594, 1653, + 1712, 1771, 1889, 1890, 2007, 2008, 2125, 2126, 2243, 2244, 2361, + 2362, 2479, 2480, 2597, 2598, 2833, 2834, 2951, 2952, 3069, 3070, + 3187, 3188, 3305, 3306, 3423, 3424, 3541, 3542, 3777, 3778, 3895, + 3896, 4013, 4014, 4131, 4132, 4249, 4250, 4367, 4368, 4485, 4486, + 4721, 4722, 4839, 4840, 4957, 4958, 5075, 5193, 5194, 5311, 5312, + 5429, 5430, 5664 }, + { 1, 61, 121, 181, 241, 301, 361, 481, 541, 601, 661, 721, 781, 841, + 961, 1021, 1081, 1141, 1201, 1261, 1321, 1441, 1501, 1561, 1621, 1681, + 1741, 1801, 1921, 1922, 2041, 2042, 2161, 2162, 2281, 2282, 2401, + 2402, 2521, 2522, 2641, 2642, 2881, 2882, 3001, 3002, 3121, 3122, + 3241, 3242, 3361, 3362, 3481, 3482, 3601, 3602, 3841, 3842, 3961, + 3962, 4081, 4082, 4201, 4202, 4321, 4322, 4441, 4442, 4561, 4562, + 4801, 4802, 4921, 4922, 5041, 5042, 5161, 5281, 5282, 5401, 5402, + 5521, 5522, 5760 }, + { 1, 62, 123, 184, 245, 306, 367, 489, 550, 611, 672, 733, 794, 855, + 977, 1038, 1099, 1160, 1221, 1282, 1343, 1465, 1526, 1587, 1648, 1709, + 1770, 1831, 1953, 1954, 2075, 2076, 2197, 2198, 2319, 2320, 2441, + 2442, 2563, 2564, 2685, 2686, 2929, 2930, 3051, 3052, 3173, 3174, + 3295, 3296, 3417, 3418, 3539, 3540, 3661, 3662, 3905, 3906, 4027, + 4028, 4149, 4150, 4271, 4272, 4393, 4394, 4515, 4516, 4637, 4638, + 4881, 4882, 5003, 5004, 5125, 5126, 5247, 5369, 5370, 5491, 5492, + 5613, 5614, 5856 }, + { 1, 63, 125, 187, 249, 311, 373, 497, 559, 621, 683, 745, 807, 869, + 993, 1055, 1117, 1179, 1241, 1303, 1365, 1489, 1551, 1613, 1675, 1737, + 1799, 1861, 1985, 1986, 2109, 2110, 2233, 2234, 2357, 2358, 2481, + 2482, 2605, 2606, 2729, 2730, 2977, 2978, 3101, 3102, 3225, 3226, + 3349, 3350, 3473, 3474, 3597, 3598, 3721, 3722, 3969, 3970, 4093, + 4094, 4217, 4218, 4341, 4342, 4465, 4466, 4589, 4590, 4713, 4714, + 4961, 4962, 5085, 5086, 5209, 5210, 5333, 5457, 5458, 5581, 5582, + 5705, 5706, 5952 }, + { 1, 64, 127, 190, 253, 316, 379, 505, 568, 631, 694, 757, 820, 883, + 1009, 1072, 1135, 1198, 1261, 1324, 1387, 1513, 1576, 1639, 1702, + 1765, 1828, 1891, 2017, 2018, 2143, 2144, 2269, 2270, 2395, 2396, + 2521, 2522, 2647, 2648, 2773, 2774, 3025, 3026, 3151, 3152, 3277, + 3278, 3403, 3404, 3529, 3530, 3655, 3656, 3781, 3782, 4033, 4034, + 4159, 4160, 4285, 4286, 4411, 4412, 4537, 4538, 4663, 4664, 4789, + 4790, 5041, 5042, 5167, 5168, 5293, 5294, 5419, 5545, 5546, 5671, + 5672, 5797, 5798, 6048 }, + { 1, 65, 129, 193, 257, 321, 385, 513, 577, 641, 705, 769, 833, 897, + 1025, 1089, 1153, 1217, 1281, 1345, 1409, 1537, 1601, 1665, 1729, + 1793, 1857, 1921, 2049, 2050, 2177, 2178, 2305, 2306, 2433, 2434, + 2561, 2562, 2689, 2690, 2817, 2818, 3073, 3074, 3201, 3202, 3329, + 3330, 3457, 3458, 3585, 3586, 3713, 3714, 3841, 3842, 4097, 4098, + 4225, 4226, 4353, 4354, 4481, 4482, 4609, 4610, 4737, 4738, 4865, + 4866, 5121, 5122, 5249, 5250, 5377, 5378, 5505, 5633, 5634, 5761, + 5762, 5889, 5890, 6144 }, + { 1, 66, 131, 196, 261, 326, 391, 521, 586, 651, 716, 781, 846, 911, + 1041, 1106, 1171, 1236, 1301, 1366, 1431, 1561, 1626, 1691, 1756, + 1821, 1886, 1951, 2081, 2082, 2211, 2212, 2341, 2342, 2471, 2472, + 2601, 2602, 2731, 2732, 2861, 2862, 3121, 3122, 3251, 3252, 3381, + 3382, 3511, 3512, 3641, 3642, 3771, 3772, 3901, 3902, 4161, 4162, + 4291, 4292, 4421, 4422, 4551, 4552, 4681, 4682, 4811, 4812, 4941, + 4942, 5201, 5202, 5331, 5332, 5461, 5462, 5591, 5721, 5722, 5851, + 5852, 5981, 5982, 6240 }, + { 1, 68, 135, 202, 269, 336, 403, 537, 604, 671, 738, 805, 872, 939, + 1073, 1140, 1207, 1274, 1341, 1408, 1475, 1609, 1676, 1743, 1810, + 1877, 1944, 2011, 2145, 2146, 2279, 2280, 2413, 2414, 2547, 2548, + 2681, 2682, 2815, 2816, 2949, 2950, 3217, 3218, 3351, 3352, 3485, + 3486, 3619, 3620, 3753, 3754, 3887, 3888, 4021, 4022, 4289, 4290, + 4423, 4424, 4557, 4558, 4691, 4692, 4825, 4826, 4959, 4960, 5093, + 5094, 5361, 5362, 5495, 5496, 5629, 5630, 5763, 5897, 5898, 6031, + 6032, 6165, 6166, 6432 }, + { 1, 70, 139, 208, 277, 346, 415, 553, 622, 691, 760, 829, 898, 967, + 1105, 1174, 1243, 1312, 1381, 1450, 1519, 1657, 1726, 1795, 1864, + 1933, 2002, 2071, 2209, 2210, 2347, 2348, 2485, 2486, 2623, 2624, + 2761, 2762, 2899, 2900, 3037, 3038, 3313, 3314, 3451, 3452, 3589, + 3590, 3727, 3728, 3865, 3866, 4003, 4004, 4141, 4142, 4417, 4418, + 4555, 4556, 4693, 4694, 4831, 4832, 4969, 4970, 5107, 5108, 5245, + 5246, 5521, 5522, 5659, 5660, 5797, 5798, 5935, 6073, 6074, 6211, + 6212, 6349, 6350, 6624 }, + { 1, 72, 143, 214, 285, 356, 427, 569, 640, 711, 782, 853, 924, 995, + 1137, 1208, 1279, 1350, 1421, 1492, 1563, 1705, 1776, 1847, 1918, + 1989, 2060, 2131, 2273, 2274, 2415, 2416, 2557, 2558, 2699, 2700, + 2841, 2842, 2983, 2984, 3125, 3126, 3409, 3410, 3551, 3552, 3693, + 3694, 3835, 3836, 3977, 3978, 4119, 4120, 4261, 4262, 4545, 4546, + 4687, 4688, 4829, 4830, 4971, 4972, 5113, 5114, 5255, 5256, 5397, + 5398, 5681, 5682, 5823, 5824, 5965, 5966, 6107, 6249, 6250, 6391, + 6392, 6533, 6534, 6816 }, + { 1, 74, 147, 220, 293, 366, 439, 585, 658, 731, 804, 877, 950, 1023, + 1169, 1242, 1315, 1388, 1461, 1534, 1607, 1753, 1826, 1899, 1972, + 2045, 2118, 2191, 2337, 2338, 2483, 2484, 2629, 2630, 2775, 2776, + 2921, 2922, 3067, 3068, 3213, 3214, 3505, 3506, 3651, 3652, 3797, + 3798, 3943, 3944, 4089, 4090, 4235, 4236, 4381, 4382, 4673, 4674, + 4819, 4820, 4965, 4966, 5111, 5112, 5257, 5258, 5403, 5404, 5549, + 5550, 5841, 5842, 5987, 5988, 6133, 6134, 6279, 6425, 6426, 6571, + 6572, 6717, 6718, 7008 }, + { 1, 76, 151, 226, 301, 376, 451, 601, 676, 751, 826, 901, 976, 1051, + 1201, 1276, 1351, 1426, 1501, 1576, 1651, 1801, 1876, 1951, 2026, + 2101, 2176, 2251, 2401, 2402, 2551, 2552, 2701, 2702, 2851, 2852, + 3001, 3002, 3151, 3152, 3301, 3302, 3601, 3602, 3751, 3752, 3901, + 3902, 4051, 4052, 4201, 4202, 4351, 4352, 4501, 4502, 4801, 4802, + 4951, 4952, 5101, 5102, 5251, 5252, 5401, 5402, 5551, 5552, 5701, + 5702, 6001, 6002, 6151, 6152, 6301, 6302, 6451, 6601, 6602, 6751, + 6752, 6901, 6902, 7200 }, + { 1, 78, 155, 232, 309, 386, 463, 617, 694, 771, 848, 925, 1002, 1079, + 1233, 1310, 1387, 1464, 1541, 1618, 1695, 1849, 1926, 2003, 2080, + 2157, 2234, 2311, 2465, 2466, 2619, 2620, 2773, 2774, 2927, 2928, + 3081, 3082, 3235, 3236, 3389, 3390, 3697, 3698, 3851, 3852, 4005, + 4006, 4159, 4160, 4313, 4314, 4467, 4468, 4621, 4622, 4929, 4930, + 5083, 5084, 5237, 5238, 5391, 5392, 5545, 5546, 5699, 5700, 5853, + 5854, 6161, 6162, 6315, 6316, 6469, 6470, 6623, 6777, 6778, 6931, + 6932, 7085, 7086, 7392 }, + { 1, 80, 159, 238, 317, 396, 475, 633, 712, 791, 870, 949, 1028, 1107, + 1265, 1344, 1423, 1502, 1581, 1660, 1739, 1897, 1976, 2055, 2134, + 2213, 2292, 2371, 2529, 2530, 2687, 2688, 2845, 2846, 3003, 3004, + 3161, 3162, 3319, 3320, 3477, 3478, 3793, 3794, 3951, 3952, 4109, + 4110, 4267, 4268, 4425, 4426, 4583, 4584, 4741, 4742, 5057, 5058, + 5215, 5216, 5373, 5374, 5531, 5532, 5689, 5690, 5847, 5848, 6005, + 6006, 6321, 6322, 6479, 6480, 6637, 6638, 6795, 6953, 6954, 7111, + 7112, 7269, 7270, 7584 }, + { 1, 82, 163, 244, 325, 406, 487, 649, 730, 811, 892, 973, 1054, 1135, + 1297, 1378, 1459, 1540, 1621, 1702, 1783, 1945, 2026, 2107, 2188, + 2269, 2350, 2431, 2593, 2594, 2755, 2756, 2917, 2918, 3079, 3080, + 3241, 3242, 3403, 3404, 3565, 3566, 3889, 3890, 4051, 4052, 4213, + 4214, 4375, 4376, 4537, 4538, 4699, 4700, 4861, 4862, 5185, 5186, + 5347, 5348, 5509, 5510, 5671, 5672, 5833, 5834, 5995, 5996, 6157, + 6158, 6481, 6482, 6643, 6644, 6805, 6806, 6967, 7129, 7130, 7291, + 7292, 7453, 7454, 7776 }, + { 1, 84, 167, 250, 333, 416, 499, 665, 748, 831, 914, 997, 1080, 1163, + 1329, 1412, 1495, 1578, 1661, 1744, 1827, 1993, 2076, 2159, 2242, + 2325, 2408, 2491, 2657, 2658, 2823, 2824, 2989, 2990, 3155, 3156, + 3321, 3322, 3487, 3488, 3653, 3654, 3985, 3986, 4151, 4152, 4317, + 4318, 4483, 4484, 4649, 4650, 4815, 4816, 4981, 4982, 5313, 5314, + 5479, 5480, 5645, 5646, 5811, 5812, 5977, 5978, 6143, 6144, 6309, + 6310, 6641, 6642, 6807, 6808, 6973, 6974, 7139, 7305, 7306, 7471, + 7472, 7637, 7638, 7968 }, + { 1, 86, 171, 256, 341, 426, 511, 681, 766, 851, 936, 1021, 1106, 1191, + 1361, 1446, 1531, 1616, 1701, 1786, 1871, 2041, 2126, 2211, 2296, + 2381, 2466, 2551, 2721, 2722, 2891, 2892, 3061, 3062, 3231, 3232, + 3401, 3402, 3571, 3572, 3741, 3742, 4081, 4082, 4251, 4252, 4421, + 4422, 4591, 4592, 4761, 4762, 4931, 4932, 5101, 5102, 5441, 5442, + 5611, 5612, 5781, 5782, 5951, 5952, 6121, 6122, 6291, 6292, 6461, + 6462, 6801, 6802, 6971, 6972, 7141, 7142, 7311, 7481, 7482, 7651, + 7652, 7821, 7822, 8160 }, + { 1, 88, 175, 262, 349, 436, 523, 697, 784, 871, 958, 1045, 1132, 1219, + 1393, 1480, 1567, 1654, 1741, 1828, 1915, 2089, 2176, 2263, 2350, + 2437, 2524, 2611, 2785, 2786, 2959, 2960, 3133, 3134, 3307, 3308, + 3481, 3482, 3655, 3656, 3829, 3830, 4177, 4178, 4351, 4352, 4525, + 4526, 4699, 4700, 4873, 4874, 5047, 5048, 5221, 5222, 5569, 5570, + 5743, 5744, 5917, 5918, 6091, 6092, 6265, 6266, 6439, 6440, 6613, + 6614, 6961, 6962, 7135, 7136, 7309, 7310, 7483, 7657, 7658, 7831, + 7832, 8005, 8006, 8352 }, + { 1, 90, 179, 268, 357, 446, 535, 713, 802, 891, 980, 1069, 1158, 1247, + 1425, 1514, 1603, 1692, 1781, 1870, 1959, 2137, 2226, 2315, 2404, + 2493, 2582, 2671, 2849, 2850, 3027, 3028, 3205, 3206, 3383, 3384, + 3561, 3562, 3739, 3740, 3917, 3918, 4273, 4274, 4451, 4452, 4629, + 4630, 4807, 4808, 4985, 4986, 5163, 5164, 5341, 5342, 5697, 5698, + 5875, 5876, 6053, 6054, 6231, 6232, 6409, 6410, 6587, 6588, 6765, + 6766, 7121, 7122, 7299, 7300, 7477, 7478, 7655, 7833, 7834, 8011, + 8012, 8189, 8190, 8544 }, + { 1, 92, 183, 274, 365, 456, 547, 729, 820, 911, 1002, 1093, 1184, 1275, + 1457, 1548, 1639, 1730, 1821, 1912, 2003, 2185, 2276, 2367, 2458, + 2549, 2640, 2731, 2913, 2914, 3095, 3096, 3277, 3278, 3459, 3460, + 3641, 3642, 3823, 3824, 4005, 4006, 4369, 4370, 4551, 4552, 4733, + 4734, 4915, 4916, 5097, 5098, 5279, 5280, 5461, 5462, 5825, 5826, + 6007, 6008, 6189, 6190, 6371, 6372, 6553, 6554, 6735, 6736, 6917, + 6918, 7281, 7282, 7463, 7464, 7645, 7646, 7827, 8009, 8010, 8191, + 8192, 8373, 8374, 8736 }, + { 1, 94, 187, 280, 373, 466, 559, 745, 838, 931, 1024, 1117, 1210, 1303, + 1489, 1582, 1675, 1768, 1861, 1954, 2047, 2233, 2326, 2419, 2512, + 2605, 2698, 2791, 2977, 2978, 3163, 3164, 3349, 3350, 3535, 3536, + 3721, 3722, 3907, 3908, 4093, 4094, 4465, 4466, 4651, 4652, 4837, + 4838, 5023, 5024, 5209, 5210, 5395, 5396, 5581, 5582, 5953, 5954, + 6139, 6140, 6325, 6326, 6511, 6512, 6697, 6698, 6883, 6884, 7069, + 7070, 7441, 7442, 7627, 7628, 7813, 7814, 7999, 8185, 8186, 8371, + 8372, 8557, 8558, 8928 }, + { 1, 96, 191, 286, 381, 476, 571, 761, 856, 951, 1046, 1141, 1236, 1331, + 1521, 1616, 1711, 1806, 1901, 1996, 2091, 2281, 2376, 2471, 2566, + 2661, 2756, 2851, 3041, 3042, 3231, 3232, 3421, 3422, 3611, 3612, + 3801, 3802, 3991, 3992, 4181, 4182, 4561, 4562, 4751, 4752, 4941, + 4942, 5131, 5132, 5321, 5322, 5511, 5512, 5701, 5702, 6081, 6082, + 6271, 6272, 6461, 6462, 6651, 6652, 6841, 6842, 7031, 7032, 7221, + 7222, 7601, 7602, 7791, 7792, 7981, 7982, 8171, 8361, 8362, 8551, + 8552, 8741, 8742, 9120 }, + { 1, 98, 195, 292, 389, 486, 583, 777, 874, 971, 1068, 1165, 1262, 1359, + 1553, 1650, 1747, 1844, 1941, 2038, 2135, 2329, 2426, 2523, 2620, + 2717, 2814, 2911, 3105, 3106, 3299, 3300, 3493, 3494, 3687, 3688, + 3881, 3882, 4075, 4076, 4269, 4270, 4657, 4658, 4851, 4852, 5045, + 5046, 5239, 5240, 5433, 5434, 5627, 5628, 5821, 5822, 6209, 6210, + 6403, 6404, 6597, 6598, 6791, 6792, 6985, 6986, 7179, 7180, 7373, + 7374, 7761, 7762, 7955, 7956, 8149, 8150, 8343, 8537, 8538, 8731, + 8732, 8925, 8926, 9312 }, + { 1, 100, 199, 298, 397, 496, 595, 793, 892, 991, 1090, 1189, 1288, + 1387, 1585, 1684, 1783, 1882, 1981, 2080, 2179, 2377, 2476, 2575, + 2674, 2773, 2872, 2971, 3169, 3170, 3367, 3368, 3565, 3566, 3763, + 3764, 3961, 3962, 4159, 4160, 4357, 4358, 4753, 4754, 4951, 4952, + 5149, 5150, 5347, 5348, 5545, 5546, 5743, 5744, 5941, 5942, 6337, + 6338, 6535, 6536, 6733, 6734, 6931, 6932, 7129, 7130, 7327, 7328, + 7525, 7526, 7921, 7922, 8119, 8120, 8317, 8318, 8515, 8713, 8714, + 8911, 8912, 9109, 9110, 9504 }, + { 1, 102, 203, 304, 405, 506, 607, 809, 910, 1011, 1112, 1213, 1314, + 1415, 1617, 1718, 1819, 1920, 2021, 2122, 2223, 2425, 2526, 2627, + 2728, 2829, 2930, 3031, 3233, 3234, 3435, 3436, 3637, 3638, 3839, + 3840, 4041, 4042, 4243, 4244, 4445, 4446, 4849, 4850, 5051, 5052, + 5253, 5254, 5455, 5456, 5657, 5658, 5859, 5860, 6061, 6062, 6465, + 6466, 6667, 6668, 6869, 6870, 7071, 7072, 7273, 7274, 7475, 7476, + 7677, 7678, 8081, 8082, 8283, 8284, 8485, 8486, 8687, 8889, 8890, + 9091, 9092, 9293, 9294, 9696 }, + { 1, 104, 207, 310, 413, 516, 619, 825, 928, 1031, 1134, 1237, 1340, + 1443, 1649, 1752, 1855, 1958, 2061, 2164, 2267, 2473, 2576, 2679, + 2782, 2885, 2988, 3091, 3297, 3298, 3503, 3504, 3709, 3710, 3915, + 3916, 4121, 4122, 4327, 4328, 4533, 4534, 4945, 4946, 5151, 5152, + 5357, 5358, 5563, 5564, 5769, 5770, 5975, 5976, 6181, 6182, 6593, + 6594, 6799, 6800, 7005, 7006, 7211, 7212, 7417, 7418, 7623, 7624, + 7829, 7830, 8241, 8242, 8447, 8448, 8653, 8654, 8859, 9065, 9066, + 9271, 9272, 9477, 9478, 9888 }, + { 1, 106, 211, 316, 421, 526, 631, 841, 946, 1051, 1156, 1261, 1366, + 1471, 1681, 1786, 1891, 1996, 2101, 2206, 2311, 2521, 2626, 2731, + 2836, 2941, 3046, 3151, 3361, 3362, 3571, 3572, 3781, 3782, 3991, + 3992, 4201, 4202, 4411, 4412, 4621, 4622, 5041, 5042, 5251, 5252, + 5461, 5462, 5671, 5672, 5881, 5882, 6091, 6092, 6301, 6302, 6721, + 6722, 6931, 6932, 7141, 7142, 7351, 7352, 7561, 7562, 7771, 7772, + 7981, 7982, 8401, 8402, 8611, 8612, 8821, 8822, 9031, 9241, 9242, + 9451, 9452, 9661, 9662, 10080 }, + { 1, 108, 215, 322, 429, 536, 643, 857, 964, 1071, 1178, 1285, 1392, + 1499, 1713, 1820, 1927, 2034, 2141, 2248, 2355, 2569, 2676, 2783, + 2890, 2997, 3104, 3211, 3425, 3426, 3639, 3640, 3853, 3854, 4067, + 4068, 4281, 4282, 4495, 4496, 4709, 4710, 5137, 5138, 5351, 5352, + 5565, 5566, 5779, 5780, 5993, 5994, 6207, 6208, 6421, 6422, 6849, + 6850, 7063, 7064, 7277, 7278, 7491, 7492, 7705, 7706, 7919, 7920, + 8133, 8134, 8561, 8562, 8775, 8776, 8989, 8990, 9203, 9417, 9418, + 9631, 9632, 9845, 9846, 10272 }, + { 1, 110, 219, 328, 437, 546, 655, 873, 982, 1091, 1200, 1309, 1418, + 1527, 1745, 1854, 1963, 2072, 2181, 2290, 2399, 2617, 2726, 2835, + 2944, 3053, 3162, 3271, 3489, 3490, 3707, 3708, 3925, 3926, 4143, + 4144, 4361, 4362, 4579, 4580, 4797, 4798, 5233, 5234, 5451, 5452, + 5669, 5670, 5887, 5888, 6105, 6106, 6323, 6324, 6541, 6542, 6977, + 6978, 7195, 7196, 7413, 7414, 7631, 7632, 7849, 7850, 8067, 8068, + 8285, 8286, 8721, 8722, 8939, 8940, 9157, 9158, 9375, 9593, 9594, + 9811, 9812, 10029, 10030, 10464 }, + { 1, 112, 223, 334, 445, 556, 667, 889, 1000, 1111, 1222, 1333, 1444, + 1555, 1777, 1888, 1999, 2110, 2221, 2332, 2443, 2665, 2776, 2887, + 2998, 3109, 3220, 3331, 3553, 3554, 3775, 3776, 3997, 3998, 4219, + 4220, 4441, 4442, 4663, 4664, 4885, 4886, 5329, 5330, 5551, 5552, + 5773, 5774, 5995, 5996, 6217, 6218, 6439, 6440, 6661, 6662, 7105, + 7106, 7327, 7328, 7549, 7550, 7771, 7772, 7993, 7994, 8215, 8216, + 8437, 8438, 8881, 8882, 9103, 9104, 9325, 9326, 9547, 9769, 9770, + 9991, 9992, 10213, 10214, 10656 }, + { 1, 114, 227, 340, 453, 566, 679, 905, 1018, 1131, 1244, 1357, 1470, + 1583, 1809, 1922, 2035, 2148, 2261, 2374, 2487, 2713, 2826, 2939, + 3052, 3165, 3278, 3391, 3617, 3618, 3843, 3844, 4069, 4070, 4295, + 4296, 4521, 4522, 4747, 4748, 4973, 4974, 5425, 5426, 5651, 5652, + 5877, 5878, 6103, 6104, 6329, 6330, 6555, 6556, 6781, 6782, 7233, + 7234, 7459, 7460, 7685, 7686, 7911, 7912, 8137, 8138, 8363, 8364, + 8589, 8590, 9041, 9042, 9267, 9268, 9493, 9494, 9719, 9945, 9946, + 10171, 10172, 10397, 10398, 10848 }, + { 1, 116, 231, 346, 461, 576, 691, 921, 1036, 1151, 1266, 1381, 1496, + 1611, 1841, 1956, 2071, 2186, 2301, 2416, 2531, 2761, 2876, 2991, + 3106, 3221, 3336, 3451, 3681, 3682, 3911, 3912, 4141, 4142, 4371, + 4372, 4601, 4602, 4831, 4832, 5061, 5062, 5521, 5522, 5751, 5752, + 5981, 5982, 6211, 6212, 6441, 6442, 6671, 6672, 6901, 6902, 7361, + 7362, 7591, 7592, 7821, 7822, 8051, 8052, 8281, 8282, 8511, 8512, + 8741, 8742, 9201, 9202, 9431, 9432, 9661, 9662, 9891, 10121, 10122, + 10351, 10352, 10581, 10582, 11040 }, + { 1, 118, 235, 352, 469, 586, 703, 937, 1054, 1171, 1288, 1405, 1522, + 1639, 1873, 1990, 2107, 2224, 2341, 2458, 2575, 2809, 2926, 3043, + 3160, 3277, 3394, 3511, 3745, 3746, 3979, 3980, 4213, 4214, 4447, + 4448, 4681, 4682, 4915, 4916, 5149, 5150, 5617, 5618, 5851, 5852, + 6085, 6086, 6319, 6320, 6553, 6554, 6787, 6788, 7021, 7022, 7489, + 7490, 7723, 7724, 7957, 7958, 8191, 8192, 8425, 8426, 8659, 8660, + 8893, 8894, 9361, 9362, 9595, 9596, 9829, 9830, 10063, 10297, 10298, + 10531, 10532, 10765, 10766, 11232 }, + { 1, 120, 239, 358, 477, 596, 715, 953, 1072, 1191, 1310, 1429, 1548, + 1667, 1905, 2024, 2143, 2262, 2381, 2500, 2619, 2857, 2976, 3095, + 3214, 3333, 3452, 3571, 3809, 3810, 4047, 4048, 4285, 4286, 4523, + 4524, 4761, 4762, 4999, 5000, 5237, 5238, 5713, 5714, 5951, 5952, + 6189, 6190, 6427, 6428, 6665, 6666, 6903, 6904, 7141, 7142, 7617, + 7618, 7855, 7856, 8093, 8094, 8331, 8332, 8569, 8570, 8807, 8808, + 9045, 9046, 9521, 9522, 9759, 9760, 9997, 9998, 10235, 10473, 10474, + 10711, 10712, 10949, 10950, 11424 }, + { 1, 122, 243, 364, 485, 606, 727, 969, 1090, 1211, 1332, 1453, 1574, + 1695, 1937, 2058, 2179, 2300, 2421, 2542, 2663, 2905, 3026, 3147, + 3268, 3389, 3510, 3631, 3873, 3874, 4115, 4116, 4357, 4358, 4599, + 4600, 4841, 4842, 5083, 5084, 5325, 5326, 5809, 5810, 6051, 6052, + 6293, 6294, 6535, 6536, 6777, 6778, 7019, 7020, 7261, 7262, 7745, + 7746, 7987, 7988, 8229, 8230, 8471, 8472, 8713, 8714, 8955, 8956, + 9197, 9198, 9681, 9682, 9923, 9924, 10165, 10166, 10407, 10649, 10650, + 10891, 10892, 11133, 11134, 11616 }, + { 1, 124, 247, 370, 493, 616, 739, 985, 1108, 1231, 1354, 1477, 1600, + 1723, 1969, 2092, 2215, 2338, 2461, 2584, 2707, 2953, 3076, 3199, + 3322, 3445, 3568, 3691, 3937, 3938, 4183, 4184, 4429, 4430, 4675, + 4676, 4921, 4922, 5167, 5168, 5413, 5414, 5905, 5906, 6151, 6152, + 6397, 6398, 6643, 6644, 6889, 6890, 7135, 7136, 7381, 7382, 7873, + 7874, 8119, 8120, 8365, 8366, 8611, 8612, 8857, 8858, 9103, 9104, + 9349, 9350, 9841, 9842, 10087, 10088, 10333, 10334, 10579, 10825, + 10826, 11071, 11072, 11317, 11318, 11808 }, + { 1, 126, 251, 376, 501, 626, 751, 1001, 1126, 1251, 1376, 1501, 1626, + 1751, 2001, 2126, 2251, 2376, 2501, 2626, 2751, 3001, 3126, 3251, + 3376, 3501, 3626, 3751, 4001, 4002, 4251, 4252, 4501, 4502, 4751, + 4752, 5001, 5002, 5251, 5252, 5501, 5502, 6001, 6002, 6251, 6252, + 6501, 6502, 6751, 6752, 7001, 7002, 7251, 7252, 7501, 7502, 8001, + 8002, 8251, 8252, 8501, 8502, 8751, 8752, 9001, 9002, 9251, 9252, + 9501, 9502, 10001, 10002, 10251, 10252, 10501, 10502, 10751, 11001, + 11002, 11251, 11252, 11501, 11502, 12000 }, + { 1, 128, 255, 382, 509, 636, 763, 1017, 1144, 1271, 1398, 1525, 1652, + 1779, 2033, 2160, 2287, 2414, 2541, 2668, 2795, 3049, 3176, 3303, + 3430, 3557, 3684, 3811, 4065, 4066, 4319, 4320, 4573, 4574, 4827, + 4828, 5081, 5082, 5335, 5336, 5589, 5590, 6097, 6098, 6351, 6352, + 6605, 6606, 6859, 6860, 7113, 7114, 7367, 7368, 7621, 7622, 8129, + 8130, 8383, 8384, 8637, 8638, 8891, 8892, 9145, 9146, 9399, 9400, + 9653, 9654, 10161, 10162, 10415, 10416, 10669, 10670, 10923, 11177, + 11178, 11431, 11432, 11685, 11686, 12192 }, + { 1, 130, 259, 388, 517, 646, 775, 1033, 1162, 1291, 1420, 1549, 1678, + 1807, 2065, 2194, 2323, 2452, 2581, 2710, 2839, 3097, 3226, 3355, + 3484, 3613, 3742, 3871, 4129, 4130, 4387, 4388, 4645, 4646, 4903, + 4904, 5161, 5162, 5419, 5420, 5677, 5678, 6193, 6194, 6451, 6452, + 6709, 6710, 6967, 6968, 7225, 7226, 7483, 7484, 7741, 7742, 8257, + 8258, 8515, 8516, 8773, 8774, 9031, 9032, 9289, 9290, 9547, 9548, + 9805, 9806, 10321, 10322, 10579, 10580, 10837, 10838, 11095, 11353, + 11354, 11611, 11612, 11869, 11870, 12384 }, + { 1, 132, 263, 394, 525, 656, 787, 1049, 1180, 1311, 1442, 1573, 1704, + 1835, 2097, 2228, 2359, 2490, 2621, 2752, 2883, 3145, 3276, 3407, + 3538, 3669, 3800, 3931, 4193, 4194, 4455, 4456, 4717, 4718, 4979, + 4980, 5241, 5242, 5503, 5504, 5765, 5766, 6289, 6290, 6551, 6552, + 6813, 6814, 7075, 7076, 7337, 7338, 7599, 7600, 7861, 7862, 8385, + 8386, 8647, 8648, 8909, 8910, 9171, 9172, 9433, 9434, 9695, 9696, + 9957, 9958, 10481, 10482, 10743, 10744, 11005, 11006, 11267, 11529, + 11530, 11791, 11792, 12053, 12054, 12576 }, + { 1, 134, 267, 400, 533, 666, 799, 1065, 1198, 1331, 1464, 1597, 1730, + 1863, 2129, 2262, 2395, 2528, 2661, 2794, 2927, 3193, 3326, 3459, + 3592, 3725, 3858, 3991, 4257, 4258, 4523, 4524, 4789, 4790, 5055, + 5056, 5321, 5322, 5587, 5588, 5853, 5854, 6385, 6386, 6651, 6652, + 6917, 6918, 7183, 7184, 7449, 7450, 7715, 7716, 7981, 7982, 8513, + 8514, 8779, 8780, 9045, 9046, 9311, 9312, 9577, 9578, 9843, 9844, + 10109, 10110, 10641, 10642, 10907, 10908, 11173, 11174, 11439, 11705, + 11706, 11971, 11972, 12237, 12238, 12768 }, + { 1, 136, 271, 406, 541, 676, 811, 1081, 1216, 1351, 1486, 1621, 1756, + 1891, 2161, 2296, 2431, 2566, 2701, 2836, 2971, 3241, 3376, 3511, + 3646, 3781, 3916, 4051, 4321, 4322, 4591, 4592, 4861, 4862, 5131, + 5132, 5401, 5402, 5671, 5672, 5941, 5942, 6481, 6482, 6751, 6752, + 7021, 7022, 7291, 7292, 7561, 7562, 7831, 7832, 8101, 8102, 8641, + 8642, 8911, 8912, 9181, 9182, 9451, 9452, 9721, 9722, 9991, 9992, + 10261, 10262, 10801, 10802, 11071, 11072, 11341, 11342, 11611, 11881, + 11882, 12151, 12152, 12421, 12422, 12960 }, + { 1, 138, 275, 412, 549, 686, 823, 1097, 1234, 1371, 1508, 1645, 1782, + 1919, 2193, 2330, 2467, 2604, 2741, 2878, 3015, 3289, 3426, 3563, + 3700, 3837, 3974, 4111, 4385, 4386, 4659, 4660, 4933, 4934, 5207, + 5208, 5481, 5482, 5755, 5756, 6029, 6030, 6577, 6578, 6851, 6852, + 7125, 7126, 7399, 7400, 7673, 7674, 7947, 7948, 8221, 8222, 8769, + 8770, 9043, 9044, 9317, 9318, 9591, 9592, 9865, 9866, 10139, 10140, + 10413, 10414, 10961, 10962, 11235, 11236, 11509, 11510, 11783, 12057, + 12058, 12331, 12332, 12605, 12606, 13152 }, + { 1, 140, 279, 418, 557, 696, 835, 1113, 1252, 1391, 1530, 1669, 1808, + 1947, 2225, 2364, 2503, 2642, 2781, 2920, 3059, 3337, 3476, 3615, + 3754, 3893, 4032, 4171, 4449, 4450, 4727, 4728, 5005, 5006, 5283, + 5284, 5561, 5562, 5839, 5840, 6117, 6118, 6673, 6674, 6951, 6952, + 7229, 7230, 7507, 7508, 7785, 7786, 8063, 8064, 8341, 8342, 8897, + 8898, 9175, 9176, 9453, 9454, 9731, 9732, 10009, 10010, 10287, 10288, + 10565, 10566, 11121, 11122, 11399, 11400, 11677, 11678, 11955, 12233, + 12234, 12511, 12512, 12789, 12790, 13344 }, + { 1, 142, 283, 424, 565, 706, 847, 1129, 1270, 1411, 1552, 1693, 1834, + 1975, 2257, 2398, 2539, 2680, 2821, 2962, 3103, 3385, 3526, 3667, + 3808, 3949, 4090, 4231, 4513, 4514, 4795, 4796, 5077, 5078, 5359, + 5360, 5641, 5642, 5923, 5924, 6205, 6206, 6769, 6770, 7051, 7052, + 7333, 7334, 7615, 7616, 7897, 7898, 8179, 8180, 8461, 8462, 9025, + 9026, 9307, 9308, 9589, 9590, 9871, 9872, 10153, 10154, 10435, 10436, + 10717, 10718, 11281, 11282, 11563, 11564, 11845, 11846, 12127, 12409, + 12410, 12691, 12692, 12973, 12974, 13536 }, + { 1, 144, 287, 430, 573, 716, 859, 1145, 1288, 1431, 1574, 1717, 1860, + 2003, 2289, 2432, 2575, 2718, 2861, 3004, 3147, 3433, 3576, 3719, + 3862, 4005, 4148, 4291, 4577, 4578, 4863, 4864, 5149, 5150, 5435, + 5436, 5721, 5722, 6007, 6008, 6293, 6294, 6865, 6866, 7151, 7152, + 7437, 7438, 7723, 7724, 8009, 8010, 8295, 8296, 8581, 8582, 9153, + 9154, 9439, 9440, 9725, 9726, 10011, 10012, 10297, 10298, 10583, + 10584, 10869, 10870, 11441, 11442, 11727, 11728, 12013, 12014, 12299, + 12585, 12586, 12871, 12872, 13157, 13158, 13728 }, + { 1, 146, 291, 436, 581, 726, 871, 1161, 1306, 1451, 1596, 1741, 1886, + 2031, 2321, 2466, 2611, 2756, 2901, 3046, 3191, 3481, 3626, 3771, + 3916, 4061, 4206, 4351, 4641, 4642, 4931, 4932, 5221, 5222, 5511, + 5512, 5801, 5802, 6091, 6092, 6381, 6382, 6961, 6962, 7251, 7252, + 7541, 7542, 7831, 7832, 8121, 8122, 8411, 8412, 8701, 8702, 9281, + 9282, 9571, 9572, 9861, 9862, 10151, 10152, 10441, 10442, 10731, + 10732, 11021, 11022, 11601, 11602, 11891, 11892, 12181, 12182, 12471, + 12761, 12762, 13051, 13052, 13341, 13342, 13920 }, + { 1, 148, 295, 442, 589, 736, 883, 1177, 1324, 1471, 1618, 1765, 1912, + 2059, 2353, 2500, 2647, 2794, 2941, 3088, 3235, 3529, 3676, 3823, + 3970, 4117, 4264, 4411, 4705, 4706, 4999, 5000, 5293, 5294, 5587, + 5588, 5881, 5882, 6175, 6176, 6469, 6470, 7057, 7058, 7351, 7352, + 7645, 7646, 7939, 7940, 8233, 8234, 8527, 8528, 8821, 8822, 9409, + 9410, 9703, 9704, 9997, 9998, 10291, 10292, 10585, 10586, 10879, + 10880, 11173, 11174, 11761, 11762, 12055, 12056, 12349, 12350, 12643, + 12937, 12938, 13231, 13232, 13525, 13526, 14112 }, + { 1, 150, 299, 448, 597, 746, 895, 1193, 1342, 1491, 1640, 1789, 1938, + 2087, 2385, 2534, 2683, 2832, 2981, 3130, 3279, 3577, 3726, 3875, + 4024, 4173, 4322, 4471, 4769, 4770, 5067, 5068, 5365, 5366, 5663, + 5664, 5961, 5962, 6259, 6260, 6557, 6558, 7153, 7154, 7451, 7452, + 7749, 7750, 8047, 8048, 8345, 8346, 8643, 8644, 8941, 8942, 9537, + 9538, 9835, 9836, 10133, 10134, 10431, 10432, 10729, 10730, 11027, + 11028, 11325, 11326, 11921, 11922, 12219, 12220, 12517, 12518, 12815, + 13113, 13114, 13411, 13412, 13709, 13710, 14304 }, + { 1, 152, 303, 454, 605, 756, 907, 1209, 1360, 1511, 1662, 1813, 1964, + 2115, 2417, 2568, 2719, 2870, 3021, 3172, 3323, 3625, 3776, 3927, + 4078, 4229, 4380, 4531, 4833, 4834, 5135, 5136, 5437, 5438, 5739, + 5740, 6041, 6042, 6343, 6344, 6645, 6646, 7249, 7250, 7551, 7552, + 7853, 7854, 8155, 8156, 8457, 8458, 8759, 8760, 9061, 9062, 9665, + 9666, 9967, 9968, 10269, 10270, 10571, 10572, 10873, 10874, 11175, + 11176, 11477, 11478, 12081, 12082, 12383, 12384, 12685, 12686, 12987, + 13289, 13290, 13591, 13592, 13893, 13894, 14496 }, + { 1, 154, 307, 460, 613, 766, 919, 1225, 1378, 1531, 1684, 1837, 1990, + 2143, 2449, 2602, 2755, 2908, 3061, 3214, 3367, 3673, 3826, 3979, + 4132, 4285, 4438, 4591, 4897, 4898, 5203, 5204, 5509, 5510, 5815, + 5816, 6121, 6122, 6427, 6428, 6733, 6734, 7345, 7346, 7651, 7652, + 7957, 7958, 8263, 8264, 8569, 8570, 8875, 8876, 9181, 9182, 9793, + 9794, 10099, 10100, 10405, 10406, 10711, 10712, 11017, 11018, 11323, + 11324, 11629, 11630, 12241, 12242, 12547, 12548, 12853, 12854, 13159, + 13465, 13466, 13771, 13772, 14077, 14078, 14688 }, + { 1, 156, 311, 466, 621, 776, 931, 1241, 1396, 1551, 1706, 1861, 2016, + 2171, 2481, 2636, 2791, 2946, 3101, 3256, 3411, 3721, 3876, 4031, + 4186, 4341, 4496, 4651, 4961, 4962, 5271, 5272, 5581, 5582, 5891, + 5892, 6201, 6202, 6511, 6512, 6821, 6822, 7441, 7442, 7751, 7752, + 8061, 8062, 8371, 8372, 8681, 8682, 8991, 8992, 9301, 9302, 9921, + 9922, 10231, 10232, 10541, 10542, 10851, 10852, 11161, 11162, 11471, + 11472, 11781, 11782, 12401, 12402, 12711, 12712, 13021, 13022, 13331, + 13641, 13642, 13951, 13952, 14261, 14262, 14880 }, + { 1, 158, 315, 472, 629, 786, 943, 1257, 1414, 1571, 1728, 1885, 2042, + 2199, 2513, 2670, 2827, 2984, 3141, 3298, 3455, 3769, 3926, 4083, + 4240, 4397, 4554, 4711, 5025, 5026, 5339, 5340, 5653, 5654, 5967, + 5968, 6281, 6282, 6595, 6596, 6909, 6910, 7537, 7538, 7851, 7852, + 8165, 8166, 8479, 8480, 8793, 8794, 9107, 9108, 9421, 9422, 10049, + 10050, 10363, 10364, 10677, 10678, 10991, 10992, 11305, 11306, 11619, + 11620, 11933, 11934, 12561, 12562, 12875, 12876, 13189, 13190, 13503, + 13817, 13818, 14131, 14132, 14445, 14446, 15072 }, + { 1, 160, 319, 478, 637, 796, 955, 1273, 1432, 1591, 1750, 1909, 2068, + 2227, 2545, 2704, 2863, 3022, 3181, 3340, 3499, 3817, 3976, 4135, + 4294, 4453, 4612, 4771, 5089, 5090, 5407, 5408, 5725, 5726, 6043, + 6044, 6361, 6362, 6679, 6680, 6997, 6998, 7633, 7634, 7951, 7952, + 8269, 8270, 8587, 8588, 8905, 8906, 9223, 9224, 9541, 9542, 10177, + 10178, 10495, 10496, 10813, 10814, 11131, 11132, 11449, 11450, 11767, + 11768, 12085, 12086, 12721, 12722, 13039, 13040, 13357, 13358, 13675, + 13993, 13994, 14311, 14312, 14629, 14630, 15264 }, + { 1, 162, 323, 484, 645, 806, 967, 1289, 1450, 1611, 1772, 1933, 2094, + 2255, 2577, 2738, 2899, 3060, 3221, 3382, 3543, 3865, 4026, 4187, + 4348, 4509, 4670, 4831, 5153, 5154, 5475, 5476, 5797, 5798, 6119, + 6120, 6441, 6442, 6763, 6764, 7085, 7086, 7729, 7730, 8051, 8052, + 8373, 8374, 8695, 8696, 9017, 9018, 9339, 9340, 9661, 9662, 10305, + 10306, 10627, 10628, 10949, 10950, 11271, 11272, 11593, 11594, 11915, + 11916, 12237, 12238, 12881, 12882, 13203, 13204, 13525, 13526, 13847, + 14169, 14170, 14491, 14492, 14813, 14814, 15456 }, + { 1, 164, 327, 490, 653, 816, 979, 1305, 1468, 1631, 1794, 1957, 2120, + 2283, 2609, 2772, 2935, 3098, 3261, 3424, 3587, 3913, 4076, 4239, + 4402, 4565, 4728, 4891, 5217, 5218, 5543, 5544, 5869, 5870, 6195, + 6196, 6521, 6522, 6847, 6848, 7173, 7174, 7825, 7826, 8151, 8152, + 8477, 8478, 8803, 8804, 9129, 9130, 9455, 9456, 9781, 9782, 10433, + 10434, 10759, 10760, 11085, 11086, 11411, 11412, 11737, 11738, 12063, + 12064, 12389, 12390, 13041, 13042, 13367, 13368, 13693, 13694, 14019, + 14345, 14346, 14671, 14672, 14997, 14998, 15648 }, + { 1, 166, 331, 496, 661, 826, 991, 1321, 1486, 1651, 1816, 1981, 2146, + 2311, 2641, 2806, 2971, 3136, 3301, 3466, 3631, 3961, 4126, 4291, + 4456, 4621, 4786, 4951, 5281, 5282, 5611, 5612, 5941, 5942, 6271, + 6272, 6601, 6602, 6931, 6932, 7261, 7262, 7921, 7922, 8251, 8252, + 8581, 8582, 8911, 8912, 9241, 9242, 9571, 9572, 9901, 9902, 10561, + 10562, 10891, 10892, 11221, 11222, 11551, 11552, 11881, 11882, 12211, + 12212, 12541, 12542, 13201, 13202, 13531, 13532, 13861, 13862, 14191, + 14521, 14522, 14851, 14852, 15181, 15182, 15840 }, + { 1, 168, 335, 502, 669, 836, 1003, 1337, 1504, 1671, 1838, 2005, 2172, + 2339, 2673, 2840, 3007, 3174, 3341, 3508, 3675, 4009, 4176, 4343, + 4510, 4677, 4844, 5011, 5345, 5346, 5679, 5680, 6013, 6014, 6347, + 6348, 6681, 6682, 7015, 7016, 7349, 7350, 8017, 8018, 8351, 8352, + 8685, 8686, 9019, 9020, 9353, 9354, 9687, 9688, 10021, 10022, 10689, + 10690, 11023, 11024, 11357, 11358, 11691, 11692, 12025, 12026, 12359, + 12360, 12693, 12694, 13361, 13362, 13695, 13696, 14029, 14030, 14363, + 14697, 14698, 15031, 15032, 15365, 15366, 16032 }, + { 1, 170, 339, 508, 677, 846, 1015, 1353, 1522, 1691, 1860, 2029, 2198, + 2367, 2705, 2874, 3043, 3212, 3381, 3550, 3719, 4057, 4226, 4395, + 4564, 4733, 4902, 5071, 5409, 5410, 5747, 5748, 6085, 6086, 6423, + 6424, 6761, 6762, 7099, 7100, 7437, 7438, 8113, 8114, 8451, 8452, + 8789, 8790, 9127, 9128, 9465, 9466, 9803, 9804, 10141, 10142, 10817, + 10818, 11155, 11156, 11493, 11494, 11831, 11832, 12169, 12170, 12507, + 12508, 12845, 12846, 13521, 13522, 13859, 13860, 14197, 14198, 14535, + 14873, 14874, 15211, 15212, 15549, 15550, 16224 }, + { 1, 172, 343, 514, 685, 856, 1027, 1369, 1540, 1711, 1882, 2053, 2224, + 2395, 2737, 2908, 3079, 3250, 3421, 3592, 3763, 4105, 4276, 4447, + 4618, 4789, 4960, 5131, 5473, 5474, 5815, 5816, 6157, 6158, 6499, + 6500, 6841, 6842, 7183, 7184, 7525, 7526, 8209, 8210, 8551, 8552, + 8893, 8894, 9235, 9236, 9577, 9578, 9919, 9920, 10261, 10262, 10945, + 10946, 11287, 11288, 11629, 11630, 11971, 11972, 12313, 12314, 12655, + 12656, 12997, 12998, 13681, 13682, 14023, 14024, 14365, 14366, 14707, + 15049, 15050, 15391, 15392, 15733, 15734, 16416 }, + { 1, 174, 347, 520, 693, 866, 1039, 1385, 1558, 1731, 1904, 2077, 2250, + 2423, 2769, 2942, 3115, 3288, 3461, 3634, 3807, 4153, 4326, 4499, + 4672, 4845, 5018, 5191, 5537, 5538, 5883, 5884, 6229, 6230, 6575, + 6576, 6921, 6922, 7267, 7268, 7613, 7614, 8305, 8306, 8651, 8652, + 8997, 8998, 9343, 9344, 9689, 9690, 10035, 10036, 10381, 10382, 11073, + 11074, 11419, 11420, 11765, 11766, 12111, 12112, 12457, 12458, 12803, + 12804, 13149, 13150, 13841, 13842, 14187, 14188, 14533, 14534, 14879, + 15225, 15226, 15571, 15572, 15917, 15918, 16608 }, + { 1, 176, 351, 526, 701, 876, 1051, 1401, 1576, 1751, 1926, 2101, 2276, + 2451, 2801, 2976, 3151, 3326, 3501, 3676, 3851, 4201, 4376, 4551, + 4726, 4901, 5076, 5251, 5601, 5602, 5951, 5952, 6301, 6302, 6651, + 6652, 7001, 7002, 7351, 7352, 7701, 7702, 8401, 8402, 8751, 8752, + 9101, 9102, 9451, 9452, 9801, 9802, 10151, 10152, 10501, 10502, 11201, + 11202, 11551, 11552, 11901, 11902, 12251, 12252, 12601, 12602, 12951, + 12952, 13301, 13302, 14001, 14002, 14351, 14352, 14701, 14702, 15051, + 15401, 15402, 15751, 15752, 16101, 16102, 16800 }, + { 1, 178, 355, 532, 709, 886, 1063, 1417, 1594, 1771, 1948, 2125, 2302, + 2479, 2833, 3010, 3187, 3364, 3541, 3718, 3895, 4249, 4426, 4603, + 4780, 4957, 5134, 5311, 5665, 5666, 6019, 6020, 6373, 6374, 6727, + 6728, 7081, 7082, 7435, 7436, 7789, 7790, 8497, 8498, 8851, 8852, + 9205, 9206, 9559, 9560, 9913, 9914, 10267, 10268, 10621, 10622, 11329, + 11330, 11683, 11684, 12037, 12038, 12391, 12392, 12745, 12746, 13099, + 13100, 13453, 13454, 14161, 14162, 14515, 14516, 14869, 14870, 15223, + 15577, 15578, 15931, 15932, 16285, 16286, 16992 }, + { 1, 180, 359, 538, 717, 896, 1075, 1433, 1612, 1791, 1970, 2149, 2328, + 2507, 2865, 3044, 3223, 3402, 3581, 3760, 3939, 4297, 4476, 4655, + 4834, 5013, 5192, 5371, 5729, 5730, 6087, 6088, 6445, 6446, 6803, + 6804, 7161, 7162, 7519, 7520, 7877, 7878, 8593, 8594, 8951, 8952, + 9309, 9310, 9667, 9668, 10025, 10026, 10383, 10384, 10741, 10742, + 11457, 11458, 11815, 11816, 12173, 12174, 12531, 12532, 12889, 12890, + 13247, 13248, 13605, 13606, 14321, 14322, 14679, 14680, 15037, 15038, + 15395, 15753, 15754, 16111, 16112, 16469, 16470, 17184 }, + { 1, 182, 363, 544, 725, 906, 1087, 1449, 1630, 1811, 1992, 2173, 2354, + 2535, 2897, 3078, 3259, 3440, 3621, 3802, 3983, 4345, 4526, 4707, + 4888, 5069, 5250, 5431, 5793, 5794, 6155, 6156, 6517, 6518, 6879, + 6880, 7241, 7242, 7603, 7604, 7965, 7966, 8689, 8690, 9051, 9052, + 9413, 9414, 9775, 9776, 10137, 10138, 10499, 10500, 10861, 10862, + 11585, 11586, 11947, 11948, 12309, 12310, 12671, 12672, 13033, 13034, + 13395, 13396, 13757, 13758, 14481, 14482, 14843, 14844, 15205, 15206, + 15567, 15929, 15930, 16291, 16292, 16653, 16654, 17376 }, + { 1, 184, 367, 550, 733, 916, 1099, 1465, 1648, 1831, 2014, 2197, 2380, + 2563, 2929, 3112, 3295, 3478, 3661, 3844, 4027, 4393, 4576, 4759, + 4942, 5125, 5308, 5491, 5857, 5858, 6223, 6224, 6589, 6590, 6955, + 6956, 7321, 7322, 7687, 7688, 8053, 8054, 8785, 8786, 9151, 9152, + 9517, 9518, 9883, 9884, 10249, 10250, 10615, 10616, 10981, 10982, + 11713, 11714, 12079, 12080, 12445, 12446, 12811, 12812, 13177, 13178, + 13543, 13544, 13909, 13910, 14641, 14642, 15007, 15008, 15373, 15374, + 15739, 16105, 16106, 16471, 16472, 16837, 16838, 17568 }, + { 1, 186, 371, 556, 741, 926, 1111, 1481, 1666, 1851, 2036, 2221, 2406, + 2591, 2961, 3146, 3331, 3516, 3701, 3886, 4071, 4441, 4626, 4811, + 4996, 5181, 5366, 5551, 5921, 5922, 6291, 6292, 6661, 6662, 7031, + 7032, 7401, 7402, 7771, 7772, 8141, 8142, 8881, 8882, 9251, 9252, + 9621, 9622, 9991, 9992, 10361, 10362, 10731, 10732, 11101, 11102, + 11841, 11842, 12211, 12212, 12581, 12582, 12951, 12952, 13321, 13322, + 13691, 13692, 14061, 14062, 14801, 14802, 15171, 15172, 15541, 15542, + 15911, 16281, 16282, 16651, 16652, 17021, 17022, 17760 }, + { 1, 188, 375, 562, 749, 936, 1123, 1497, 1684, 1871, 2058, 2245, 2432, + 2619, 2993, 3180, 3367, 3554, 3741, 3928, 4115, 4489, 4676, 4863, + 5050, 5237, 5424, 5611, 5985, 5986, 6359, 6360, 6733, 6734, 7107, + 7108, 7481, 7482, 7855, 7856, 8229, 8230, 8977, 8978, 9351, 9352, + 9725, 9726, 10099, 10100, 10473, 10474, 10847, 10848, 11221, 11222, + 11969, 11970, 12343, 12344, 12717, 12718, 13091, 13092, 13465, 13466, + 13839, 13840, 14213, 14214, 14961, 14962, 15335, 15336, 15709, 15710, + 16083, 16457, 16458, 16831, 16832, 17205, 17206, 17952 }, + { 1, 190, 379, 568, 757, 946, 1135, 1513, 1702, 1891, 2080, 2269, 2458, + 2647, 3025, 3214, 3403, 3592, 3781, 3970, 4159, 4537, 4726, 4915, + 5104, 5293, 5482, 5671, 6049, 6050, 6427, 6428, 6805, 6806, 7183, + 7184, 7561, 7562, 7939, 7940, 8317, 8318, 9073, 9074, 9451, 9452, + 9829, 9830, 10207, 10208, 10585, 10586, 10963, 10964, 11341, 11342, + 12097, 12098, 12475, 12476, 12853, 12854, 13231, 13232, 13609, 13610, + 13987, 13988, 14365, 14366, 15121, 15122, 15499, 15500, 15877, 15878, + 16255, 16633, 16634, 17011, 17012, 17389, 17390, 18144 }, + { 1, 192, 383, 574, 765, 956, 1147, 1529, 1720, 1911, 2102, 2293, 2484, + 2675, 3057, 3248, 3439, 3630, 3821, 4012, 4203, 4585, 4776, 4967, + 5158, 5349, 5540, 5731, 6113, 6114, 6495, 6496, 6877, 6878, 7259, + 7260, 7641, 7642, 8023, 8024, 8405, 8406, 9169, 9170, 9551, 9552, + 9933, 9934, 10315, 10316, 10697, 10698, 11079, 11080, 11461, 11462, + 12225, 12226, 12607, 12608, 12989, 12990, 13371, 13372, 13753, 13754, + 14135, 14136, 14517, 14518, 15281, 15282, 15663, 15664, 16045, 16046, + 16427, 16809, 16810, 17191, 17192, 17573, 17574, 18336 }, + { 1, 194, 387, 580, 773, 966, 1159, 1545, 1738, 1931, 2124, 2317, 2510, + 2703, 3089, 3282, 3475, 3668, 3861, 4054, 4247, 4633, 4826, 5019, + 5212, 5405, 5598, 5791, 6177, 6178, 6563, 6564, 6949, 6950, 7335, + 7336, 7721, 7722, 8107, 8108, 8493, 8494, 9265, 9266, 9651, 9652, + 10037, 10038, 10423, 10424, 10809, 10810, 11195, 11196, 11581, 11582, + 12353, 12354, 12739, 12740, 13125, 13126, 13511, 13512, 13897, 13898, + 14283, 14284, 14669, 14670, 15441, 15442, 15827, 15828, 16213, 16214, + 16599, 16985, 16986, 17371, 17372, 17757, 17758, 18528 } +}; + +#endif /* _BBDEV_TURBO_SW_TABLES_H */ diff --git a/drivers/bbdev/turbo_sw/rte_pmd_bbdev_turbo_sw_version.map b/drivers/bbdev/turbo_sw/rte_pmd_bbdev_turbo_sw_version.map new file mode 100644 index 0000000..a753031 --- /dev/null +++ b/drivers/bbdev/turbo_sw/rte_pmd_bbdev_turbo_sw_version.map @@ -0,0 +1,3 @@ +DPDK_17.11 { + local: *; +}; -- 2.7.4 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [dpdk-dev] [PATCH v2 3/5] bbdev: test applications 2017-10-18 2:14 [dpdk-dev] [PATCH v2 1/5] bbdev: librte_bbdev library Amr Mokhtar 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 2/5] bbdev: PMD drivers (null/turbo_sw) Amr Mokhtar @ 2017-10-18 2:14 ` Amr Mokhtar 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 4/5] bbdev: sample app Amr Mokhtar ` (2 subsequent siblings) 4 siblings, 0 replies; 6+ messages in thread From: Amr Mokhtar @ 2017-10-18 2:14 UTC (permalink / raw) To: dev Cc: thomas, anatoly.burakov, pablo.de.lara.guarch, niall.power, chris.macnamara, Amr Mokhtar - Full test suite for bbdev - Test App works seamlessly on all PMDs registered with bbdev framework - A python script is provided to make our life easier - Supports execution of tests by parsing Test Vector files - Test Vectors can be added/deleted/modified with no need for re-compilation - Various Tests can be executed: (a) Throughput test (b) Offload latency test (c) Operation latency test (d) Validation test (c) Sanity checks Signed-off-by: Amr Mokhtar <amr.mokhtar@intel.com> --- app/Makefile | 4 + app/test-bbdev/Makefile | 53 + app/test-bbdev/main.c | 317 +++ app/test-bbdev/main.h | 144 ++ app/test-bbdev/test-bbdev.py | 132 ++ app/test-bbdev/test_bbdev.c | 1406 +++++++++++++ app/test-bbdev/test_bbdev_perf.c | 2090 ++++++++++++++++++++ app/test-bbdev/test_bbdev_vector.c | 884 +++++++++ app/test-bbdev/test_bbdev_vector.h | 98 + app/test-bbdev/test_vectors/bbdev_vector_null.data | 32 + .../test_vectors/bbdev_vector_td_default.data | 80 + .../test_vectors/bbdev_vector_te_default.data | 60 + 12 files changed, 5300 insertions(+) create mode 100644 app/test-bbdev/Makefile create mode 100644 app/test-bbdev/main.c create mode 100644 app/test-bbdev/main.h create mode 100755 app/test-bbdev/test-bbdev.py create mode 100644 app/test-bbdev/test_bbdev.c create mode 100644 app/test-bbdev/test_bbdev_perf.c create mode 100644 app/test-bbdev/test_bbdev_vector.c create mode 100644 app/test-bbdev/test_bbdev_vector.h create mode 100644 app/test-bbdev/test_vectors/bbdev_vector_null.data create mode 100644 app/test-bbdev/test_vectors/bbdev_vector_td_default.data create mode 100644 app/test-bbdev/test_vectors/bbdev_vector_te_default.data diff --git a/app/Makefile b/app/Makefile index 7ea02b0..6b0ed67 100644 --- a/app/Makefile +++ b/app/Makefile @@ -43,4 +43,8 @@ ifeq ($(CONFIG_RTE_LIBRTE_EVENTDEV),y) DIRS-$(CONFIG_RTE_APP_EVENTDEV) += test-eventdev endif +ifeq ($(CONFIG_RTE_LIBRTE_BBDEV),y) +DIRS-$(CONFIG_RTE_TEST_BBDEV) += test-bbdev +endif + include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/app/test-bbdev/Makefile b/app/test-bbdev/Makefile new file mode 100644 index 0000000..29c9be5 --- /dev/null +++ b/app/test-bbdev/Makefile @@ -0,0 +1,53 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +include $(RTE_SDK)/mk/rte.vars.mk + +ifeq ($(CONFIG_RTE_TEST_BBDEV),y) + +# +# library name +# +APP = testbbdev + +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) + +# +# all sources are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_BBDEV) += main.c +SRCS-$(CONFIG_RTE_LIBRTE_BBDEV) += test_bbdev.c +SRCS-$(CONFIG_RTE_LIBRTE_BBDEV) += test_bbdev_perf.c +SRCS-$(CONFIG_RTE_LIBRTE_BBDEV) += test_bbdev_vector.c + +include $(RTE_SDK)/mk/rte.app.mk + +endif diff --git a/app/test-bbdev/main.c b/app/test-bbdev/main.c new file mode 100644 index 0000000..59b01f9 --- /dev/null +++ b/app/test-bbdev/main.c @@ -0,0 +1,317 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <getopt.h> +#include <inttypes.h> +#include <stdio.h> +#include <string.h> +#include <stdbool.h> + +#include <rte_eal.h> +#include <rte_common.h> +#include <rte_string_fns.h> +#include <rte_cycles.h> + +#include "main.h" + +/* Defines how many testcases can be specified as cmdline args */ +#define MAX_CMDLINE_TESTCASES 8 + +static const char tc_sep = ','; + +static struct test_params { + struct test_command *test_to_run[MAX_CMDLINE_TESTCASES]; + unsigned int num_tests; + unsigned int num_ops; + unsigned int burst_sz; + char test_vector_filename[PATH_MAX]; +} test_params; + +static struct test_commands_list commands_list = + TAILQ_HEAD_INITIALIZER(commands_list); + +void +add_test_command(struct test_command *t) +{ + TAILQ_INSERT_TAIL(&commands_list, t, next); +} + +int +unit_test_suite_runner(struct unit_test_suite *suite) +{ + int test_result = TEST_SUCCESS; + unsigned int total = 0, skipped = 0, succeeded = 0, failed = 0; + uint64_t start, end; + + printf( + "\n + ------------------------------------------------------- +\n"); + printf(" + Starting Test Suite : %s\n", suite->suite_name); + + start = rte_rdtsc_precise(); + + if (suite->setup) { + test_result = suite->setup(); + if (test_result == TEST_FAILED) { + printf(" + Test suite setup %s failed!\n", + suite->suite_name); + printf( + " + ------------------------------------------------------- +\n"); + return 1; + } + if (test_result == TEST_SKIPPED) { + printf(" + Test suite setup %s skipped!\n", + suite->suite_name); + printf( + " + ------------------------------------------------------- +\n"); + return 0; + } + } + + while (suite->unit_test_cases[total].testcase) { + if (suite->unit_test_cases[total].setup) + test_result = suite->unit_test_cases[total].setup(); + + if (test_result == TEST_SUCCESS) + test_result = suite->unit_test_cases[total].testcase(); + + if (suite->unit_test_cases[total].teardown) + suite->unit_test_cases[total].teardown(); + + if (test_result == TEST_SUCCESS) { + succeeded++; + printf(" + TestCase [%2d] : %s passed\n", total, + suite->unit_test_cases[total].name); + } else if (test_result == TEST_SKIPPED) { + skipped++; + printf(" + TestCase [%2d] : %s skipped\n", total, + suite->unit_test_cases[total].name); + } else { + failed++; + printf(" + TestCase [%2d] : %s failed\n", total, + suite->unit_test_cases[total].name); + } + + total++; + } + + /* Run test suite teardown */ + if (suite->teardown) + suite->teardown(); + + end = rte_rdtsc_precise(); + + printf(" + ------------------------------------------------------- +\n"); + printf(" + Test Suite Summary : %s\n", suite->suite_name); + printf(" + Tests Total : %2d\n", total); + printf(" + Tests Skipped : %2d\n", skipped); + printf(" + Tests Passed : %2d\n", succeeded); + printf(" + Tests Failed : %2d\n", failed); + printf(" + Tests Lasted : %lg ms\n", + ((end - start) * 1000) / (double)rte_get_tsc_hz()); + printf(" + ------------------------------------------------------- +\n"); + + return (failed > 0) ? 1 : 0; +} + +const char * +get_vector_filename(void) +{ + return test_params.test_vector_filename; +} + +unsigned int +get_num_ops(void) +{ + return test_params.num_ops; +} + +unsigned int +get_burst_sz(void) +{ + return test_params.burst_sz; +} + +static void +print_usage(const char *prog_name) +{ + struct test_command *t; + + printf("Usage: %s [EAL params] [-- [-n/--num-ops NUM_OPS]\n" + "\t[-b/--burst-size BURST_SIZE]\n" + "\t[-v/--test-vector VECTOR_FILE]\n" + "\t[-c/--test-cases TEST_CASE[,TEST_CASE,...]]]\n", + prog_name); + + printf("Available testcases: "); + TAILQ_FOREACH(t, &commands_list, next) + printf("%s ", t->command); + printf("\n"); +} + +static int +parse_args(int argc, char **argv, struct test_params *tp) +{ + int opt, option_index; + unsigned int num_tests = 0; + bool test_cases_present = false; + bool test_vector_present = false; + struct test_command *t; + char *tokens[MAX_CMDLINE_TESTCASES]; + int tc, ret; + + static struct option lgopts[] = { + { "num-ops", 1, 0, 'n' }, + { "burst-size", 1, 0, 'b' }, + { "test-cases", 1, 0, 'c' }, + { "test-vector", 1, 0, 'v' }, + { "help", 0, 0, 'h' }, + { NULL, 0, 0, 0 } + }; + + while ((opt = getopt_long(argc, argv, "hn:b:c:v:", lgopts, + &option_index)) != EOF) + switch (opt) { + case 'n': + TEST_ASSERT(strlen(optarg) > 0, + "Num of operations is not provided"); + tp->num_ops = strtol(optarg, NULL, 10); + break; + case 'b': + TEST_ASSERT(strlen(optarg) > 0, + "Burst size is not provided"); + tp->burst_sz = strtol(optarg, NULL, 10); + TEST_ASSERT(tp->burst_sz <= MAX_BURST, + "Burst size mustn't be greater than %u", + MAX_BURST); + break; + case 'c': + TEST_ASSERT(test_cases_present == false, + "Test cases provided more than once"); + test_cases_present = true; + + ret = rte_strsplit(optarg, strlen(optarg), + tokens, MAX_CMDLINE_TESTCASES, tc_sep); + + TEST_ASSERT(ret <= MAX_CMDLINE_TESTCASES, + "Too many test cases (max=%d)", + MAX_CMDLINE_TESTCASES); + + for (tc = 0; tc < ret; ++tc) { + /* Find matching test case */ + TAILQ_FOREACH(t, &commands_list, next) + if (!strcmp(tokens[tc], t->command)) + tp->test_to_run[num_tests] = t; + + TEST_ASSERT(tp->test_to_run[num_tests] != NULL, + "Unknown test case: %s", + tokens[tc]); + ++num_tests; + } + break; + case 'v': + TEST_ASSERT(test_vector_present == false, + "Test vector provided more than once"); + test_vector_present = true; + + TEST_ASSERT(strlen(optarg) > 0, + "Config file name is null"); + + strncpy(tp->test_vector_filename, optarg, + sizeof(tp->test_vector_filename)); + break; + case 'h': + print_usage(argv[0]); + return 0; + default: + printf("ERROR: Unknown option: -%c\n", opt); + return -1; + } + + TEST_ASSERT(tp->burst_sz <= tp->num_ops, + "Burst size (%u) mustn't be greater than num ops (%u)", + tp->burst_sz, tp->num_ops); + + tp->num_tests = num_tests; + return 0; +} + +static int +run_all_tests(void) +{ + int ret = TEST_SUCCESS; + struct test_command *t; + + TAILQ_FOREACH(t, &commands_list, next) + ret |= t->callback(); + + return ret; +} + +static int +run_parsed_tests(struct test_params *tp) +{ + int ret = TEST_SUCCESS; + unsigned int i; + + for (i = 0; i < tp->num_tests; ++i) + ret |= tp->test_to_run[i]->callback(); + + return ret; +} + +int +main(int argc, char **argv) +{ + int ret; + + /* Init EAL */ + ret = rte_eal_init(argc, argv); + if (ret < 0) + return 1; + argc -= ret; + argv += ret; + + /* Parse application arguments (after the EAL ones) */ + ret = parse_args(argc, argv, &test_params); + if (ret < 0) { + print_usage(argv[0]); + return 1; + } + + rte_log_set_global_level(RTE_LOG_INFO); + + /* If no argument provided - run all tests */ + if (test_params.num_tests == 0) + return run_all_tests(); + else + return run_parsed_tests(&test_params); +} diff --git a/app/test-bbdev/main.h b/app/test-bbdev/main.h new file mode 100644 index 0000000..a60076d --- /dev/null +++ b/app/test-bbdev/main.h @@ -0,0 +1,144 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef _MAIN_H_ +#define _MAIN_H_ + +#include <stddef.h> +#include <sys/queue.h> + +#include <rte_common.h> +#include <rte_hexdump.h> +#include <rte_log.h> + +#define TEST_SUCCESS 0 +#define TEST_FAILED -1 +#define TEST_SKIPPED 1 + +#define MAX_BURST 512U + +#define TEST_ASSERT(cond, msg, ...) do { \ + if (!(cond)) { \ + printf("TestCase %s() line %d failed: " \ + msg "\n", __func__, __LINE__, ##__VA_ARGS__); \ + return TEST_FAILED; \ + } \ +} while (0) + +/* Compare two buffers (length in bytes) */ +#define TEST_ASSERT_BUFFERS_ARE_EQUAL(a, b, len, msg, ...) do { \ + if (memcmp((a), (b), len)) { \ + printf("TestCase %s() line %d failed: " \ + msg "\n", __func__, __LINE__, ##__VA_ARGS__); \ + rte_memdump(stdout, "Buffer A", (a), len); \ + rte_memdump(stdout, "Buffer B", (b), len); \ + return TEST_FAILED; \ + } \ +} while (0) + +#define TEST_ASSERT_SUCCESS(val, msg, ...) do { \ + typeof(val) _val = (val); \ + if (!(_val == 0)) { \ + printf("TestCase %s() line %d failed (err %d): " \ + msg "\n", __func__, __LINE__, _val, \ + ##__VA_ARGS__); \ + return TEST_FAILED; \ + } \ +} while (0) + +#define TEST_ASSERT_FAIL(val, msg, ...) \ + TEST_ASSERT_SUCCESS(!(val), msg, ##__VA_ARGS__) + +#define TEST_ASSERT_NOT_NULL(val, msg, ...) do { \ + if ((val) == NULL) { \ + printf("TestCase %s() line %d failed (null): " \ + msg "\n", __func__, __LINE__, ##__VA_ARGS__); \ + return TEST_FAILED; \ + } \ +} while (0) + +struct unit_test_case { + int (*setup)(void); + void (*teardown)(void); + int (*testcase)(void); + const char *name; +}; + +#define TEST_CASE(testcase) {NULL, NULL, testcase, #testcase} + +#define TEST_CASE_ST(setup, teardown, testcase) \ + {setup, teardown, testcase, #testcase} + +#define TEST_CASES_END() {NULL, NULL, NULL, NULL} + +struct unit_test_suite { + const char *suite_name; + int (*setup)(void); + void (*teardown)(void); + struct unit_test_case unit_test_cases[]; +}; + +int unit_test_suite_runner(struct unit_test_suite *suite); + +typedef int (test_callback)(void); +TAILQ_HEAD(test_commands_list, test_command); +struct test_command { + TAILQ_ENTRY(test_command) next; + const char *command; + test_callback *callback; +}; + +void add_test_command(struct test_command *t); + +/* Register a test function */ +#define REGISTER_TEST_COMMAND(name, testsuite) \ + static int test_func_##name(void) \ + { \ + return unit_test_suite_runner(&testsuite); \ + } \ + static struct test_command test_struct_##name = { \ + .command = RTE_STR(name), \ + .callback = test_func_##name, \ + }; \ + static void __attribute__((constructor, used)) \ + test_register_##name(void) \ + { \ + add_test_command(&test_struct_##name); \ + } + +const char *get_vector_filename(void); + +unsigned int get_num_ops(void); + +unsigned int get_burst_sz(void); + +#endif diff --git a/app/test-bbdev/test-bbdev.py b/app/test-bbdev/test-bbdev.py new file mode 100755 index 0000000..09dd2d7 --- /dev/null +++ b/app/test-bbdev/test-bbdev.py @@ -0,0 +1,132 @@ +#!/usr/bin/env python + +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +import sys +import os +import argparse +import subprocess +import shlex + +from threading import Timer + +def kill(process): + print "ERROR: Test app timed out" + process.kill() + +if "RTE_SDK" in os.environ: + dpdk_path = os.environ["RTE_SDK"] +else: + dpdk_path = "../.." + +if "RTE_TARGET" in os.environ: + dpdk_target = os.environ["RTE_TARGET"] +else: + dpdk_target = "x86_64-native-linuxapp-gcc" + +parser = argparse.ArgumentParser( + description='BBdev Unit Test Application', + formatter_class=argparse.ArgumentDefaultsHelpFormatter) +parser.add_argument("-p", "--testapp-path", + help="specifies path to the bbdev test app", + default=dpdk_path + "/" + dpdk_target + "/app/testbbdev") +parser.add_argument("-e", "--eal-params", + help="EAL arguments which are passed to the test app", + default="--vdev=bbdev_null0") +parser.add_argument("-t", "--timeout", + type=int, + help="Timeout in seconds", + default=300) +parser.add_argument("-c", "--test-cases", + nargs="+", + help="Defines test cases to run. Run all if not specified") +parser.add_argument("-v", "--test-vector", + nargs="+", + help="Specifies paths to the test vector files.", + default=[dpdk_path + + "/app/test-bbdev/test_vectors/bbdev_vector_null.data"]) +parser.add_argument("-n", "--num-ops", + type=int, + help="Number of operations to process on device.", + default=32) +parser.add_argument("-b", "--burst-size", + nargs="+", + type=int, + help="Operations enqueue/dequeue burst size.", + default=[32]) + +args = parser.parse_args() + +if not os.path.exists(args.testapp_path): + print "No such file: " + args.testapp_path + sys.exit(1) + +params = [args.testapp_path] +if args.eal_params: + params.extend(shlex.split(args.eal_params)) + +params.extend(["--"]) + +if args.num_ops: + params.extend(["-n", str(args.num_ops)]) + +if args.test_cases: + params.extend(["-c"]) + params.extend([",".join(args.test_cases)]) + +exit_status = 0 +for vector in args.test_vector: + for burst_size in args.burst_size: + call_params = params[:] + call_params.extend(["-v", vector]) + call_params.extend(["-b", str(burst_size)]) + params_string = " ".join(call_params) + + print("Executing: {}".format(params_string)) + app_proc = subprocess.Popen(call_params) + if args.timeout > 0: + timer = Timer(args.timeout, kill, [app_proc]) + timer.start() + + try: + app_proc.communicate() + except: + print("Error: failed to execute: {}".format(params_string)) + finally: + timer.cancel() + + if app_proc.returncode != 0: + exit_status = 1 + print("ERROR TestCase failed. Failed test for vector {}. Return code: {}".format( + vector, app_proc.returncode)) + +sys.exit(exit_status) diff --git a/app/test-bbdev/test_bbdev.c b/app/test-bbdev/test_bbdev.c new file mode 100644 index 0000000..982546d --- /dev/null +++ b/app/test-bbdev/test_bbdev.c @@ -0,0 +1,1406 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <rte_common.h> +#include <rte_hexdump.h> +#include <rte_mbuf.h> +#include <rte_malloc.h> +#include <rte_memcpy.h> +#include <rte_cycles.h> + +#include <rte_dev.h> + +#include <rte_bbdev.h> +#include <rte_bbdev_op.h> +#include <rte_bbdev_pmd.h> + +#include "main.h" + + +#define BBDEV_NAME_NULL ("bbdev_null") + +struct bbdev_testsuite_params { + struct rte_bbdev_queue_conf qconf; +}; + +static struct bbdev_testsuite_params testsuite_params; + +static uint8_t null_dev_id; + +static int +testsuite_setup(void) +{ + uint8_t nb_devs; + int ret; + char buf[RTE_BBDEV_NAME_MAX_LEN]; + + /* Create test device */ + snprintf(buf, sizeof(buf), "%s_unittest", BBDEV_NAME_NULL); + ret = rte_vdev_init(buf, NULL); + TEST_ASSERT(ret == 0, "Failed to create instance of pmd: %s", buf); + + nb_devs = rte_bbdev_count(); + TEST_ASSERT(nb_devs != 0, "No devices found"); + + /* Most recently created device is our device */ + null_dev_id = nb_devs - 1; + + return TEST_SUCCESS; +} + +static void +testsuite_teardown(void) +{ + char buf[RTE_BBDEV_NAME_MAX_LEN]; + + snprintf(buf, sizeof(buf), "%s_unittest", BBDEV_NAME_NULL); + rte_vdev_uninit(buf); +} + +static int +ut_setup(void) +{ + struct bbdev_testsuite_params *ts_params = &testsuite_params; + uint8_t num_queues; + + /* Valid queue configuration */ + ts_params->qconf.priority = 0; + ts_params->qconf.socket = SOCKET_ID_ANY; + ts_params->qconf.deferred_start = 1; + + num_queues = 1; + TEST_ASSERT_SUCCESS(rte_bbdev_setup_queues(null_dev_id, num_queues, + SOCKET_ID_ANY), "Failed to setup queues for bbdev %u", + 0); + + /* Start the device */ + TEST_ASSERT_SUCCESS(rte_bbdev_start(null_dev_id), + "Failed to start bbdev %u", 0); + + return TEST_SUCCESS; +} + +static void +ut_teardown(void) +{ + rte_bbdev_close(null_dev_id); +} + +static int +test_bbdev_configure_invalid_dev_id(void) +{ + uint8_t dev_id; + uint8_t num_queues; + + num_queues = 1; + for (dev_id = 0; dev_id < RTE_BBDEV_MAX_DEVS; dev_id++) { + if (!rte_bbdev_is_valid(dev_id)) { + TEST_ASSERT_FAIL(rte_bbdev_setup_queues(dev_id, + num_queues, SOCKET_ID_ANY), + "Failed test for rte_bbdev_setup_queues: " + "invalid dev_num %u", dev_id); + TEST_ASSERT(rte_bbdev_intr_enable(dev_id) == -ENODEV, + "Failed test for rte_bbdev_intr_enable: " + "invalid dev_num %u", dev_id); + break; + } + } + + return TEST_SUCCESS; +} + +static int +test_bbdev_configure_invalid_num_queues(void) +{ + struct rte_bbdev_info info; + uint8_t dev_id, num_devs; + uint8_t num_queues; + int return_value; + + TEST_ASSERT((num_devs = rte_bbdev_count()) >= 1, + "Need at least %d devices for test", 1); + + /* valid num_queues values */ + num_queues = 8; + + /* valid dev_id values */ + dev_id = null_dev_id; + + /* Stop the device in case it's started so it can be configured */ + rte_bbdev_stop(dev_id); + + TEST_ASSERT_FAIL(rte_bbdev_setup_queues(dev_id, 0, SOCKET_ID_ANY), + "Failed test for rte_bbdev_setup_queues: " + "invalid num_queues %d", 0); + + TEST_ASSERT_SUCCESS(rte_bbdev_setup_queues(dev_id, num_queues, + SOCKET_ID_ANY), + "Failed test for rte_bbdev_setup_queues: " + "invalid dev_num %u", dev_id); + + TEST_ASSERT_FAIL(return_value = rte_bbdev_info_get(dev_id, NULL), + "Failed test for rte_bbdev_info_get: " + "returned value:%i", return_value); + + TEST_ASSERT_SUCCESS(return_value = rte_bbdev_info_get(dev_id, &info), + "Failed test for rte_bbdev_info_get: " + "invalid return value:%i", return_value); + + TEST_ASSERT(info.num_queues == num_queues, + "Failed test for rte_bbdev_info_get: " + "invalid num_queues:%u", info.num_queues); + + num_queues = info.drv.max_num_queues; + TEST_ASSERT_SUCCESS(rte_bbdev_setup_queues(dev_id, num_queues, + SOCKET_ID_ANY), + "Failed test for rte_bbdev_setup_queues: " + "invalid num_queues: %u", num_queues); + + num_queues++; + TEST_ASSERT_FAIL(rte_bbdev_setup_queues(dev_id, num_queues, + SOCKET_ID_ANY), + "Failed test for rte_bbdev_setup_queues: " + "invalid num_queues: %u", num_queues); + + return TEST_SUCCESS; +} + +static int +test_bbdev_configure_stop_device(void) +{ + struct rte_bbdev_info info; + uint8_t dev_id; + int return_value; + + /* valid dev_id values */ + dev_id = null_dev_id; + + /* Stop the device so it can be configured */ + rte_bbdev_stop(dev_id); + + TEST_ASSERT_SUCCESS(return_value = rte_bbdev_info_get(dev_id, &info), + "Failed test for rte_bbdev_info_get: " + "invalid return value from " + "rte_bbdev_info_get function: %i", return_value); + + TEST_ASSERT_SUCCESS(info.started, "Failed test for rte_bbdev_info_get: " + "started value: %u", info.started); + + TEST_ASSERT_SUCCESS(rte_bbdev_setup_queues(dev_id, + info.drv.max_num_queues, SOCKET_ID_ANY), + "Failed test for rte_bbdev_setup_queues: " + "device should be stopped, dev_id: %u", dev_id); + + return_value = rte_bbdev_intr_enable(dev_id); + TEST_ASSERT(return_value != -EBUSY, + "Failed test for rte_bbdev_intr_enable: device should be stopped, dev_id: %u", + dev_id); + + /* Start the device so it cannot be configured */ + TEST_ASSERT_FAIL(rte_bbdev_start(RTE_BBDEV_MAX_DEVS), + "Failed to start bbdev %u", dev_id); + + TEST_ASSERT_SUCCESS(rte_bbdev_start(dev_id), + "Failed to start bbdev %u", dev_id); + + TEST_ASSERT_SUCCESS(return_value = rte_bbdev_info_get(dev_id, &info), + "Failed test for rte_bbdev_info_get: " + "invalid return value from " + "rte_bbdev_info_get function: %i", return_value); + + TEST_ASSERT_FAIL(info.started, "Failed test for rte_bbdev_info_get: " + "started value: %u", info.started); + + TEST_ASSERT_FAIL(rte_bbdev_setup_queues(dev_id, + info.drv.max_num_queues, SOCKET_ID_ANY), + "Failed test for rte_bbdev_setup_queues: " + "device should be started, dev_id: %u", dev_id); + + return_value = rte_bbdev_intr_enable(dev_id); + TEST_ASSERT(return_value == -EBUSY, + "Failed test for rte_bbdev_intr_enable: device should be started, dev_id: %u", + dev_id); + + /* Stop again the device so it can be once again configured */ + TEST_ASSERT_FAIL(rte_bbdev_stop(RTE_BBDEV_MAX_DEVS), + "Failed to start bbdev %u", dev_id); + + TEST_ASSERT_SUCCESS(rte_bbdev_stop(dev_id), "Failed to stop bbdev %u", + dev_id); + + TEST_ASSERT_SUCCESS(return_value = rte_bbdev_info_get(dev_id, &info), + "Failed test for rte_bbdev_info_get: " + "invalid return value from " + "rte_bbdev_info_get function: %i", return_value); + + TEST_ASSERT_SUCCESS(info.started, "Failed test for rte_bbdev_info_get: " + "started value: %u", info.started); + + TEST_ASSERT_SUCCESS(rte_bbdev_setup_queues(dev_id, + info.drv.max_num_queues, SOCKET_ID_ANY), + "Failed test for rte_bbdev_setup_queues: " + "device should be stopped, dev_id: %u", dev_id); + + return_value = rte_bbdev_intr_enable(dev_id); + TEST_ASSERT(return_value != -EBUSY, + "Failed test for rte_bbdev_intr_enable: device should be stopped, dev_id: %u", + dev_id); + + return TEST_SUCCESS; +} + +static int +test_bbdev_configure_stop_queue(void) +{ + struct bbdev_testsuite_params *ts_params = &testsuite_params; + struct rte_bbdev_info info; + struct rte_bbdev_queue_info qinfo; + uint8_t dev_id; + uint16_t queue_id; + int return_value; + + /* Valid dev_id values */ + dev_id = null_dev_id; + + /* Valid queue_id values */ + queue_id = 0; + + rte_bbdev_stop(dev_id); + TEST_ASSERT_SUCCESS(return_value = rte_bbdev_info_get(dev_id, &info), + "Failed test for rte_bbdev_info_get: " + "invalid return value:%i", return_value); + + /* Valid queue configuration */ + ts_params->qconf.queue_size = info.drv.queue_size_lim; + ts_params->qconf.priority = info.drv.max_queue_priority; + + /* Device - started; queue - started */ + rte_bbdev_start(dev_id); + + TEST_ASSERT_FAIL(rte_bbdev_queue_configure(dev_id, queue_id, + &ts_params->qconf), + "Failed test for rte_bbdev_queue_configure: " + "queue:%u on device:%u should be stopped", + queue_id, dev_id); + + /* Device - stopped; queue - started */ + rte_bbdev_stop(dev_id); + + TEST_ASSERT_FAIL(rte_bbdev_queue_configure(dev_id, queue_id, + &ts_params->qconf), + "Failed test for rte_bbdev_queue_configure: " + "queue:%u on device:%u should be stopped", + queue_id, dev_id); + + TEST_ASSERT_FAIL(rte_bbdev_queue_stop(RTE_BBDEV_MAX_DEVS, queue_id), + "Failed test for rte_bbdev_queue_stop " + "invalid dev_id "); + + TEST_ASSERT_FAIL(rte_bbdev_queue_stop(dev_id, RTE_MAX_QUEUES_PER_PORT), + "Failed test for rte_bbdev_queue_stop " + "invalid queue_id "); + + /* Device - stopped; queue - stopped */ + rte_bbdev_queue_stop(dev_id, queue_id); + + TEST_ASSERT_SUCCESS(rte_bbdev_queue_configure(dev_id, queue_id, + &ts_params->qconf), + "Failed test for rte_bbdev_queue_configure: " + "queue:%u on device:%u should be stopped", queue_id, + dev_id); + + TEST_ASSERT_SUCCESS(return_value = rte_bbdev_queue_info_get(dev_id, + queue_id, &qinfo), + "Failed test for rte_bbdev_info_get: " + "invalid return value from " + "rte_bbdev_queue_info_get function: %i", return_value); + + TEST_ASSERT(qinfo.conf.socket == ts_params->qconf.socket, + "Failed test for rte_bbdev_queue_info_get: " + "invalid queue_size:%u", qinfo.conf.socket); + + TEST_ASSERT(qinfo.conf.queue_size == ts_params->qconf.queue_size, + "Failed test for rte_bbdev_queue_info_get: " + "invalid queue_size:%u", qinfo.conf.queue_size); + + TEST_ASSERT(qinfo.conf.priority == ts_params->qconf.priority, + "Failed test for rte_bbdev_queue_info_get: " + "invalid queue_size:%u", qinfo.conf.priority); + + TEST_ASSERT(qinfo.conf.deferred_start == + ts_params->qconf.deferred_start, + "Failed test for rte_bbdev_queue_info_get: " + "invalid queue_size:%u", qinfo.conf.deferred_start); + + /* Device - started; queue - stopped */ + rte_bbdev_start(dev_id); + rte_bbdev_queue_stop(dev_id, queue_id); + + TEST_ASSERT_FAIL(rte_bbdev_queue_configure(dev_id, queue_id, + &ts_params->qconf), + "Failed test for rte_bbdev_queue_configure: " + "queue:%u on device:%u should be stopped", queue_id, + dev_id); + + rte_bbdev_stop(dev_id); + + /* After rte_bbdev_start(dev_id): + * - queue should be still stopped if deferred_start == + */ + rte_bbdev_start(dev_id); + + TEST_ASSERT_SUCCESS(return_value = rte_bbdev_queue_info_get(dev_id, + queue_id, &qinfo), + "Failed test for rte_bbdev_info_get: " + "invalid return value from " + "rte_bbdev_queue_info_get function: %i", return_value); + + TEST_ASSERT(qinfo.started == 0, + "Failed test for rte_bbdev_queue_info_get: " + "invalid value for qinfo.started:%u", qinfo.started); + + rte_bbdev_stop(dev_id); + + /* After rte_bbdev_start(dev_id): + * - queue should be started if deferred_start == + */ + ts_params->qconf.deferred_start = 0; + rte_bbdev_queue_configure(dev_id, queue_id, &ts_params->qconf); + rte_bbdev_start(dev_id); + + TEST_ASSERT_SUCCESS(return_value = rte_bbdev_queue_info_get(dev_id, + queue_id, &qinfo), + "Failed test for rte_bbdev_info_get: " + "invalid return value from " + "rte_bbdev_queue_info_get function: %i", return_value); + + TEST_ASSERT(qinfo.started == 1, + "Failed test for rte_bbdev_queue_info_get: " + "invalid value for qinfo.started:%u", qinfo.started); + + return TEST_SUCCESS; +} + +static int +test_bbdev_configure_invalid_queue_configure(void) +{ + struct bbdev_testsuite_params *ts_params = &testsuite_params; + int return_value; + struct rte_bbdev_info info; + uint8_t dev_id; + uint16_t queue_id; + + /* Valid dev_id values */ + dev_id = null_dev_id; + + /* Valid queue_id values */ + queue_id = 0; + + rte_bbdev_stop(dev_id); + + TEST_ASSERT_SUCCESS(return_value = rte_bbdev_info_get(dev_id, &info), + "Failed test for rte_bbdev_info_get: " + "invalid return value:%i", return_value); + + rte_bbdev_queue_stop(dev_id, queue_id); + + ts_params->qconf.queue_size = info.drv.queue_size_lim + 1; + TEST_ASSERT_FAIL(rte_bbdev_queue_configure(dev_id, queue_id, + &ts_params->qconf), + "Failed test for rte_bbdev_queue_configure: " + "invalid value qconf.queue_size: %u", + ts_params->qconf.queue_size); + + ts_params->qconf.queue_size = info.drv.queue_size_lim; + ts_params->qconf.priority = info.drv.max_queue_priority + 1; + TEST_ASSERT_FAIL(rte_bbdev_queue_configure(dev_id, queue_id, + &ts_params->qconf), + "Failed test for rte_bbdev_queue_configure: " + "invalid value qconf.queue_size: %u", + ts_params->qconf.queue_size); + + ts_params->qconf.priority = info.drv.max_queue_priority; + queue_id = info.num_queues; + TEST_ASSERT_FAIL(rte_bbdev_queue_configure(dev_id, queue_id, + &ts_params->qconf), + "Failed test for rte_bbdev_queue_configure: " + "invalid value queue_id: %u", queue_id); + + queue_id = 0; + TEST_ASSERT_SUCCESS(rte_bbdev_queue_configure(dev_id, queue_id, NULL), + "Failed test for rte_bbdev_queue_configure: " + "NULL qconf structure "); + + ts_params->qconf.socket = RTE_MAX_NUMA_NODES; + TEST_ASSERT_FAIL(rte_bbdev_queue_configure(dev_id, queue_id, + &ts_params->qconf), + "Failed test for rte_bbdev_queue_configure: " + "invalid socket number "); + + ts_params->qconf.socket = SOCKET_ID_ANY; + TEST_ASSERT_SUCCESS(rte_bbdev_queue_configure(dev_id, queue_id, + &ts_params->qconf), + "Failed test for rte_bbdev_queue_configure: " + "invalid value qconf.queue_size: %u", + ts_params->qconf.queue_size); + + TEST_ASSERT_FAIL(rte_bbdev_queue_configure(RTE_BBDEV_MAX_DEVS, queue_id, + &ts_params->qconf), + "Failed test for rte_bbdev_queue_configure: " + "invalid dev_id"); + + TEST_ASSERT_SUCCESS(rte_bbdev_queue_configure(dev_id, queue_id, NULL), + "Failed test for rte_bbdev_queue_configure: " + "invalid value qconf.queue_size: %u", + ts_params->qconf.queue_size); + + return TEST_SUCCESS; +} + +static int +test_bbdev_op_pool(void) +{ + struct rte_mempool *mp; + + unsigned int dec_size = sizeof(struct rte_bbdev_dec_op); + unsigned int enc_size = sizeof(struct rte_bbdev_enc_op); + + const char *pool_dec = "Test_DEC"; + const char *pool_enc = "Test_ENC"; + + /* Valid pool configuration */ + uint32_t size = 256; + uint32_t cache_size = 128; + + TEST_ASSERT(rte_bbdev_op_pool_create(NULL, + RTE_BBDEV_OP_TURBO_DEC, size, cache_size, 0) == NULL, + "Failed test for rte_bbdev_op_pool_create: " + "NULL name parameter"); + + TEST_ASSERT((mp = rte_bbdev_op_pool_create(pool_dec, + RTE_BBDEV_OP_TURBO_DEC, size, cache_size, 0)) != NULL, + "Failed test for rte_bbdev_op_pool_create: " + "returned value is empty"); + + TEST_ASSERT(mp->size == size, + "Failed test for rte_bbdev_op_pool_create: " + "invalid size of the mempool, mp->size: %u", mp->size); + + TEST_ASSERT(mp->cache_size == cache_size, + "Failed test for rte_bbdev_op_pool_create: " + "invalid size of the mempool, mp->size: %u", + mp->cache_size); + + TEST_ASSERT_SUCCESS(strcmp(mp->name, pool_dec), + "Failed test for rte_bbdev_op_pool_create: " + "invalid name of mempool, mp->name: %s", mp->name); + + TEST_ASSERT(mp->elt_size == dec_size, + "Failed test for rte_bbdev_op_pool_create: " + "invalid element size for RTE_BBDEV_OP_TURBO_DEC, " + "mp->elt_size: %u", mp->elt_size); + + rte_mempool_free(mp); + + TEST_ASSERT((mp = rte_bbdev_op_pool_create(pool_enc, + RTE_BBDEV_OP_TURBO_ENC, size, cache_size, 0)) != NULL, + "Failed test for rte_bbdev_op_pool_create: " + "returned value is empty"); + + TEST_ASSERT(mp->elt_size == enc_size, + "Failed test for rte_bbdev_op_pool_create: " + "invalid element size for RTE_BBDEV_OP_TURBO_ENC, " + "mp->elt_size: %u", mp->elt_size); + + rte_mempool_free(mp); + + TEST_ASSERT((mp = rte_bbdev_op_pool_create("Test_NONE", + RTE_BBDEV_OP_NONE, size, cache_size, 0)) != NULL, + "Failed test for rte_bbdev_op_pool_create: " + "returned value is empty for RTE_BBDEV_OP_NONE"); + + TEST_ASSERT(mp->elt_size == (enc_size > dec_size ? enc_size : dec_size), + "Failed test for rte_bbdev_op_pool_create: " + "invalid size for RTE_BBDEV_OP_NONE, mp->elt_size: %u", + mp->elt_size); + + rte_mempool_free(mp); + + TEST_ASSERT((mp = rte_bbdev_op_pool_create("Test_INV", + RTE_BBDEV_OP_TYPE_COUNT, size, cache_size, 0)) == NULL, + "Failed test for rte_bbdev_op_pool_create: " + "returned value is not NULL for invalid type"); + + /* Invalid pool configuration */ + size = 128; + cache_size = 256; + + TEST_ASSERT((mp = rte_bbdev_op_pool_create("Test_InvSize", + RTE_BBDEV_OP_NONE, size, cache_size, 0)) == NULL, + "Failed test for rte_bbdev_op_pool_create: " + "returned value should be empty " + "because size of per-lcore local cache " + "is greater than size of the mempool."); + + return TEST_SUCCESS; +} + +/** + * Create pool of OP types RTE_BBDEV_OP_NONE, RTE_BBDEV_OP_TURBO_DEC and + * RTE_BBDEV_OP_TURBO_ENC and check that only ops of that type can be + * allocated + */ +static int +test_bbdev_op_type(void) +{ + struct rte_mempool *mp_dec; + + const unsigned int OPS_COUNT = 32; + struct rte_bbdev_dec_op *dec_ops_arr[OPS_COUNT]; + struct rte_bbdev_enc_op *enc_ops_arr[OPS_COUNT]; + + const char *pool_dec = "Test_op_dec"; + + /* Valid pool configuration */ + uint32_t num_elements = 256; + uint32_t cache_size = 128; + + /* mempool type : RTE_BBDEV_OP_TURBO_DEC */ + mp_dec = rte_bbdev_op_pool_create(pool_dec, + RTE_BBDEV_OP_TURBO_DEC, num_elements, cache_size, 0); + TEST_ASSERT(mp_dec != NULL, "Failed to create %s mempool", pool_dec); + + TEST_ASSERT(rte_bbdev_dec_op_alloc_bulk(mp_dec, dec_ops_arr, 1) == 0, + "Failed test for rte_bbdev_op_alloc_bulk TURBO_DEC: " + "OPs type: RTE_BBDEV_OP_TURBO_DEC"); + + TEST_ASSERT(rte_bbdev_enc_op_alloc_bulk(mp_dec, enc_ops_arr, 1) != 0, + "Failed test for rte_bbdev_op_alloc_bulk TURBO_DEC: " + "OPs type: RTE_BBDEV_OP_TURBO_ENC"); + + rte_mempool_free(mp_dec); + + return TEST_SUCCESS; +} + +static int +test_bbdev_op_pool_size(void) +{ + struct rte_mempool *mp_none; + + const unsigned int OPS_COUNT = 128; + struct rte_bbdev_enc_op *ops_enc_arr[OPS_COUNT]; + struct rte_bbdev_enc_op *ops_ext_arr[OPS_COUNT]; + struct rte_bbdev_enc_op *ops_ext2_arr[OPS_COUNT]; + + const char *pool_none = "Test_pool_size"; + + /* Valid pool configuration */ + uint32_t num_elements = 256; + uint32_t cache_size = 0; + + /* Create mempool type : RTE_BBDEV_OP_TURBO_ENC, size : 256 */ + mp_none = rte_bbdev_op_pool_create(pool_none, RTE_BBDEV_OP_TURBO_ENC, + num_elements, cache_size, 0); + TEST_ASSERT(mp_none != NULL, "Failed to create %s mempool", pool_none); + + /* Add 128 RTE_BBDEV_OP_TURBO_ENC ops */ + rte_bbdev_enc_op_alloc_bulk(mp_none, ops_enc_arr, OPS_COUNT); + + /* Add 128 RTE_BBDEV_OP_TURBO_ENC ops */ + TEST_ASSERT(rte_bbdev_enc_op_alloc_bulk(mp_none, ops_ext_arr, + OPS_COUNT) == 0, + "Failed test for allocating bbdev ops: " + "Mempool size: 256, Free : 128, Attempted to add: 128"); + + /* Try adding 128 more RTE_BBDEV_OP_TURBO_ENC ops, this should fail */ + TEST_ASSERT(rte_bbdev_enc_op_alloc_bulk(mp_none, ops_ext2_arr, + OPS_COUNT) != 0, + "Failed test for allocating bbdev ops: " + "Mempool size: 256, Free : 0, Attempted to add: 128"); + + /* Free-up 128 RTE_BBDEV_OP_TURBO_ENC ops */ + rte_bbdev_enc_op_free_bulk(ops_enc_arr, OPS_COUNT); + + /* Try adding 128 RTE_BBDEV_OP_TURBO_DEC ops, this should succeed */ + /* Cache size > 0 causes reallocation of ops size > 127 fail */ + TEST_ASSERT(rte_bbdev_enc_op_alloc_bulk(mp_none, ops_ext2_arr, + OPS_COUNT) == 0, + "Failed test for allocating ops after mempool freed: " + "Mempool size: 256, Free : 128, Attempted to add: 128"); + + rte_mempool_free(mp_none); + + return TEST_SUCCESS; +} + +static int +test_bbdev_count(void) +{ + uint8_t num_devs, num_valid_devs = 0; + + for (num_devs = 0; num_devs < RTE_BBDEV_MAX_DEVS; num_devs++) { + if (rte_bbdev_is_valid(num_devs)) + num_valid_devs++; + } + + num_devs = rte_bbdev_count(); + TEST_ASSERT(num_valid_devs == num_devs, + "Failed test for rte_bbdev_is_valid: " + "invalid num_devs %u ", num_devs); + + return TEST_SUCCESS; +} + +static int +test_bbdev_stats(void) +{ + uint8_t dev_id = null_dev_id; + uint16_t queue_id = 0; + struct rte_bbdev_dec_op *dec_ops[4096] = { 0 }; + struct rte_bbdev_dec_op *dec_proc_ops[4096] = { 0 }; + struct rte_bbdev_enc_op *enc_ops[4096] = { 0 }; + struct rte_bbdev_enc_op *enc_proc_ops[4096] = { 0 }; + uint16_t num_ops = 236; + struct rte_bbdev_stats stats; + struct bbdev_testsuite_params *ts_params = &testsuite_params; + + TEST_ASSERT_SUCCESS(rte_bbdev_queue_stop(dev_id, queue_id), + "Failed to stop queue %u on device %u ", queue_id, + dev_id); + TEST_ASSERT_SUCCESS(rte_bbdev_stop(dev_id), + "Failed to stop bbdev %u ", dev_id); + + TEST_ASSERT_SUCCESS(rte_bbdev_queue_configure(dev_id, queue_id, + &ts_params->qconf), + "Failed to configure queue %u on device %u ", + queue_id, dev_id); + + TEST_ASSERT_SUCCESS(rte_bbdev_start(dev_id), + "Failed to start bbdev %u ", dev_id); + + TEST_ASSERT_SUCCESS(rte_bbdev_queue_start(dev_id, queue_id), + "Failed to start queue %u on device %u ", queue_id, + dev_id); + + TEST_ASSERT_SUCCESS(rte_bbdev_queue_start(dev_id, queue_id), + "Failed to start queue %u on device %u ", queue_id, + dev_id); + + /* Tests after enqueue operation */ + rte_bbdev_enqueue_enc_ops(dev_id, queue_id, enc_ops, num_ops); + rte_bbdev_enqueue_dec_ops(dev_id, queue_id, dec_ops, num_ops); + + TEST_ASSERT_FAIL(rte_bbdev_stats_get(RTE_BBDEV_MAX_DEVS, &stats), + "Failed test for rte_bbdev_stats_get on device %u ", + dev_id); + + TEST_ASSERT_FAIL(rte_bbdev_stats_get(dev_id, NULL), + "Failed test for rte_bbdev_stats_get on device %u ", + dev_id); + + TEST_ASSERT_SUCCESS(rte_bbdev_stats_get(dev_id, &stats), + "Failed test for rte_bbdev_stats_get on device %u ", + dev_id); + + TEST_ASSERT(stats.enqueued_count == 2 * num_ops, + "Failed test for rte_bbdev_enqueue_ops: " + "invalid enqueued_count %" PRIu64 " ", + stats.enqueued_count); + + TEST_ASSERT(stats.dequeued_count == 0, + "Failed test for rte_bbdev_stats_reset: " + "invalid dequeued_count %" PRIu64 " ", + stats.dequeued_count); + + /* Tests after dequeue operation */ + rte_bbdev_dequeue_enc_ops(dev_id, queue_id, enc_proc_ops, num_ops); + rte_bbdev_dequeue_dec_ops(dev_id, queue_id, dec_proc_ops, num_ops); + + TEST_ASSERT_SUCCESS(rte_bbdev_stats_get(dev_id, &stats), + "Failed test for rte_bbdev_stats_get on device %u ", + dev_id); + + TEST_ASSERT(stats.dequeued_count == 2 * num_ops, + "Failed test for rte_bbdev_dequeue_ops: " + "invalid enqueued_count %" PRIu64 " ", + stats.dequeued_count); + + TEST_ASSERT(stats.enqueue_err_count == 0, + "Failed test for rte_bbdev_stats_reset: " + "invalid enqueue_err_count %" PRIu64 " ", + stats.enqueue_err_count); + + TEST_ASSERT(stats.dequeue_err_count == 0, + "Failed test for rte_bbdev_stats_reset: " + "invalid dequeue_err_count %" PRIu64 " ", + stats.dequeue_err_count); + + /* Tests after reset operation */ + TEST_ASSERT_FAIL(rte_bbdev_stats_reset(RTE_BBDEV_MAX_DEVS), + "Failed to reset statistic for device %u ", dev_id); + + TEST_ASSERT_SUCCESS(rte_bbdev_stats_reset(dev_id), + "Failed to reset statistic for device %u ", dev_id); + TEST_ASSERT_SUCCESS(rte_bbdev_stats_get(dev_id, &stats), + "Failed test for rte_bbdev_stats_get on device %u ", + dev_id); + + TEST_ASSERT(stats.enqueued_count == 0, + "Failed test for rte_bbdev_stats_reset: " + "invalid enqueued_count %" PRIu64 " ", + stats.enqueued_count); + + TEST_ASSERT(stats.dequeued_count == 0, + "Failed test for rte_bbdev_stats_reset: " + "invalid dequeued_count %" PRIu64 " ", + stats.dequeued_count); + + TEST_ASSERT(stats.enqueue_err_count == 0, + "Failed test for rte_bbdev_stats_reset: " + "invalid enqueue_err_count %" PRIu64 " ", + stats.enqueue_err_count); + + TEST_ASSERT(stats.dequeue_err_count == 0, + "Failed test for rte_bbdev_stats_reset: " + "invalid dequeue_err_count %" PRIu64 " ", + stats.dequeue_err_count); + + return TEST_SUCCESS; +} + +static int +test_bbdev_driver_init(void) +{ + struct rte_bbdev *dev1, *dev2; + const char *name = "dev_name"; + char name_tmp[16]; + int num_devs, num_devs_tmp; + + dev1 = rte_bbdev_allocate(NULL); + TEST_ASSERT(dev1 == NULL, + "Failed initialize bbdev driver with NULL name"); + + dev1 = rte_bbdev_allocate(name); + TEST_ASSERT(dev1 != NULL, "Failed to initialize bbdev driver"); + + dev2 = rte_bbdev_allocate(name); + TEST_ASSERT(dev2 == NULL, + "Failed to initialize bbdev driver: " + "driver with the same name has been initialized before"); + + num_devs = rte_bbdev_count() - 1; + num_devs_tmp = num_devs; + + /* Initialize the maximum amount of devices */ + do { + sprintf(name_tmp, "%s%i", "name_", num_devs); + dev2 = rte_bbdev_allocate(name_tmp); + TEST_ASSERT(dev2 != NULL, + "Failed to initialize bbdev driver"); + ++num_devs; + } while (num_devs < (RTE_BBDEV_MAX_DEVS - 1)); + + sprintf(name_tmp, "%s%i", "name_", num_devs); + dev2 = rte_bbdev_allocate(name_tmp); + TEST_ASSERT(dev2 == NULL, "Failed to initialize bbdev driver number %d " + "more drivers than RTE_BBDEV_MAX_DEVS: %d ", num_devs, + RTE_BBDEV_MAX_DEVS); + + num_devs--; + + while (num_devs >= num_devs_tmp) { + sprintf(name_tmp, "%s%i", "name_", num_devs); + dev2 = rte_bbdev_get_named_dev(name_tmp); + TEST_ASSERT_SUCCESS(rte_bbdev_release(dev2), + "Failed to uninitialize bbdev driver %s ", + name_tmp); + num_devs--; + } + + TEST_ASSERT(dev1->data->dev_id < RTE_BBDEV_MAX_DEVS, + "Failed test rte_bbdev_allocate: " + "invalid dev_id %" PRIu8 ", max number of devices %d ", + dev1->data->dev_id, RTE_BBDEV_MAX_DEVS); + + TEST_ASSERT(dev1->state == RTE_BBDEV_INITALIZED, + "Failed test rte_bbdev_allocate: " + "invalid state %d (0 - RTE_BBDEV_UNUSED, 1 - RTE_BBDEV_INITALIZED", + dev1->state); + + TEST_ASSERT_FAIL(rte_bbdev_release(NULL), + "Failed to uninitialize bbdev driver with NULL bbdev"); + + sprintf(name_tmp, "%s", "invalid_name"); + dev2 = rte_bbdev_get_named_dev(name_tmp); + TEST_ASSERT_FAIL(rte_bbdev_release(dev2), + "Failed to uninitialize bbdev driver with invalid name"); + + dev2 = rte_bbdev_get_named_dev(name); + TEST_ASSERT_SUCCESS(rte_bbdev_release(dev2), + "Failed to uninitialize bbdev driver: %s ", name); + + return TEST_SUCCESS; +} + +static void +event_callback(uint16_t dev_id, enum rte_bbdev_event_type type, void *param, + void *ret_param) +{ + RTE_SET_USED(dev_id); + RTE_SET_USED(ret_param); + + if (param == NULL) + return; + + if (type == RTE_BBDEV_EVENT_UNKNOWN || + type == RTE_BBDEV_EVENT_ERROR || + type == RTE_BBDEV_EVENT_MAX) + *(int *)param = type; +} + +static int +test_bbdev_callback(void) +{ + struct rte_bbdev *dev1, *dev2; + const char *name = "dev_name1"; + const char *name2 = "dev_name2"; + int event_status; + uint8_t invalid_dev_id = 7; + enum rte_bbdev_event_type invalid_event_type = RTE_BBDEV_EVENT_MAX; + uint8_t dev_id; + + dev1 = rte_bbdev_allocate(name); + TEST_ASSERT(dev1 != NULL, "Failed to initialize bbdev driver"); + + /* + * RTE_BBDEV_EVENT_UNKNOWN - unregistered + * RTE_BBDEV_EVENT_ERROR - unregistered + */ + event_status = -1; + rte_bbdev_pmd_callback_process(dev1, RTE_BBDEV_EVENT_UNKNOWN, NULL); + rte_bbdev_pmd_callback_process(dev1, RTE_BBDEV_EVENT_ERROR, NULL); + TEST_ASSERT(event_status == -1, + "Failed test for rte_bbdev_pmd_callback_process: " + "events were not registered "); + + TEST_ASSERT_FAIL(rte_bbdev_callback_register(dev1->data->dev_id, + RTE_BBDEV_EVENT_MAX, event_callback, NULL), + "Failed to callback register for RTE_BBDEV_EVENT_MAX "); + + TEST_ASSERT_FAIL(rte_bbdev_callback_unregister(dev1->data->dev_id, + RTE_BBDEV_EVENT_MAX, event_callback, NULL), + "Failed to unregister RTE_BBDEV_EVENT_MAX "); + + /* + * RTE_BBDEV_EVENT_UNKNOWN - registered + * RTE_BBDEV_EVENT_ERROR - unregistered + */ + TEST_ASSERT_SUCCESS(rte_bbdev_callback_register(dev1->data->dev_id, + RTE_BBDEV_EVENT_UNKNOWN, event_callback, &event_status), + "Failed to callback rgstr for RTE_BBDEV_EVENT_UNKNOWN"); + + rte_bbdev_pmd_callback_process(dev1, RTE_BBDEV_EVENT_UNKNOWN, NULL); + TEST_ASSERT(event_status == 0, + "Failed test for rte_bbdev_pmd_callback_process " + "for RTE_BBDEV_EVENT_UNKNOWN "); + + rte_bbdev_pmd_callback_process(dev1, RTE_BBDEV_EVENT_ERROR, NULL); + TEST_ASSERT(event_status == 0, + "Failed test for rte_bbdev_pmd_callback_process: " + "event RTE_BBDEV_EVENT_ERROR was not registered "); + + /* + * RTE_BBDEV_EVENT_UNKNOWN - registered + * RTE_BBDEV_EVENT_ERROR - registered + */ + TEST_ASSERT_SUCCESS(rte_bbdev_callback_register(dev1->data->dev_id, + RTE_BBDEV_EVENT_ERROR, event_callback, &event_status), + "Failed to callback rgstr for RTE_BBDEV_EVENT_ERROR "); + + TEST_ASSERT_SUCCESS(rte_bbdev_callback_register(dev1->data->dev_id, + RTE_BBDEV_EVENT_ERROR, event_callback, &event_status), + "Failed to callback register for RTE_BBDEV_EVENT_ERROR" + "(re-registration) "); + + event_status = -1; + rte_bbdev_pmd_callback_process(dev1, RTE_BBDEV_EVENT_UNKNOWN, NULL); + TEST_ASSERT(event_status == 0, + "Failed test for rte_bbdev_pmd_callback_process " + "for RTE_BBDEV_EVENT_UNKNOWN "); + + rte_bbdev_pmd_callback_process(dev1, RTE_BBDEV_EVENT_ERROR, NULL); + TEST_ASSERT(event_status == 1, + "Failed test for rte_bbdev_pmd_callback_process " + "for RTE_BBDEV_EVENT_ERROR "); + + /* + * RTE_BBDEV_EVENT_UNKNOWN - registered + * RTE_BBDEV_EVENT_ERROR - unregistered + */ + TEST_ASSERT_SUCCESS(rte_bbdev_callback_unregister(dev1->data->dev_id, + RTE_BBDEV_EVENT_ERROR, event_callback, &event_status), + "Failed to unregister RTE_BBDEV_EVENT_ERROR "); + + event_status = -1; + rte_bbdev_pmd_callback_process(dev1, RTE_BBDEV_EVENT_UNKNOWN, NULL); + TEST_ASSERT(event_status == 0, + "Failed test for rte_bbdev_pmd_callback_process " + "for RTE_BBDEV_EVENT_UNKNOWN "); + + rte_bbdev_pmd_callback_process(dev1, RTE_BBDEV_EVENT_ERROR, NULL); + TEST_ASSERT(event_status == 0, + "Failed test for rte_bbdev_pmd_callback_process: " + "event RTE_BBDEV_EVENT_ERROR was unregistered "); + + /* rte_bbdev_callback_register with invalid inputs */ + TEST_ASSERT_FAIL(rte_bbdev_callback_register(invalid_dev_id, + RTE_BBDEV_EVENT_ERROR, event_callback, &event_status), + "Failed test for rte_bbdev_callback_register " + "for invalid_dev_id "); + + TEST_ASSERT_FAIL(rte_bbdev_callback_register(dev1->data->dev_id, + invalid_event_type, event_callback, &event_status), + "Failed to callback register for invalid event type "); + + TEST_ASSERT_FAIL(rte_bbdev_callback_register(dev1->data->dev_id, + RTE_BBDEV_EVENT_ERROR, NULL, &event_status), + "Failed to callback register - no callback function "); + + /* The impact of devices on each other */ + dev2 = rte_bbdev_allocate(name2); + TEST_ASSERT(dev2 != NULL, + "Failed to initialize bbdev driver"); + + /* + * dev2: + * RTE_BBDEV_EVENT_UNKNOWN - unregistered + * RTE_BBDEV_EVENT_ERROR - unregistered + */ + event_status = -1; + rte_bbdev_pmd_callback_process(dev2, RTE_BBDEV_EVENT_UNKNOWN, NULL); + rte_bbdev_pmd_callback_process(dev2, RTE_BBDEV_EVENT_ERROR, NULL); + TEST_ASSERT(event_status == -1, + "Failed test for rte_bbdev_pmd_callback_process: " + "events were not registered "); + + /* + * dev1: RTE_BBDEV_EVENT_ERROR - unregistered + * dev2: RTE_BBDEV_EVENT_ERROR - registered + */ + TEST_ASSERT_SUCCESS(rte_bbdev_callback_register(dev2->data->dev_id, + RTE_BBDEV_EVENT_ERROR, event_callback, &event_status), + "Failed to callback rgstr for RTE_BBDEV_EVENT_ERROR"); + + rte_bbdev_pmd_callback_process(dev1, RTE_BBDEV_EVENT_ERROR, NULL); + TEST_ASSERT(event_status == -1, + "Failed test for rte_bbdev_pmd_callback_process in dev1 " + "for RTE_BBDEV_EVENT_ERROR "); + + rte_bbdev_pmd_callback_process(dev2, RTE_BBDEV_EVENT_ERROR, NULL); + TEST_ASSERT(event_status == 1, + "Failed test for rte_bbdev_pmd_callback_process in dev2 " + "for RTE_BBDEV_EVENT_ERROR "); + + /* + * dev1: RTE_BBDEV_EVENT_UNKNOWN - registered + * dev2: RTE_BBDEV_EVENT_UNKNOWN - unregistered + */ + TEST_ASSERT_SUCCESS(rte_bbdev_callback_register(dev2->data->dev_id, + RTE_BBDEV_EVENT_UNKNOWN, event_callback, &event_status), + "Failed to callback register for RTE_BBDEV_EVENT_UNKNOWN " + "in dev 2 "); + + rte_bbdev_pmd_callback_process(dev2, RTE_BBDEV_EVENT_UNKNOWN, NULL); + TEST_ASSERT(event_status == 0, + "Failed test for rte_bbdev_pmd_callback_process in dev2" + " for RTE_BBDEV_EVENT_UNKNOWN "); + + TEST_ASSERT_SUCCESS(rte_bbdev_callback_unregister(dev2->data->dev_id, + RTE_BBDEV_EVENT_UNKNOWN, event_callback, &event_status), + "Failed to unregister RTE_BBDEV_EVENT_UNKNOWN "); + + TEST_ASSERT_SUCCESS(rte_bbdev_callback_unregister(dev2->data->dev_id, + RTE_BBDEV_EVENT_UNKNOWN, event_callback, &event_status), + "Failed to unregister RTE_BBDEV_EVENT_UNKNOWN : " + "unregister function called once again "); + + event_status = -1; + rte_bbdev_pmd_callback_process(dev2, RTE_BBDEV_EVENT_UNKNOWN, NULL); + TEST_ASSERT(event_status == -1, + "Failed test for rte_bbdev_pmd_callback_process in dev2" + " for RTE_BBDEV_EVENT_UNKNOWN "); + + rte_bbdev_pmd_callback_process(dev1, RTE_BBDEV_EVENT_UNKNOWN, NULL); + TEST_ASSERT(event_status == 0, + "Failed test for rte_bbdev_pmd_callback_process in dev2 " + "for RTE_BBDEV_EVENT_UNKNOWN "); + + /* rte_bbdev_pmd_callback_process with invalid inputs */ + rte_bbdev_pmd_callback_process(NULL, RTE_BBDEV_EVENT_UNKNOWN, NULL); + + event_status = -1; + rte_bbdev_pmd_callback_process(dev1, invalid_event_type, NULL); + TEST_ASSERT(event_status == -1, + "Failed test for rte_bbdev_pmd_callback_process: " + "for invalid event type "); + + /* rte_dev_callback_unregister with invalid inputs */ + TEST_ASSERT_FAIL(rte_bbdev_callback_unregister(invalid_dev_id, + RTE_BBDEV_EVENT_UNKNOWN, event_callback, &event_status), + "Failed test for rte_dev_callback_unregister " + "for invalid_dev_id "); + + TEST_ASSERT_FAIL(rte_bbdev_callback_unregister(dev1->data->dev_id, + invalid_event_type, event_callback, &event_status), + "Failed rte_dev_callback_unregister " + "for invalid event type "); + + TEST_ASSERT_FAIL(rte_bbdev_callback_unregister(dev1->data->dev_id, + invalid_event_type, NULL, &event_status), + "Failed rte_dev_callback_unregister " + "when no callback function "); + + dev_id = dev1->data->dev_id; + + rte_bbdev_release(dev1); + rte_bbdev_release(dev2); + + TEST_ASSERT_FAIL(rte_bbdev_callback_register(dev_id, + RTE_BBDEV_EVENT_ERROR, event_callback, &event_status), + "Failed test for rte_bbdev_callback_register: " + "function called after rte_bbdev_driver_uninit ."); + + TEST_ASSERT_FAIL(rte_bbdev_callback_unregister(dev_id, + RTE_BBDEV_EVENT_ERROR, event_callback, &event_status), + "Failed test for rte_dev_callback_unregister: " + "function called after rte_bbdev_driver_uninit. "); + + event_status = -1; + rte_bbdev_pmd_callback_process(dev1, RTE_BBDEV_EVENT_UNKNOWN, NULL); + rte_bbdev_pmd_callback_process(dev1, RTE_BBDEV_EVENT_ERROR, NULL); + rte_bbdev_pmd_callback_process(dev2, RTE_BBDEV_EVENT_UNKNOWN, NULL); + rte_bbdev_pmd_callback_process(dev2, RTE_BBDEV_EVENT_ERROR, NULL); + TEST_ASSERT(event_status == -1, + "Failed test for rte_bbdev_pmd_callback_process: " + "callback function was called after rte_bbdev_driver_uninit"); + + return TEST_SUCCESS; +} + +static int +test_bbdev_invalid_driver(void) +{ + struct rte_bbdev dev1, *dev2; + uint8_t dev_id = null_dev_id; + uint16_t queue_id = 0; + struct rte_bbdev_stats stats; + struct bbdev_testsuite_params *ts_params = &testsuite_params; + struct rte_bbdev_queue_info qinfo; + struct rte_bbdev_ops dev_ops_tmp; + + TEST_ASSERT_SUCCESS(rte_bbdev_stop(dev_id), "Failed to stop bbdev %u ", + dev_id); + + dev1 = rte_bbdev_devices[dev_id]; + dev2 = &rte_bbdev_devices[dev_id]; + + /* Tests for rte_bbdev_setup_queues */ + dev2->dev_ops = NULL; + TEST_ASSERT_FAIL(rte_bbdev_setup_queues(dev_id, 1, SOCKET_ID_ANY), + "Failed test for rte_bbdev_setup_queues: " + "NULL dev_ops structure "); + dev2->dev_ops = dev1.dev_ops; + + dev_ops_tmp = *dev2->dev_ops; + dev_ops_tmp.info_get = NULL; + dev2->dev_ops = &dev_ops_tmp; + TEST_ASSERT_FAIL(rte_bbdev_setup_queues(dev_id, 1, SOCKET_ID_ANY), + "Failed test for rte_bbdev_setup_queues: " + "NULL info_get "); + dev2->dev_ops = dev1.dev_ops; + + dev_ops_tmp = *dev2->dev_ops; + dev_ops_tmp.queue_release = NULL; + dev2->dev_ops = &dev_ops_tmp; + TEST_ASSERT_FAIL(rte_bbdev_setup_queues(dev_id, 1, SOCKET_ID_ANY), + "Failed test for rte_bbdev_setup_queues: " + "NULL queue_release "); + dev2->dev_ops = dev1.dev_ops; + + dev2->data->socket_id = SOCKET_ID_ANY; + TEST_ASSERT_SUCCESS(rte_bbdev_setup_queues(dev_id, 1, + SOCKET_ID_ANY), "Failed to configure bbdev %u", dev_id); + + /* Test for rte_bbdev_queue_configure */ + dev2->dev_ops = NULL; + TEST_ASSERT_FAIL(rte_bbdev_queue_configure(dev_id, queue_id, + &ts_params->qconf), + "Failed to configure queue %u on device %u " + "with NULL dev_ops structure ", queue_id, dev_id); + dev2->dev_ops = dev1.dev_ops; + + dev_ops_tmp = *dev2->dev_ops; + dev_ops_tmp.queue_setup = NULL; + dev2->dev_ops = &dev_ops_tmp; + TEST_ASSERT_FAIL(rte_bbdev_queue_configure(dev_id, queue_id, + &ts_params->qconf), + "Failed to configure queue %u on device %u " + "with NULL queue_setup ", queue_id, dev_id); + dev2->dev_ops = dev1.dev_ops; + + dev_ops_tmp = *dev2->dev_ops; + dev_ops_tmp.info_get = NULL; + dev2->dev_ops = &dev_ops_tmp; + TEST_ASSERT_FAIL(rte_bbdev_queue_configure(dev_id, queue_id, + &ts_params->qconf), + "Failed to configure queue %u on device %u " + "with NULL info_get ", queue_id, dev_id); + dev2->dev_ops = dev1.dev_ops; + + TEST_ASSERT_FAIL(rte_bbdev_queue_configure(RTE_BBDEV_MAX_DEVS, + queue_id, &ts_params->qconf), + "Failed to configure queue %u on device %u ", + queue_id, dev_id); + + TEST_ASSERT_SUCCESS(rte_bbdev_queue_configure(dev_id, queue_id, + &ts_params->qconf), + "Failed to configure queue %u on device %u ", + queue_id, dev_id); + + /* Test for rte_bbdev_queue_info_get */ + dev2->dev_ops = NULL; + TEST_ASSERT_SUCCESS(rte_bbdev_queue_info_get(dev_id, queue_id, &qinfo), + "Failed test for rte_bbdev_info_get: " + "NULL dev_ops structure "); + dev2->dev_ops = dev1.dev_ops; + + TEST_ASSERT_FAIL(rte_bbdev_queue_info_get(RTE_BBDEV_MAX_DEVS, + queue_id, &qinfo), + "Failed test for rte_bbdev_info_get: " + "invalid dev_id "); + + TEST_ASSERT_FAIL(rte_bbdev_queue_info_get(dev_id, + RTE_MAX_QUEUES_PER_PORT, &qinfo), + "Failed test for rte_bbdev_info_get: " + "invalid queue_id "); + + TEST_ASSERT_FAIL(rte_bbdev_queue_info_get(dev_id, queue_id, NULL), + "Failed test for rte_bbdev_info_get: " + "invalid dev_info "); + + /* Test for rte_bbdev_start */ + dev2->dev_ops = NULL; + TEST_ASSERT_FAIL(rte_bbdev_start(dev_id), + "Failed to start bbdev %u " + "with NULL dev_ops structure ", dev_id); + dev2->dev_ops = dev1.dev_ops; + + TEST_ASSERT_SUCCESS(rte_bbdev_start(dev_id), + "Failed to start bbdev %u ", dev_id); + + /* Test for rte_bbdev_queue_start */ + dev2->dev_ops = NULL; + TEST_ASSERT_FAIL(rte_bbdev_queue_start(dev_id, queue_id), + "Failed to start queue %u on device %u: " + "NULL dev_ops structure", queue_id, dev_id); + dev2->dev_ops = dev1.dev_ops; + + TEST_ASSERT_SUCCESS(rte_bbdev_queue_start(dev_id, queue_id), + "Failed to start queue %u on device %u ", queue_id, + dev_id); + + /* Tests for rte_bbdev_stats_get */ + dev2->dev_ops = NULL; + TEST_ASSERT_FAIL(rte_bbdev_stats_get(dev_id, &stats), + "Failed test for rte_bbdev_stats_get on device %u ", + dev_id); + dev2->dev_ops = dev1.dev_ops; + + dev_ops_tmp = *dev2->dev_ops; + dev_ops_tmp.stats_reset = NULL; + dev2->dev_ops = &dev_ops_tmp; + TEST_ASSERT_SUCCESS(rte_bbdev_stats_get(dev_id, &stats), + "Failed test for rte_bbdev_stats_get: " + "NULL stats_get "); + dev2->dev_ops = dev1.dev_ops; + + TEST_ASSERT_SUCCESS(rte_bbdev_stats_get(dev_id, &stats), + "Failed test for rte_bbdev_stats_get on device %u ", + dev_id); + + /* + * Tests for: + * rte_bbdev_callback_register, + * rte_bbdev_pmd_callback_process, + * rte_dev_callback_unregister + */ + dev2->dev_ops = NULL; + TEST_ASSERT_SUCCESS(rte_bbdev_callback_register(dev_id, + RTE_BBDEV_EVENT_UNKNOWN, event_callback, NULL), + "Failed to callback rgstr for RTE_BBDEV_EVENT_UNKNOWN"); + rte_bbdev_pmd_callback_process(dev2, RTE_BBDEV_EVENT_UNKNOWN, NULL); + + TEST_ASSERT_SUCCESS(rte_bbdev_callback_unregister(dev_id, + RTE_BBDEV_EVENT_UNKNOWN, event_callback, NULL), + "Failed to unregister RTE_BBDEV_EVENT_ERROR "); + dev2->dev_ops = dev1.dev_ops; + + /* Tests for rte_bbdev_stats_reset */ + dev2->dev_ops = NULL; + TEST_ASSERT_FAIL(rte_bbdev_stats_reset(dev_id), + "Failed to reset statistic for device %u ", dev_id); + dev2->dev_ops = dev1.dev_ops; + + dev_ops_tmp = *dev2->dev_ops; + dev_ops_tmp.stats_reset = NULL; + dev2->dev_ops = &dev_ops_tmp; + TEST_ASSERT_SUCCESS(rte_bbdev_stats_reset(dev_id), + "Failed test for rte_bbdev_stats_reset: " + "NULL stats_reset "); + dev2->dev_ops = dev1.dev_ops; + + TEST_ASSERT_SUCCESS(rte_bbdev_stats_reset(dev_id), + "Failed to reset statistic for device %u ", dev_id); + + /* Tests for rte_bbdev_queue_stop */ + dev2->dev_ops = NULL; + TEST_ASSERT_FAIL(rte_bbdev_queue_stop(dev_id, queue_id), + "Failed to stop queue %u on device %u: " + "NULL dev_ops structure", queue_id, dev_id); + dev2->dev_ops = dev1.dev_ops; + + TEST_ASSERT_SUCCESS(rte_bbdev_queue_stop(dev_id, queue_id), + "Failed to stop queue %u on device %u ", queue_id, + dev_id); + + /* Tests for rte_bbdev_stop */ + dev2->dev_ops = NULL; + TEST_ASSERT_FAIL(rte_bbdev_stop(dev_id), + "Failed to stop bbdev %u with NULL dev_ops structure ", + dev_id); + dev2->dev_ops = dev1.dev_ops; + + TEST_ASSERT_SUCCESS(rte_bbdev_stop(dev_id), + "Failed to stop bbdev %u ", dev_id); + + /* Tests for rte_bbdev_close */ + TEST_ASSERT_FAIL(rte_bbdev_close(RTE_BBDEV_MAX_DEVS), + "Failed to close bbdev with invalid dev_id"); + + dev2->dev_ops = NULL; + TEST_ASSERT_FAIL(rte_bbdev_close(dev_id), + "Failed to close bbdev %u with NULL dev_ops structure ", + dev_id); + dev2->dev_ops = dev1.dev_ops; + + TEST_ASSERT_SUCCESS(rte_bbdev_close(dev_id), + "Failed to close bbdev %u ", dev_id); + + return TEST_SUCCESS; +} + +static int +test_bbdev_get_named_dev(void) +{ + struct rte_bbdev *dev, *dev_tmp; + const char *name = "name"; + + dev = rte_bbdev_allocate(name); + TEST_ASSERT(dev != NULL, "Failed to initialize bbdev driver"); + + dev_tmp = rte_bbdev_get_named_dev(NULL); + TEST_ASSERT(dev_tmp == NULL, "Failed test for rte_bbdev_get_named_dev: " + "function called with NULL parameter"); + + dev_tmp = rte_bbdev_get_named_dev(name); + + TEST_ASSERT(dev == dev_tmp, "Failed test for rte_bbdev_get_named_dev: " + "wrong device was returned "); + + TEST_ASSERT_SUCCESS(rte_bbdev_release(dev), + "Failed to uninitialize bbdev driver %s ", name); + + return TEST_SUCCESS; +} + +static struct unit_test_suite bbdev_null_testsuite = { + .suite_name = "BBDEV NULL Unit Test Suite", + .setup = testsuite_setup, + .teardown = testsuite_teardown, + .unit_test_cases = { + + TEST_CASE(test_bbdev_configure_invalid_dev_id), + + TEST_CASE_ST(ut_setup, ut_teardown, + test_bbdev_configure_invalid_num_queues), + + TEST_CASE_ST(ut_setup, ut_teardown, + test_bbdev_configure_stop_device), + + TEST_CASE_ST(ut_setup, ut_teardown, + test_bbdev_configure_stop_queue), + + TEST_CASE_ST(ut_setup, ut_teardown, + test_bbdev_configure_invalid_queue_configure), + + TEST_CASE_ST(ut_setup, ut_teardown, + test_bbdev_op_pool), + + TEST_CASE_ST(ut_setup, ut_teardown, + test_bbdev_op_type), + + TEST_CASE_ST(ut_setup, ut_teardown, + test_bbdev_op_pool_size), + + TEST_CASE_ST(ut_setup, ut_teardown, + test_bbdev_stats), + + TEST_CASE_ST(ut_setup, ut_teardown, + test_bbdev_driver_init), + + TEST_CASE_ST(ut_setup, ut_teardown, + test_bbdev_callback), + + TEST_CASE_ST(ut_setup, ut_teardown, + test_bbdev_invalid_driver), + + TEST_CASE_ST(ut_setup, ut_teardown, + test_bbdev_get_named_dev), + + TEST_CASE(test_bbdev_count), + + TEST_CASES_END() /**< NULL terminate unit test array */ + } +}; + +REGISTER_TEST_COMMAND(unittest, bbdev_null_testsuite); diff --git a/app/test-bbdev/test_bbdev_perf.c b/app/test-bbdev/test_bbdev_perf.c new file mode 100644 index 0000000..1ab67ca --- /dev/null +++ b/app/test-bbdev/test_bbdev_perf.c @@ -0,0 +1,2090 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stdio.h> +#include <inttypes.h> + +#include <rte_eal.h> +#include <rte_common.h> +#include <rte_dev.h> +#include <rte_launch.h> +#include <rte_bbdev.h> +#include <rte_cycles.h> +#include <rte_lcore.h> +#include <rte_malloc.h> +#include <rte_random.h> +#include <rte_hexdump.h> + +#include "main.h" +#include "test_bbdev_vector.h" + +#define GET_SOCKET(socket_id) (((socket_id) == SOCKET_ID_ANY) ? 0 : (socket_id)) + +#define MAX_QUEUES RTE_MAX_LCORE + +#define OPS_CACHE_SIZE 256U +#define OPS_POOL_SIZE_MIN 511U /* 0.5K per queue */ + +#define SYNC_WAIT 0 +#define SYNC_START 1 + +#define INVALID_QUEUE_ID -1 + +static struct test_bbdev_vector test_vector; + +/* Switch between PMD and Interrupt for throughput TC */ +static bool intr_enabled; + +/* Represents tested active devices */ +static struct active_device { + const char *driver_name; + uint8_t dev_id; + uint16_t supported_ops; + uint16_t queue_ids[MAX_QUEUES]; + uint16_t nb_queues; + struct rte_mempool *ops_mempool; + struct rte_mempool *in_mbuf_pool; + struct rte_mempool *hard_out_mbuf_pool; + struct rte_mempool *soft_out_mbuf_pool; +} active_devs[RTE_BBDEV_MAX_DEVS]; + +static uint8_t nb_active_devs; + +/* Data buffers used by BBDEV ops */ +struct test_buffers { + struct rte_bbdev_op_data *inputs; + struct rte_bbdev_op_data *hard_outputs; + struct rte_bbdev_op_data *soft_outputs; +}; + +/* Operation parameters specific for given test case */ +struct test_op_params { + struct rte_mempool *mp; + struct rte_bbdev_dec_op *ref_dec_op; + struct rte_bbdev_enc_op *ref_enc_op; + uint16_t burst_sz; + uint16_t num_to_process; + int vector_mask; + rte_atomic16_t sync; + struct test_buffers q_bufs[RTE_MAX_NUMA_NODES][MAX_QUEUES]; +}; + +/* Contains per lcore params */ +struct thread_params { + uint8_t dev_id; + uint16_t queue_id; + uint64_t start_time; + rte_atomic16_t nb_dequeued; + rte_atomic16_t processing_status; + struct test_op_params *op_params; +}; + +typedef int (test_case_function)(struct active_device *ad, + struct test_op_params *op_params); + +static inline void +set_avail_op(struct active_device *ad, enum rte_bbdev_op_type op_type) +{ + ad->supported_ops |= (1 << op_type); +} + +static inline bool +is_avail_op(struct active_device *ad, enum rte_bbdev_op_type op_type) +{ + return ad->supported_ops & (1 << op_type); +} + +/* calculates optimal mempool size not smaller than the val */ +static unsigned int +optimal_mempool_size(unsigned int val) +{ + return rte_align32pow2(val + 1) - 1; +} + +/* allocates mbuf mempool for inputs and outputs */ +static struct rte_mempool * +create_mbuf_pool(struct op_data_entries *entries, uint8_t dev_id, + int socket_id, unsigned int mbuf_pool_size, + const char *op_type_str) +{ + unsigned int i; + uint32_t max_seg_sz = 0; + char pool_name[RTE_MEMPOOL_NAMESIZE]; + + /* find max input segment size */ + for (i = 0; i < entries->nb_segments; ++i) + if (entries->segments[i].length > max_seg_sz) + max_seg_sz = entries->segments[i].length; + + snprintf(pool_name, sizeof(pool_name), "%s_pool_%u", op_type_str, + dev_id); + return rte_pktmbuf_pool_create(pool_name, mbuf_pool_size, 0, 0, + RTE_MAX(max_seg_sz + RTE_PKTMBUF_HEADROOM, + (unsigned int)RTE_MBUF_DEFAULT_BUF_SIZE), socket_id); +} + +static int +create_mempools(struct active_device *ad, int socket_id, + enum rte_bbdev_op_type op_type, uint16_t num_ops) +{ + struct rte_mempool *mp; + unsigned int ops_pool_size, mbuf_pool_size = 0; + char pool_name[RTE_MEMPOOL_NAMESIZE]; + + struct op_data_entries *in = &test_vector.entries[DATA_INPUT]; + struct op_data_entries *hard_out = + &test_vector.entries[DATA_HARD_OUTPUT]; + struct op_data_entries *soft_out = + &test_vector.entries[DATA_SOFT_OUTPUT]; + + /* allocate ops mempool */ + ops_pool_size = optimal_mempool_size(RTE_MAX( + /* Ops used plus 1 reference op */ + RTE_MAX((unsigned int)(ad->nb_queues * num_ops + 1), + /* Minimal cache size plus 1 reference op */ + (unsigned int)(1.5 * rte_lcore_count() * + OPS_CACHE_SIZE + 1)), + OPS_POOL_SIZE_MIN)); + snprintf(pool_name, sizeof(pool_name), "%s_pool_%u", + rte_bbdev_op_type_str(op_type), ad->dev_id); + mp = rte_bbdev_op_pool_create(pool_name, op_type, + ops_pool_size, OPS_CACHE_SIZE, socket_id); + TEST_ASSERT_NOT_NULL(mp, + "ERROR Failed to create %u items ops pool for dev %u on socket %u.", + ops_pool_size, + ad->dev_id, + socket_id); + ad->ops_mempool = mp; + + /* Inputs */ + mbuf_pool_size = optimal_mempool_size(ops_pool_size * in->nb_segments); + mp = create_mbuf_pool(in, ad->dev_id, socket_id, mbuf_pool_size, "in"); + TEST_ASSERT_NOT_NULL(mp, + "ERROR Failed to create %u items input pktmbuf pool for dev %u on socket %u.", + mbuf_pool_size, + ad->dev_id, + socket_id); + ad->in_mbuf_pool = mp; + + /* Hard outputs */ + mbuf_pool_size = optimal_mempool_size(ops_pool_size * + hard_out->nb_segments); + mp = create_mbuf_pool(hard_out, ad->dev_id, socket_id, mbuf_pool_size, + "hard_out"); + TEST_ASSERT_NOT_NULL(mp, + "ERROR Failed to create %u items hard output pktmbuf pool for dev %u on socket %u.", + mbuf_pool_size, + ad->dev_id, + socket_id); + ad->hard_out_mbuf_pool = mp; + + if (soft_out->nb_segments == 0) + return TEST_SUCCESS; + + /* Soft outputs */ + mbuf_pool_size = optimal_mempool_size(ops_pool_size * + soft_out->nb_segments); + mp = create_mbuf_pool(soft_out, ad->dev_id, socket_id, mbuf_pool_size, + "soft_out"); + TEST_ASSERT_NOT_NULL(mp, + "ERROR Failed to create %uB soft output pktmbuf pool for dev %u on socket %u.", + mbuf_pool_size, + ad->dev_id, + socket_id); + ad->soft_out_mbuf_pool = mp; + + return 0; +} + +static int +add_bbdev_dev(uint8_t dev_id, struct rte_bbdev_info *info, + struct test_bbdev_vector *vector) +{ + int ret; + unsigned int queue_id; + struct rte_bbdev_queue_conf qconf; + struct active_device *ad = &active_devs[nb_active_devs]; + unsigned int nb_queues; + enum rte_bbdev_op_type op_type = vector->op_type; + + nb_queues = RTE_MIN(rte_lcore_count(), info->drv.max_num_queues); + /* setup device */ + ret = rte_bbdev_setup_queues(dev_id, nb_queues, info->socket_id); + if (ret < 0) { + printf("rte_bbdev_setup_queues(%u, %u, %d) ret %i\n", + dev_id, nb_queues, info->socket_id, ret); + return TEST_FAILED; + } + + /* configure interrupts if needed */ + if (intr_enabled) { + ret = rte_bbdev_intr_enable(dev_id); + if (ret < 0) { + printf("rte_bbdev_intr_enable(%u) ret %i\n", dev_id, + ret); + return TEST_FAILED; + } + } + + /* setup device queues */ + qconf.socket = info->socket_id; + qconf.queue_size = info->drv.default_queue_conf.queue_size; + qconf.priority = 0; + qconf.deferred_start = 0; + qconf.op_type = op_type; + + for (queue_id = 0; queue_id < nb_queues; ++queue_id) { + ret = rte_bbdev_queue_configure(dev_id, queue_id, &qconf); + if (ret != 0) { + printf( + "Allocated all queues (id=%u) at prio%u on dev%u\n", + queue_id, qconf.priority, dev_id); + qconf.priority++; + ret = rte_bbdev_queue_configure(ad->dev_id, queue_id, + &qconf); + } + if (ret != 0) { + printf("All queues on dev %u allocated: %u\n", + dev_id, queue_id); + break; + } + ad->queue_ids[queue_id] = queue_id; + } + TEST_ASSERT(queue_id != 0, + "ERROR Failed to configure any queues on dev %u", + dev_id); + ad->nb_queues = queue_id; + + set_avail_op(ad, op_type); + + return TEST_SUCCESS; +} + +static int +add_active_device(uint8_t dev_id, struct rte_bbdev_info *info, + struct test_bbdev_vector *vector) +{ + int ret; + + active_devs[nb_active_devs].driver_name = info->drv.driver_name; + active_devs[nb_active_devs].dev_id = dev_id; + + ret = add_bbdev_dev(dev_id, info, vector); + if (ret == TEST_SUCCESS) + ++nb_active_devs; + return ret; +} + +static inline bool +flags_match(uint32_t flags_req, uint32_t flags_present) +{ + return (flags_req & flags_present) == flags_req; +} + +static int +check_dev_cap(const struct rte_bbdev_info *dev_info) +{ + unsigned int i; + unsigned int nb_inputs, nb_soft_outputs, nb_hard_outputs; + const struct rte_bbdev_op_cap *op_cap = dev_info->drv.capabilities; + + nb_inputs = test_vector.entries[DATA_INPUT].nb_segments; + nb_soft_outputs = test_vector.entries[DATA_SOFT_OUTPUT].nb_segments; + nb_hard_outputs = test_vector.entries[DATA_HARD_OUTPUT].nb_segments; + + for (i = 0; op_cap->type != RTE_BBDEV_OP_NONE; ++i, ++op_cap) { + if (op_cap->type != test_vector.op_type) + continue; + + if (op_cap->type == RTE_BBDEV_OP_TURBO_DEC) { + const struct rte_bbdev_op_cap_turbo_dec *cap = + &op_cap->cap.turbo_dec; + /* Ignore lack of soft output capability, just skip + * checking if soft output is valid. + */ + if ((test_vector.turbo_dec.op_flags & + RTE_BBDEV_TURBO_SOFT_OUTPUT) && + !(cap->capability_flags & + RTE_BBDEV_TURBO_SOFT_OUTPUT)) { + printf( + "WARNING: Device \"%s\" does not support soft output, RTE_BBDEV_TURBO_SOFT_OUTPUT flag will be ignored.\n", + dev_info->dev_name); + test_vector.turbo_dec.op_flags &= + ~RTE_BBDEV_TURBO_SOFT_OUTPUT; + } + + if (!flags_match(test_vector.turbo_dec.op_flags, + cap->capability_flags)) + return TEST_FAILED; + if (nb_inputs > cap->num_buffers_src) { + printf("Too many inputs defined: %u, max: %u\n", + nb_inputs, cap->num_buffers_src); + return TEST_FAILED; + } + if (nb_soft_outputs > cap->num_buffers_soft_out && + (test_vector.turbo_dec.op_flags & + RTE_BBDEV_TURBO_SOFT_OUTPUT)) { + printf( + "Too many soft outputs defined: %u, max: %u\n", + nb_soft_outputs, + cap->num_buffers_soft_out); + return TEST_FAILED; + } + if (nb_hard_outputs > cap->num_buffers_hard_out) { + printf( + "Too many hard outputs defined: %u, max: %u\n", + nb_hard_outputs, + cap->num_buffers_hard_out); + return TEST_FAILED; + } + if (intr_enabled && !(cap->capability_flags & + RTE_BBDEV_TURBO_DEC_INTERRUPTS)) { + printf( + "Dequeue interrupts are not supported!\n"); + return TEST_FAILED; + } + + return TEST_SUCCESS; + } else if (op_cap->type == RTE_BBDEV_OP_TURBO_ENC) { + const struct rte_bbdev_op_cap_turbo_enc *cap = + &op_cap->cap.turbo_enc; + + if (!flags_match(test_vector.turbo_enc.op_flags, + cap->capability_flags)) + return TEST_FAILED; + if (nb_inputs > cap->num_buffers_src) { + printf("Too many inputs defined: %u, max: %u\n", + nb_inputs, cap->num_buffers_src); + return TEST_FAILED; + } + if (nb_hard_outputs > cap->num_buffers_dst) { + printf( + "Too many hard outputs defined: %u, max: %u\n", + nb_hard_outputs, cap->num_buffers_src); + return TEST_FAILED; + } + if (intr_enabled && !(cap->capability_flags & + RTE_BBDEV_TURBO_ENC_INTERRUPTS)) { + printf( + "Dequeue interrupts are not supported!\n"); + return TEST_FAILED; + } + + return TEST_SUCCESS; + } + } + + if ((i == 0) && (test_vector.op_type == RTE_BBDEV_OP_NONE)) + return TEST_SUCCESS; /* Special case for NULL device */ + + return TEST_FAILED; +} + +static uint8_t +populate_active_devices(void) +{ + int ret; + uint8_t dev_id; + uint8_t nb_devs_added = 0; + struct rte_bbdev_info info; + + RTE_BBDEV_FOREACH(dev_id) { + rte_bbdev_info_get(dev_id, &info); + + if (check_dev_cap(&info)) { + printf( + "Device %d (%s) does not support specified capabilities\n", + dev_id, info.dev_name); + continue; + } + + ret = add_active_device(dev_id, &info, &test_vector); + if (ret != 0) { + printf("Adding active bbdev %s skipped\n", + info.dev_name); + continue; + } + nb_devs_added++; + } + + return nb_devs_added; +} + +static int +read_test_vector(void) +{ + int ret; + + memset(&test_vector, 0, sizeof(test_vector)); + printf("Test vector file = %s\n", get_vector_filename()); + ret = test_bbdev_vector_read(get_vector_filename(), &test_vector); + TEST_ASSERT_SUCCESS(ret, "Failed to parse file %s\n", + get_vector_filename()); + + return TEST_SUCCESS; +} + +static int +testsuite_setup(void) +{ + TEST_ASSERT_SUCCESS(read_test_vector(), "Test suite setup failed\n"); + + if (populate_active_devices() == 0) { + printf("No suitable devices found!\n"); + return TEST_SKIPPED; + } + + return TEST_SUCCESS; +} + +static int +interrupt_testsuite_setup(void) +{ + TEST_ASSERT_SUCCESS(read_test_vector(), "Test suite setup failed\n"); + + /* Enable interrupts */ + intr_enabled = true; + + /* Special case for NULL device (RTE_BBDEV_OP_NONE) */ + if (populate_active_devices() == 0 || + test_vector.op_type == RTE_BBDEV_OP_NONE) { + intr_enabled = false; + printf("No suitable devices found!\n"); + return TEST_SKIPPED; + } + + return TEST_SUCCESS; +} + +static void +testsuite_teardown(void) +{ + uint8_t dev_id; + + /* Unconfigure devices */ + RTE_BBDEV_FOREACH(dev_id) + rte_bbdev_close(dev_id); + + /* Clear active devices structs. */ + memset(active_devs, 0, sizeof(active_devs)); + nb_active_devs = 0; +} + +static int +ut_setup(void) +{ + uint8_t i, dev_id; + + for (i = 0; i < nb_active_devs; i++) { + dev_id = active_devs[i].dev_id; + /* reset bbdev stats */ + TEST_ASSERT_SUCCESS(rte_bbdev_stats_reset(dev_id), + "Failed to reset stats of bbdev %u", dev_id); + /* start the device */ + TEST_ASSERT_SUCCESS(rte_bbdev_start(dev_id), + "Failed to start bbdev %u", dev_id); + } + + return TEST_SUCCESS; +} + +static void +ut_teardown(void) +{ + uint8_t i, dev_id; + struct rte_bbdev_stats stats; + + for (i = 0; i < nb_active_devs; i++) { + dev_id = active_devs[i].dev_id; + /* read stats and print */ + rte_bbdev_stats_get(dev_id, &stats); + /* Stop the device */ + rte_bbdev_stop(dev_id); + } +} + +static int +init_op_data_objs(struct rte_bbdev_op_data *bufs, + struct op_data_entries *ref_entries, + struct rte_mempool *mbuf_pool, const uint16_t n, + enum op_data_type op_type, uint16_t min_alignment) +{ + int ret; + unsigned int i, j; + + for (i = 0; i < n; ++i) { + char *data; + struct op_data_buf *seg = &ref_entries->segments[0]; + struct rte_mbuf *m_head = rte_pktmbuf_alloc(mbuf_pool); + TEST_ASSERT_NOT_NULL(m_head, + "Not enough mbufs in %d data type mbuf pool (needed %u, available %u)", + op_type, n * ref_entries->nb_segments, + mbuf_pool->size); + + bufs[i].data = m_head; + bufs[i].offset = 0; + bufs[i].length = 0; + + if (op_type == DATA_INPUT) { + data = rte_pktmbuf_append(m_head, seg->length); + TEST_ASSERT_NOT_NULL(data, + "Couldn't append %u bytes to mbuf from %d data type mbuf pool", + seg->length, op_type); + + TEST_ASSERT(data == RTE_PTR_ALIGN(data, min_alignment), + "Data addr in mbuf (%p) is not aligned to device min alignment (%u)", + data, min_alignment); + rte_memcpy(data, seg->addr, seg->length); + bufs[i].length += seg->length; + + + for (j = 1; j < ref_entries->nb_segments; ++j) { + struct rte_mbuf *m_tail = + rte_pktmbuf_alloc(mbuf_pool); + TEST_ASSERT_NOT_NULL(m_tail, + "Not enough mbufs in %d data type mbuf pool (needed %u, available %u)", + op_type, + n * ref_entries->nb_segments, + mbuf_pool->size); + seg += 1; + + data = rte_pktmbuf_append(m_tail, seg->length); + TEST_ASSERT_NOT_NULL(data, + "Couldn't append %u bytes to mbuf from %d data type mbuf pool", + seg->length, op_type); + + TEST_ASSERT(data == RTE_PTR_ALIGN(data, + min_alignment), + "Data addr in mbuf (%p) is not aligned to device min alignment (%u)", + data, min_alignment); + rte_memcpy(data, seg->addr, seg->length); + bufs[i].length += seg->length; + + ret = rte_pktmbuf_chain(m_head, m_tail); + TEST_ASSERT_SUCCESS(ret, + "Couldn't chain mbufs from %d data type mbuf pool", + op_type); + } + } + } + + return 0; +} + +static int +allocate_buffers_on_socket(struct rte_bbdev_op_data **buffers, const int len, + const int socket) +{ + int i; + + *buffers = rte_zmalloc_socket(NULL, len, 0, socket); + if (*buffers == NULL) { + printf("WARNING: Failed to allocate op_data on socket %d\n", + socket); + /* try to allocate memory on other detected sockets */ + for (i = 0; i < socket; i++) { + *buffers = rte_zmalloc_socket(NULL, len, 0, i); + if (*buffers != NULL) + break; + } + } + + return (*buffers == NULL) ? TEST_FAILED : TEST_SUCCESS; +} + +static int +fill_queue_buffers(struct test_op_params *op_params, + struct rte_mempool *in_mp, struct rte_mempool *hard_out_mp, + struct rte_mempool *soft_out_mp, uint16_t queue_id, + uint16_t min_alignment, const int socket_id) +{ + int ret; + enum op_data_type type; + const uint16_t n = op_params->num_to_process; + + struct rte_mempool *mbuf_pools[DATA_NUM_TYPES] = { + in_mp, + soft_out_mp, + hard_out_mp, + }; + + struct rte_bbdev_op_data **queue_ops[DATA_NUM_TYPES] = { + &op_params->q_bufs[socket_id][queue_id].inputs, + &op_params->q_bufs[socket_id][queue_id].soft_outputs, + &op_params->q_bufs[socket_id][queue_id].hard_outputs, + }; + + for (type = DATA_INPUT; type < DATA_NUM_TYPES; ++type) { + struct op_data_entries *ref_entries = + &test_vector.entries[type]; + if (ref_entries->nb_segments == 0) + continue; + + ret = allocate_buffers_on_socket(queue_ops[type], + n * sizeof(struct rte_bbdev_op_data), + socket_id); + TEST_ASSERT_SUCCESS(ret, + "Couldn't allocate memory for rte_bbdev_op_data structs"); + + ret = init_op_data_objs(*queue_ops[type], ref_entries, + mbuf_pools[type], n, type, min_alignment); + TEST_ASSERT_SUCCESS(ret, + "Couldn't init rte_bbdev_op_data structs"); + } + + return 0; +} + +static void +free_buffers(struct active_device *ad, struct test_op_params *op_params) +{ + unsigned int i, j; + + rte_mempool_free(ad->ops_mempool); + rte_mempool_free(ad->in_mbuf_pool); + rte_mempool_free(ad->hard_out_mbuf_pool); + rte_mempool_free(ad->soft_out_mbuf_pool); + + for (i = 0; i < rte_lcore_count(); ++i) { + for (j = 0; j < RTE_MAX_NUMA_NODES; ++j) { + rte_free(op_params->q_bufs[j][i].inputs); + rte_free(op_params->q_bufs[j][i].hard_outputs); + rte_free(op_params->q_bufs[j][i].soft_outputs); + } + } +} + +static void +copy_reference_dec_op(struct rte_bbdev_dec_op **ops, unsigned int n, + unsigned int start_idx, + struct rte_bbdev_op_data *inputs, + struct rte_bbdev_op_data *hard_outputs, + struct rte_bbdev_op_data *soft_outputs, + struct rte_bbdev_dec_op *ref_op) +{ + unsigned int i; + struct rte_bbdev_op_turbo_dec *turbo_dec = &ref_op->turbo_dec; + + for (i = 0; i < n; ++i) { + if (turbo_dec->code_block_mode == 0) { + ops[i]->turbo_dec.tb_params.ea = + turbo_dec->tb_params.ea; + ops[i]->turbo_dec.tb_params.eb = + turbo_dec->tb_params.eb; + ops[i]->turbo_dec.tb_params.k_pos = + turbo_dec->tb_params.k_pos; + ops[i]->turbo_dec.tb_params.k_neg = + turbo_dec->tb_params.k_neg; + ops[i]->turbo_dec.tb_params.c = + turbo_dec->tb_params.c; + ops[i]->turbo_dec.tb_params.c_neg = + turbo_dec->tb_params.c_neg; + ops[i]->turbo_dec.tb_params.cab = + turbo_dec->tb_params.cab; + } else { + ops[i]->turbo_dec.cb_params.e = turbo_dec->cb_params.e; + ops[i]->turbo_dec.cb_params.k = turbo_dec->cb_params.k; + } + + ops[i]->turbo_dec.ext_scale = turbo_dec->ext_scale; + ops[i]->turbo_dec.iter_max = turbo_dec->iter_max; + ops[i]->turbo_dec.iter_min = turbo_dec->iter_min; + ops[i]->turbo_dec.op_flags = turbo_dec->op_flags; + ops[i]->turbo_dec.rv_index = turbo_dec->rv_index; + ops[i]->turbo_dec.num_maps = turbo_dec->num_maps; + ops[i]->turbo_dec.code_block_mode = turbo_dec->code_block_mode; + + ops[i]->turbo_dec.hard_output = hard_outputs[start_idx + i]; + ops[i]->turbo_dec.input = inputs[start_idx + i]; + if (soft_outputs != NULL) + ops[i]->turbo_dec.soft_output = + soft_outputs[start_idx + i]; + } +} + +static void +copy_reference_enc_op(struct rte_bbdev_enc_op **ops, unsigned int n, + unsigned int start_idx, + struct rte_bbdev_op_data *inputs, + struct rte_bbdev_op_data *outputs, + struct rte_bbdev_enc_op *ref_op) +{ + unsigned int i; + struct rte_bbdev_op_turbo_enc *turbo_enc = &ref_op->turbo_enc; + for (i = 0; i < n; ++i) { + if (turbo_enc->code_block_mode == 0) { + ops[i]->turbo_enc.tb_params.ea = + turbo_enc->tb_params.ea; + ops[i]->turbo_enc.tb_params.eb = + turbo_enc->tb_params.eb; + ops[i]->turbo_enc.tb_params.k_pos = + turbo_enc->tb_params.k_pos; + ops[i]->turbo_enc.tb_params.k_neg = + turbo_enc->tb_params.k_neg; + ops[i]->turbo_enc.tb_params.c = + turbo_enc->tb_params.c; + ops[i]->turbo_enc.tb_params.c_neg = + turbo_enc->tb_params.c_neg; + ops[i]->turbo_enc.tb_params.cab = + turbo_enc->tb_params.cab; + ops[i]->turbo_enc.tb_params.ncb_pos = + turbo_enc->tb_params.ncb_pos; + ops[i]->turbo_enc.tb_params.ncb_neg = + turbo_enc->tb_params.ncb_neg; + /* total number of bits available for the + * transmission of one TB + */ + ops[i]->turbo_enc.g = turbo_enc->tb_params.ncb_pos * 3; + } else { + ops[i]->turbo_enc.cb_params.e = turbo_enc->cb_params.e; + ops[i]->turbo_enc.cb_params.k = turbo_enc->cb_params.k; + ops[i]->turbo_enc.cb_params.ncb = + turbo_enc->cb_params.ncb; + /* total number of bits available for the + * transmission of one TB + */ + ops[i]->turbo_enc.g = turbo_enc->cb_params.ncb * 3; + } + /* For UE category 1 number of soft bits is 250368 + * according to TS 36.306 Table 4.1-1 + */ + ops[i]->turbo_enc.n_soft = 250368; + ops[i]->turbo_enc.k_mimo = 1; + /* Max number of DL HARQ depends on UL/DL configuration + * according to TS 36.213 Table 7-1 + */ + ops[i]->turbo_enc.mdl_harq = 4; + ops[i]->turbo_enc.nl = 1; + ops[i]->turbo_enc.qm = 2; + ops[i]->turbo_enc.rv_index = turbo_enc->rv_index; + ops[i]->turbo_enc.op_flags = turbo_enc->op_flags; + ops[i]->turbo_enc.code_block_mode = turbo_enc->code_block_mode; + + ops[i]->turbo_enc.output = outputs[start_idx + i]; + ops[i]->turbo_enc.input = inputs[start_idx + i]; + } +} + +static int +check_dec_status_and_ordering(struct rte_bbdev_dec_op *op, + unsigned int order_idx, const int expected_status) +{ + TEST_ASSERT(op->status == expected_status, + "op_status (%d) != expected_status (%d)", + op->status, expected_status); + + TEST_ASSERT((void *)(uintptr_t)order_idx == op->opaque_data, + "Ordering error, expected %p, got %p", + (void *)(uintptr_t)order_idx, op->opaque_data); + + return TEST_SUCCESS; +} + +static int +check_enc_status_and_ordering(struct rte_bbdev_enc_op *op, + unsigned int order_idx, const int expected_status) +{ + TEST_ASSERT(op->status == expected_status, + "op_status (%d) != expected_status (%d)", + op->status, expected_status); + + TEST_ASSERT((void *)(uintptr_t)order_idx == op->opaque_data, + "Ordering error, expected %p, got %p", + (void *)(uintptr_t)order_idx, op->opaque_data); + + return TEST_SUCCESS; +} + +static inline int +validate_op_chain(struct rte_bbdev_op_data *op, + struct op_data_entries *orig_op) +{ + uint8_t i; + struct rte_mbuf *m = op->data; + uint8_t nb_dst_segments = orig_op->nb_segments; + + TEST_ASSERT(nb_dst_segments == m->nb_segs, + "Number of segments differ in original (%u) and filled (%u) op", + nb_dst_segments, m->nb_segs); + + for (i = 0; i < nb_dst_segments; ++i) { + /* Apply offset to the first mbuf segment */ + uint16_t offset = (i == 0) ? op->offset : 0; + uint16_t data_len = m->data_len - offset; + + TEST_ASSERT(orig_op->segments[i].length == data_len, + "Length of segment differ in original (%u) and filled (%u) op", + orig_op->segments[i].length, data_len); + TEST_ASSERT_BUFFERS_ARE_EQUAL(orig_op->segments[i].addr, + rte_pktmbuf_mtod_offset(m, uint32_t *, offset), + data_len, + "Output buffers (CB=%u) are not equal", i); + m = m->next; + } + + return TEST_SUCCESS; +} + +static int +validate_dec_buffers(struct rte_bbdev_dec_op *ref_op, struct test_buffers *bufs, + const uint16_t num_to_process) +{ + int i; + + struct op_data_entries *hard_data_orig = + &test_vector.entries[DATA_HARD_OUTPUT]; + struct op_data_entries *soft_data_orig = + &test_vector.entries[DATA_SOFT_OUTPUT]; + + for (i = 0; i < num_to_process; i++) { + TEST_ASSERT_SUCCESS(validate_op_chain(&bufs->hard_outputs[i], + hard_data_orig), + "Hard output buffers are not equal"); + if (ref_op->turbo_dec.op_flags & + RTE_BBDEV_TURBO_SOFT_OUTPUT) + TEST_ASSERT_SUCCESS(validate_op_chain( + &bufs->soft_outputs[i], + soft_data_orig), + "Soft output buffers are not equal"); + } + + return TEST_SUCCESS; +} + +static int +validate_enc_buffers(struct test_buffers *bufs, const uint16_t num_to_process) +{ + int i; + + struct op_data_entries *hard_data_orig = + &test_vector.entries[DATA_HARD_OUTPUT]; + + for (i = 0; i < num_to_process; i++) + TEST_ASSERT_SUCCESS(validate_op_chain(&bufs->hard_outputs[i], + hard_data_orig), ""); + + return TEST_SUCCESS; +} + +static int +validate_dec_op(struct rte_bbdev_dec_op **ops, const uint16_t n, + struct rte_bbdev_dec_op *ref_op, const int vector_mask) +{ + unsigned int i; + int ret; + struct op_data_entries *hard_data_orig = + &test_vector.entries[DATA_HARD_OUTPUT]; + struct op_data_entries *soft_data_orig = + &test_vector.entries[DATA_SOFT_OUTPUT]; + struct rte_bbdev_op_turbo_dec *ops_td; + struct rte_bbdev_op_data *hard_output; + struct rte_bbdev_op_data *soft_output; + struct rte_bbdev_op_turbo_dec *ref_td = &ref_op->turbo_dec; + + for (i = 0; i < n; ++i) { + ops_td = &ops[i]->turbo_dec; + hard_output = &ops_td->hard_output; + soft_output = &ops_td->soft_output; + + if (vector_mask & TEST_BBDEV_VF_EXPECTED_ITER_COUNT) + TEST_ASSERT(ops_td->iter_count <= ref_td->iter_count, + "Returned iter_count (%d) > expected iter_count (%d)", + ops_td->iter_count, ref_td->iter_count); + ret = check_dec_status_and_ordering(ops[i], i, ref_op->status); + TEST_ASSERT_SUCCESS(ret, + "Checking status and ordering for decoder failed"); + + TEST_ASSERT_SUCCESS(validate_op_chain(hard_output, + hard_data_orig), + "Hard output buffers (CB=%u) are not equal", + i); + + if (ref_op->turbo_dec.op_flags & RTE_BBDEV_TURBO_SOFT_OUTPUT) + TEST_ASSERT_SUCCESS(validate_op_chain(soft_output, + soft_data_orig), + "Soft output buffers (CB=%u) are not equal", + i); + } + + return TEST_SUCCESS; +} + +static int +validate_enc_op(struct rte_bbdev_enc_op **ops, const uint16_t n, + struct rte_bbdev_enc_op *ref_op) +{ + unsigned int i; + int ret; + struct op_data_entries *hard_data_orig = + &test_vector.entries[DATA_HARD_OUTPUT]; + + for (i = 0; i < n; ++i) { + ret = check_enc_status_and_ordering(ops[i], i, ref_op->status); + TEST_ASSERT_SUCCESS(ret, + "Checking status and ordering for encoder failed"); + TEST_ASSERT_SUCCESS(validate_op_chain( + &ops[i]->turbo_enc.output, + hard_data_orig), + "Output buffers (CB=%u) are not equal", + i); + } + + return TEST_SUCCESS; +} + +static void +create_reference_dec_op(struct rte_bbdev_dec_op *op) +{ + op->turbo_dec = test_vector.turbo_dec; +} + +static void +create_reference_enc_op(struct rte_bbdev_enc_op *op) +{ + op->turbo_enc = test_vector.turbo_enc; +} + +static int +init_test_op_params(struct test_op_params *op_params, + enum rte_bbdev_op_type op_type, const int expected_status, + const int vector_mask, struct rte_mempool *ops_mp, + uint16_t burst_sz, uint16_t num_to_process) +{ + int ret = 0; + if (op_type == RTE_BBDEV_OP_TURBO_DEC) + ret = rte_bbdev_dec_op_alloc_bulk(ops_mp, + &op_params->ref_dec_op, 1); + else + ret = rte_bbdev_enc_op_alloc_bulk(ops_mp, + &op_params->ref_enc_op, 1); + + TEST_ASSERT_SUCCESS(ret, "rte_bbdev_op_alloc_bulk() failed"); + + op_params->mp = ops_mp; + op_params->burst_sz = burst_sz; + op_params->num_to_process = num_to_process; + op_params->vector_mask = vector_mask; + if (op_type == RTE_BBDEV_OP_TURBO_DEC) + op_params->ref_dec_op->status = expected_status; + else if (op_type == RTE_BBDEV_OP_TURBO_ENC) + op_params->ref_enc_op->status = expected_status; + + return 0; +} + +static int +run_test_case_on_device(test_case_function *test_case_func, uint8_t dev_id, + struct test_op_params *op_params) +{ + int t_ret, f_ret, socket_id = SOCKET_ID_ANY; + unsigned int i; + struct active_device *ad; + unsigned int burst_sz = get_burst_sz(); + enum rte_bbdev_op_type op_type = test_vector.op_type; + + ad = &active_devs[dev_id]; + + /* Check if device supports op_type */ + if (!is_avail_op(ad, test_vector.op_type)) + return TEST_SUCCESS; + + struct rte_bbdev_info info; + rte_bbdev_info_get(ad->dev_id, &info); + socket_id = GET_SOCKET(info.socket_id); + + if (op_type == RTE_BBDEV_OP_NONE) + op_type = RTE_BBDEV_OP_TURBO_ENC; + f_ret = create_mempools(ad, socket_id, op_type, + get_num_ops()); + if (f_ret != TEST_SUCCESS) { + printf("Couldn't create mempools"); + goto fail; + } + + f_ret = init_test_op_params(op_params, test_vector.op_type, + test_vector.expected_status, + test_vector.mask, + ad->ops_mempool, + burst_sz, + get_num_ops()); + if (f_ret != TEST_SUCCESS) { + printf("Couldn't init test op params"); + goto fail; + } + + if (test_vector.op_type == RTE_BBDEV_OP_TURBO_DEC) + create_reference_dec_op(op_params->ref_dec_op); + else if (test_vector.op_type == RTE_BBDEV_OP_TURBO_ENC) + create_reference_enc_op(op_params->ref_enc_op); + + for (i = 0; i < ad->nb_queues; ++i) { + f_ret = fill_queue_buffers(op_params, + ad->in_mbuf_pool, + ad->hard_out_mbuf_pool, + ad->soft_out_mbuf_pool, + ad->queue_ids[i], + info.drv.min_alignment, + socket_id); + if (f_ret != TEST_SUCCESS) { + printf("Couldn't init queue buffers"); + goto fail; + } + } + + /* Run test case function */ + t_ret = test_case_func(ad, op_params); + + /* Free active device resources and return */ + free_buffers(ad, op_params); + return t_ret; + +fail: + free_buffers(ad, op_params); + return TEST_FAILED; +} + +/* Run given test function per active device per supported op type + * per burst size. + */ +static int +run_test_case(test_case_function *test_case_func) +{ + int ret = 0; + uint8_t dev; + + /* Alloc op_params */ + struct test_op_params *op_params = rte_zmalloc(NULL, + sizeof(struct test_op_params), RTE_CACHE_LINE_SIZE); + TEST_ASSERT_NOT_NULL(op_params, "Failed to alloc %zuB for op_params", + RTE_ALIGN(sizeof(struct test_op_params), + RTE_CACHE_LINE_SIZE)); + + /* For each device run test case function */ + for (dev = 0; dev < nb_active_devs; ++dev) + ret |= run_test_case_on_device(test_case_func, dev, op_params); + + rte_free(op_params); + + return ret; +} + +static void +dequeue_event_callback(uint16_t dev_id, + enum rte_bbdev_event_type event, void *cb_arg, + void *ret_param) +{ + int ret; + uint16_t i; + uint64_t total_time; + uint16_t deq, burst_sz, num_to_process; + uint16_t queue_id = INVALID_QUEUE_ID; + struct rte_bbdev_dec_op *dec_ops[MAX_BURST]; + struct rte_bbdev_enc_op *enc_ops[MAX_BURST]; + struct test_buffers *bufs; + struct rte_bbdev_info info; + + /* Input length in bytes, million operations per second, + * million bits per second. + */ + double in_len, mops, mbps; + + struct thread_params *tp = cb_arg; + + RTE_SET_USED(ret_param); + + /* Find matching thread params using queue_id */ + for (i = 0; i < MAX_QUEUES; ++i, ++tp) + if (tp->queue_id == queue_id) + break; + + if (i == MAX_QUEUES) { + printf("%s: Queue_id from interrupt details was not found!\n", + __func__); + return; + } + + if (unlikely(event != RTE_BBDEV_EVENT_DEQUEUE)) { + rte_atomic16_set(&tp->processing_status, TEST_FAILED); + printf( + "Dequeue interrupt handler called for incorrect event!\n"); + return; + } + + burst_sz = tp->op_params->burst_sz; + num_to_process = tp->op_params->num_to_process; + + if (test_vector.op_type == RTE_BBDEV_OP_TURBO_DEC) + deq = rte_bbdev_dequeue_dec_ops(dev_id, queue_id, dec_ops, + burst_sz); + else + deq = rte_bbdev_dequeue_enc_ops(dev_id, queue_id, enc_ops, + burst_sz); + + if (deq < burst_sz) { + printf( + "After receiving the interrupt all operations should be dequeued. Expected: %u, got: %u\n", + burst_sz, deq); + rte_atomic16_set(&tp->processing_status, TEST_FAILED); + return; + } + + if (rte_atomic16_read(&tp->nb_dequeued) + deq < num_to_process) { + rte_atomic16_add(&tp->nb_dequeued, deq); + return; + } + + total_time = rte_rdtsc_precise() - tp->start_time; + + rte_bbdev_info_get(dev_id, &info); + + bufs = &tp->op_params->q_bufs[GET_SOCKET(info.socket_id)][queue_id]; + + ret = TEST_SUCCESS; + if (test_vector.op_type == RTE_BBDEV_OP_TURBO_DEC) + ret = validate_dec_buffers(tp->op_params->ref_dec_op, bufs, + num_to_process); + else if (test_vector.op_type == RTE_BBDEV_OP_TURBO_ENC) + ret = validate_dec_buffers(tp->op_params->ref_dec_op, bufs, + num_to_process); + + if (ret) { + printf("Buffers validation failed\n"); + rte_atomic16_set(&tp->processing_status, TEST_FAILED); + } + + switch (test_vector.op_type) { + case RTE_BBDEV_OP_TURBO_DEC: + in_len = tp->op_params->ref_dec_op->turbo_dec.input.length; + break; + case RTE_BBDEV_OP_TURBO_ENC: + in_len = tp->op_params->ref_enc_op->turbo_enc.input.length; + break; + case RTE_BBDEV_OP_NONE: + in_len = 0.0; + break; + default: + printf("Unknown op type: %d\n", test_vector.op_type); + rte_atomic16_set(&tp->processing_status, TEST_FAILED); + return; + } + + mops = ((double)num_to_process / 1000000.0) / + ((double)total_time / (double)rte_get_tsc_hz()); + mbps = ((double)num_to_process * in_len * 8 / 1000000.0) / + ((double)total_time / (double)rte_get_tsc_hz()); + + if (test_vector.op_type != RTE_BBDEV_OP_NONE) + printf("\tqueue_id: %u, throughput: %lg MOPS, %lg Mbps\n", + queue_id, mops, mbps); + else + printf("\tqueue_id: %u, throughput: %lg MOPS\n", + queue_id, mops); + + rte_atomic16_add(&tp->nb_dequeued, deq); +} + +static int +throughput_intr_lcore_dec(void *arg) +{ + struct thread_params *tp = arg; + unsigned int enqueued; + struct rte_bbdev_dec_op *ops[MAX_BURST]; + const uint16_t queue_id = tp->queue_id; + const uint16_t burst_sz = tp->op_params->burst_sz; + const uint16_t num_to_process = tp->op_params->num_to_process; + struct test_buffers *bufs = NULL; + unsigned int allocs_failed = 0; + struct rte_bbdev_info info; + int ret; + + if (burst_sz > MAX_BURST) { + printf("Operations table size > MAX_BURST\n"); + return TEST_FAILED; + } + + TEST_ASSERT_SUCCESS(rte_bbdev_queue_intr_enable(tp->dev_id, queue_id), + "Failed to enable interrupts for dev: %u, queue_id: %u", + tp->dev_id, queue_id); + + rte_bbdev_info_get(tp->dev_id, &info); + bufs = &tp->op_params->q_bufs[GET_SOCKET(info.socket_id)][queue_id]; + + rte_atomic16_clear(&tp->processing_status); + rte_atomic16_clear(&tp->nb_dequeued); + + while (rte_atomic16_read(&tp->op_params->sync) == SYNC_WAIT) + rte_pause(); + + tp->start_time = rte_rdtsc_precise(); + for (enqueued = 0; enqueued < num_to_process;) { + + uint16_t num_to_enq = burst_sz; + + if (unlikely(num_to_process - enqueued < num_to_enq)) + num_to_enq = num_to_process - enqueued; + + ret = rte_bbdev_dec_op_alloc_bulk(tp->op_params->mp, ops, + num_to_enq); + if (ret != 0) { + allocs_failed++; + continue; + } + + if (test_vector.op_type != RTE_BBDEV_OP_NONE) + copy_reference_dec_op(ops, num_to_enq, enqueued, + bufs->inputs, + bufs->hard_outputs, + bufs->soft_outputs, + tp->op_params->ref_dec_op); + + enqueued += rte_bbdev_enqueue_dec_ops(tp->dev_id, queue_id, ops, + num_to_enq); + + rte_bbdev_dec_op_free_bulk(ops, num_to_enq); + } + + if (allocs_failed > 0) + printf("WARNING: op allocations failed: %u times\n", + allocs_failed); + + return TEST_SUCCESS; +} + +static int +throughput_intr_lcore_enc(void *arg) +{ + struct thread_params *tp = arg; + unsigned int enqueued; + struct rte_bbdev_enc_op *ops[MAX_BURST]; + const uint16_t queue_id = tp->queue_id; + const uint16_t burst_sz = tp->op_params->burst_sz; + const uint16_t num_to_process = tp->op_params->num_to_process; + struct test_buffers *bufs = NULL; + unsigned int allocs_failed = 0; + struct rte_bbdev_info info; + int ret; + + if (burst_sz > MAX_BURST) { + printf("Operations table size > MAX_BURST\n"); + return TEST_FAILED; + } + + TEST_ASSERT_SUCCESS(rte_bbdev_queue_intr_enable(tp->dev_id, queue_id), + "Failed to enable interrupts for dev: %u, queue_id: %u", + tp->dev_id, queue_id); + + rte_bbdev_info_get(tp->dev_id, &info); + bufs = &tp->op_params->q_bufs[GET_SOCKET(info.socket_id)][queue_id]; + + rte_atomic16_clear(&tp->processing_status); + rte_atomic16_clear(&tp->nb_dequeued); + + while (rte_atomic16_read(&tp->op_params->sync) == SYNC_WAIT) + rte_pause(); + + tp->start_time = rte_rdtsc_precise(); + for (enqueued = 0; enqueued < num_to_process;) { + + uint16_t num_to_enq = burst_sz; + + if (unlikely(num_to_process - enqueued < num_to_enq)) + num_to_enq = num_to_process - enqueued; + + ret = rte_bbdev_enc_op_alloc_bulk(tp->op_params->mp, ops, + num_to_enq); + if (ret != 0) { + allocs_failed++; + continue; + } + + if (test_vector.op_type != RTE_BBDEV_OP_NONE) + copy_reference_enc_op(ops, num_to_enq, enqueued, + bufs->inputs, + bufs->hard_outputs, + tp->op_params->ref_enc_op); + + enqueued += rte_bbdev_enqueue_enc_ops(tp->dev_id, queue_id, ops, + num_to_enq); + + rte_bbdev_enc_op_free_bulk(ops, num_to_enq); + } + + if (allocs_failed > 0) + printf("WARNING: op allocations failed: %u times\n", + allocs_failed); + + return TEST_SUCCESS; +} + +static int +throughput_pmd_lcore_dec(void *arg) +{ + struct thread_params *tp = arg; + unsigned int enqueued, dequeued; + struct rte_bbdev_dec_op *ops[MAX_BURST]; + uint64_t total_time, start_time; + const uint16_t queue_id = tp->queue_id; + const uint16_t burst_sz = tp->op_params->burst_sz; + const uint16_t num_to_process = tp->op_params->num_to_process; + struct rte_bbdev_dec_op *ref_op = tp->op_params->ref_dec_op; + struct test_buffers *bufs = NULL; + unsigned int allocs_failed = 0; + int ret; + struct rte_bbdev_info info; + + /* Input length in bytes, million operations per second, million bits + * per second. + */ + double in_len, mops, mbps; + + if (burst_sz > MAX_BURST) { + printf("Operations table size > MAX_BURST\n"); + return TEST_FAILED; + } + + rte_bbdev_info_get(tp->dev_id, &info); + bufs = &tp->op_params->q_bufs[GET_SOCKET(info.socket_id)][queue_id]; + + while (rte_atomic16_read(&tp->op_params->sync) == SYNC_WAIT) + rte_pause(); + + start_time = rte_rdtsc_precise(); + for (enqueued = 0, dequeued = 0; dequeued < num_to_process;) { + uint16_t deq; + + if (likely(enqueued < num_to_process)) { + + uint16_t num_to_enq = burst_sz; + + if (unlikely(num_to_process - enqueued < num_to_enq)) + num_to_enq = num_to_process - enqueued; + + ret = rte_bbdev_dec_op_alloc_bulk(tp->op_params->mp, + ops, num_to_enq); + if (ret != 0) { + allocs_failed++; + goto do_dequeue; + } + + if (test_vector.op_type != RTE_BBDEV_OP_NONE) + copy_reference_dec_op(ops, num_to_enq, enqueued, + bufs->inputs, + bufs->hard_outputs, + bufs->soft_outputs, + ref_op); + + enqueued += rte_bbdev_enqueue_dec_ops(tp->dev_id, + queue_id, ops, num_to_enq); + } +do_dequeue: + deq = rte_bbdev_dequeue_dec_ops(tp->dev_id, queue_id, ops, + burst_sz); + dequeued += deq; + rte_bbdev_dec_op_free_bulk(ops, deq); + } + total_time = rte_rdtsc_precise() - start_time; + + if (allocs_failed > 0) + printf("WARNING: op allocations failed: %u times\n", + allocs_failed); + + TEST_ASSERT(enqueued == dequeued, "enqueued (%u) != dequeued (%u)", + enqueued, dequeued); + + if (test_vector.op_type != RTE_BBDEV_OP_NONE) { + ret = validate_dec_buffers(ref_op, bufs, num_to_process); + TEST_ASSERT_SUCCESS(ret, "Buffers validation failed"); + } + + in_len = ref_op->turbo_dec.input.length; + mops = ((double)num_to_process / 1000000.0) / + ((double)total_time / (double)rte_get_tsc_hz()); + mbps = ((double)in_len * 8 / 1000000.0) / + ((double)total_time / (double)rte_get_tsc_hz()); + + printf("\tlcore_id: %u, throughput: %lg MOPS, %lg Mbps\n", + rte_lcore_id(), mops, mbps); + + return TEST_SUCCESS; +} + +static int +throughput_pmd_lcore_enc(void *arg) +{ + struct thread_params *tp = arg; + unsigned int enqueued, dequeued; + struct rte_bbdev_enc_op *ops[MAX_BURST]; + uint64_t total_time, start_time; + const uint16_t queue_id = tp->queue_id; + const uint16_t burst_sz = tp->op_params->burst_sz; + const uint16_t num_to_process = tp->op_params->num_to_process; + struct rte_bbdev_enc_op *ref_op = tp->op_params->ref_enc_op; + struct test_buffers *bufs = NULL; + unsigned int allocs_failed = 0; + int ret; + struct rte_bbdev_info info; + + /* Input length in bytes, million operations per second, million bits + * per second. + */ + double in_len, mops, mbps; + + if (burst_sz > MAX_BURST) { + printf("Operations table size > MAX_BURST\n"); + return TEST_FAILED; + } + + rte_bbdev_info_get(tp->dev_id, &info); + bufs = &tp->op_params->q_bufs[GET_SOCKET(info.socket_id)][queue_id]; + + while (rte_atomic16_read(&tp->op_params->sync) == SYNC_WAIT) + rte_pause(); + + start_time = rte_rdtsc_precise(); + for (enqueued = 0, dequeued = 0; dequeued < num_to_process;) { + uint16_t deq; + + if (likely(enqueued < num_to_process)) { + + uint16_t num_to_enq = burst_sz; + + if (unlikely(num_to_process - enqueued < num_to_enq)) + num_to_enq = num_to_process - enqueued; + + ret = rte_bbdev_enc_op_alloc_bulk(tp->op_params->mp, + ops, num_to_enq); + if (ret != 0) { + allocs_failed++; + goto do_dequeue; + } + + if (test_vector.op_type != RTE_BBDEV_OP_NONE) + copy_reference_enc_op(ops, num_to_enq, enqueued, + bufs->inputs, + bufs->hard_outputs, + ref_op); + + enqueued += rte_bbdev_enqueue_enc_ops(tp->dev_id, + queue_id, ops, num_to_enq); + } +do_dequeue: + deq = rte_bbdev_dequeue_enc_ops(tp->dev_id, queue_id, ops, + burst_sz); + dequeued += deq; + rte_bbdev_enc_op_free_bulk(ops, deq); + } + total_time = rte_rdtsc_precise() - start_time; + + if (allocs_failed > 0) + printf("WARNING: op allocations failed: %u times\n", + allocs_failed); + + TEST_ASSERT(enqueued == dequeued, "enqueued (%u) != dequeued (%u)", + enqueued, dequeued); + + if (test_vector.op_type != RTE_BBDEV_OP_NONE) { + ret = validate_enc_buffers(bufs, num_to_process); + TEST_ASSERT_SUCCESS(ret, "Buffers validation failed"); + } + + in_len = ref_op->turbo_enc.input.length; + + mops = ((double)num_to_process / 1000000.0) / + ((double)total_time / (double)rte_get_tsc_hz()); + mbps = ((double)in_len * 8 / 1000000.0) / + ((double)total_time / (double)rte_get_tsc_hz()); + + printf("\tlcore_id: %u, throughput: %lg MOPS, %lg Mbps\n", + rte_lcore_id(), mops, mbps); + + return TEST_SUCCESS; +} + +/* + * Test function that determines how long an enqueue + dequeue of a burst + * takes on available lcores. + */ +static int +throughput_test(struct active_device *ad, + struct test_op_params *op_params) +{ + int ret; + unsigned int lcore_id, used_cores = 0; + struct thread_params t_params[MAX_QUEUES]; + struct rte_bbdev_info info; + lcore_function_t *throughput_function; + + rte_bbdev_info_get(ad->dev_id, &info); + + printf( + "Throughput test: dev: %s, burst size: %u, num ops: %u, op type: %s, int mode: %s, GHz: %lg\n", + info.dev_name, op_params->burst_sz, + op_params->num_to_process, + rte_bbdev_op_type_str(test_vector.op_type), + intr_enabled ? "Interrupt mode" : "PMD mode", + (double)rte_get_tsc_hz() / 1000000000.0); + + if (intr_enabled) { + if (test_vector.op_type == RTE_BBDEV_OP_TURBO_DEC) + throughput_function = throughput_intr_lcore_dec; + else + throughput_function = throughput_intr_lcore_enc; + + /* Dequeue interrupt callback registration */ + rte_bbdev_callback_register(ad->dev_id, RTE_BBDEV_EVENT_DEQUEUE, + dequeue_event_callback, + &t_params); + } else { + if (test_vector.op_type == RTE_BBDEV_OP_TURBO_DEC) + throughput_function = throughput_pmd_lcore_dec; + else + throughput_function = throughput_pmd_lcore_enc; + } + + rte_atomic16_set(&op_params->sync, SYNC_WAIT); + + t_params[rte_lcore_id()].dev_id = ad->dev_id; + t_params[rte_lcore_id()].op_params = op_params; + t_params[rte_lcore_id()].queue_id = + ad->queue_ids[used_cores++]; + + RTE_LCORE_FOREACH_SLAVE(lcore_id) { + if (used_cores >= ad->nb_queues) + break; + + t_params[lcore_id].dev_id = ad->dev_id; + t_params[lcore_id].op_params = op_params; + t_params[lcore_id].queue_id = ad->queue_ids[used_cores++]; + + rte_eal_remote_launch(throughput_function, &t_params[lcore_id], + lcore_id); + } + + rte_atomic16_set(&op_params->sync, SYNC_START); + ret = throughput_function(&t_params[rte_lcore_id()]); + + /* Master core is always used */ + used_cores = 1; + RTE_LCORE_FOREACH_SLAVE(lcore_id) { + if (used_cores++ >= ad->nb_queues) + break; + + ret |= rte_eal_wait_lcore(lcore_id); + } + + /* If interrupts are disabled or throughput_intr_lcore failed, return */ + if (!intr_enabled || ret) + return ret; + + /* In interrupt TC we need to wait for the interrupt callback to deqeue + * all pending operations. Skip waiting for queues which reported an + * error using processing_status variable. + */ + used_cores = 1; + RTE_LCORE_FOREACH_SLAVE(lcore_id) { + struct thread_params *tp = &t_params[lcore_id]; + if (used_cores++ >= ad->nb_queues) + break; + + while ((rte_atomic16_read(&tp->nb_dequeued) < + op_params->num_to_process) && + (rte_atomic16_read(&tp->processing_status) != + TEST_FAILED)) + rte_pause(); + + ret |= rte_atomic16_read(&tp->processing_status); + } + + return ret; +} + +static int +operation_latency_test_dec(struct rte_mempool *mempool, + struct test_buffers *bufs, struct rte_bbdev_dec_op *ref_op, + int vector_mask, uint16_t dev_id, uint16_t queue_id, + const uint16_t num_to_process, uint16_t burst_sz, + uint64_t *total_time) +{ + int ret = TEST_SUCCESS; + uint16_t i, j, dequeued; + struct rte_bbdev_dec_op *ops[MAX_BURST]; + uint64_t start_time = 0; + + for (i = 0, dequeued = 0; dequeued < num_to_process; ++i) { + uint16_t enq = 0, deq = 0; + bool first_time = true; + + if (unlikely(num_to_process - dequeued < burst_sz)) + burst_sz = num_to_process - dequeued; + + rte_bbdev_dec_op_alloc_bulk(mempool, ops, burst_sz); + if (test_vector.op_type != RTE_BBDEV_OP_NONE) + copy_reference_dec_op(ops, burst_sz, dequeued, + bufs->inputs, + bufs->hard_outputs, + bufs->soft_outputs, + ref_op); + + /* Set counter to validate the ordering */ + for (j = 0; j < burst_sz; ++j) + ops[j]->opaque_data = (void *)(uintptr_t)j; + + start_time = rte_rdtsc_precise(); + + enq = rte_bbdev_enqueue_dec_ops(dev_id, queue_id, &ops[enq], + burst_sz); + TEST_ASSERT(enq == burst_sz, + "Error enqueueing burst, expected %u, got %u", + burst_sz, enq); + + /* Dequeue */ + do { + deq += rte_bbdev_dequeue_dec_ops(dev_id, queue_id, + &ops[deq], burst_sz - deq); + if (likely(first_time && (deq > 0))) { + *total_time += rte_rdtsc_precise() - start_time; + first_time = false; + } + } while (unlikely(burst_sz != deq)); + + if (test_vector.op_type != RTE_BBDEV_OP_NONE) { + ret = validate_dec_op(ops, burst_sz, ref_op, + vector_mask); + TEST_ASSERT_SUCCESS(ret, "Validation failed!"); + } + + rte_bbdev_dec_op_free_bulk(ops, deq); + dequeued += deq; + } + + return i; +} + +static int +operation_latency_test_enc(struct rte_mempool *mempool, + struct test_buffers *bufs, struct rte_bbdev_enc_op *ref_op, + uint16_t dev_id, uint16_t queue_id, + const uint16_t num_to_process, uint16_t burst_sz, + uint64_t *total_time) +{ + int ret = TEST_SUCCESS; + uint16_t i, j, dequeued; + struct rte_bbdev_enc_op *ops[MAX_BURST]; + uint64_t start_time = 0; + + for (i = 0, dequeued = 0; dequeued < num_to_process; ++i) { + uint16_t enq = 0, deq = 0; + bool first_time = true; + + if (unlikely(num_to_process - dequeued < burst_sz)) + burst_sz = num_to_process - dequeued; + + rte_bbdev_enc_op_alloc_bulk(mempool, ops, burst_sz); + if (test_vector.op_type != RTE_BBDEV_OP_NONE) + copy_reference_enc_op(ops, burst_sz, dequeued, + bufs->inputs, + bufs->hard_outputs, + ref_op); + + /* Set counter to validate the ordering */ + for (j = 0; j < burst_sz; ++j) + ops[j]->opaque_data = (void *)(uintptr_t)j; + + start_time = rte_rdtsc_precise(); + + enq = rte_bbdev_enqueue_enc_ops(dev_id, queue_id, &ops[enq], + burst_sz); + TEST_ASSERT(enq == burst_sz, + "Error enqueueing burst, expected %u, got %u", + burst_sz, enq); + + /* Dequeue */ + do { + deq += rte_bbdev_dequeue_enc_ops(dev_id, queue_id, + &ops[deq], burst_sz - deq); + if (likely(first_time && (deq > 0))) { + *total_time += rte_rdtsc_precise() - start_time; + first_time = false; + } + } while (unlikely(burst_sz != deq)); + + if (test_vector.op_type != RTE_BBDEV_OP_NONE) { + ret = validate_enc_op(ops, burst_sz, ref_op); + TEST_ASSERT_SUCCESS(ret, "Validation failed!"); + } + + rte_bbdev_enc_op_free_bulk(ops, deq); + dequeued += deq; + } + + return i; +} + +static int +operation_latency_test(struct active_device *ad, + struct test_op_params *op_params) +{ + uint16_t iter; + uint16_t burst_sz = op_params->burst_sz; + const uint16_t num_to_process = op_params->num_to_process; + const enum rte_bbdev_op_type op_type = test_vector.op_type; + const uint16_t queue_id = ad->queue_ids[0]; + struct test_buffers *bufs = NULL; + struct rte_bbdev_info info; + uint64_t total_time = 0; + + rte_bbdev_info_get(ad->dev_id, &info); + bufs = &op_params->q_bufs[GET_SOCKET(info.socket_id)][queue_id]; + + printf( + "Validation/Latency test: dev: %s, burst size: %u, num ops: %u, op type: %s\n", + info.dev_name, burst_sz, num_to_process, + rte_bbdev_op_type_str(op_type)); + + if (op_type == RTE_BBDEV_OP_TURBO_DEC) + iter = operation_latency_test_dec(op_params->mp, bufs, + op_params->ref_dec_op, op_params->vector_mask, + ad->dev_id, queue_id, num_to_process, + burst_sz, &total_time); + else + iter = operation_latency_test_enc(op_params->mp, bufs, + op_params->ref_enc_op, ad->dev_id, queue_id, + num_to_process, burst_sz, &total_time); + + printf("\toperation avg. latency: %lg cycles, %lg us\n", + (double)total_time / (double)iter, + (double)(total_time * 1000000) / (double)iter / + (double)rte_get_tsc_hz()); + + return TEST_SUCCESS; +} + +static uint16_t +offload_latency_test_dec(struct rte_mempool *mempool, struct test_buffers *bufs, + struct rte_bbdev_dec_op *ref_op, uint16_t dev_id, + uint16_t queue_id, const uint16_t num_to_process, + uint16_t burst_sz, uint64_t *enq_total_time, + uint64_t *deq_total_time) +{ + uint16_t i, dequeued; + struct rte_bbdev_dec_op *ops[MAX_BURST]; + uint64_t enq_start_time, deq_start_time; + + for (i = 0, dequeued = 0; dequeued < num_to_process; ++i) { + uint16_t enq = 0, deq = 0; + + if (unlikely(num_to_process - dequeued < burst_sz)) + burst_sz = num_to_process - dequeued; + + rte_bbdev_dec_op_alloc_bulk(mempool, ops, burst_sz); + if (test_vector.op_type != RTE_BBDEV_OP_NONE) + copy_reference_dec_op(ops, burst_sz, dequeued, + bufs->inputs, + bufs->hard_outputs, + bufs->soft_outputs, + ref_op); + + /* Start time measurment for enqueue function offload latency */ + enq_start_time = rte_rdtsc(); + do { + enq += rte_bbdev_enqueue_dec_ops(dev_id, queue_id, + &ops[enq], burst_sz - enq); + } while (unlikely(burst_sz != enq)); + *enq_total_time += rte_rdtsc() - enq_start_time; + + /* ensure enqueue has been completed */ + rte_delay_ms(10); + + /* Start time measurment for dequeue function offload latency */ + deq_start_time = rte_rdtsc(); + do { + deq += rte_bbdev_dequeue_dec_ops(dev_id, queue_id, + &ops[deq], burst_sz - deq); + } while (unlikely(burst_sz != deq)); + *deq_total_time += rte_rdtsc() - deq_start_time; + + rte_bbdev_dec_op_free_bulk(ops, deq); + dequeued += deq; + } + + return i; +} + +static uint16_t +offload_latency_test_enc(struct rte_mempool *mempool, struct test_buffers *bufs, + struct rte_bbdev_enc_op *ref_op, uint16_t dev_id, + uint16_t queue_id, const uint16_t num_to_process, + uint16_t burst_sz, uint64_t *enq_total_time, + uint64_t *deq_total_time) +{ + uint16_t i, dequeued; + struct rte_bbdev_enc_op *ops[MAX_BURST]; + uint64_t enq_start_time, deq_start_time; + + for (i = 0, dequeued = 0; dequeued < num_to_process; ++i) { + uint16_t enq = 0, deq = 0; + + if (unlikely(num_to_process - dequeued < burst_sz)) + burst_sz = num_to_process - dequeued; + + rte_bbdev_enc_op_alloc_bulk(mempool, ops, burst_sz); + if (test_vector.op_type != RTE_BBDEV_OP_NONE) + copy_reference_enc_op(ops, burst_sz, dequeued, + bufs->inputs, + bufs->hard_outputs, + ref_op); + + /* Start time measurment for enqueue function offload latency */ + enq_start_time = rte_rdtsc(); + do { + enq += rte_bbdev_enqueue_enc_ops(dev_id, queue_id, + &ops[enq], burst_sz - enq); + } while (unlikely(burst_sz != enq)); + *enq_total_time += rte_rdtsc() - enq_start_time; + + /* ensure enqueue has been completed */ + rte_delay_ms(10); + + /* Start time measurment for dequeue function offload latency */ + deq_start_time = rte_rdtsc(); + do { + deq += rte_bbdev_dequeue_enc_ops(dev_id, queue_id, + &ops[deq], burst_sz - deq); + } while (unlikely(burst_sz != deq)); + *deq_total_time += rte_rdtsc() - deq_start_time; + + rte_bbdev_enc_op_free_bulk(ops, deq); + dequeued += deq; + } + + return i; +} + +static int +offload_latency_test(struct active_device *ad, + struct test_op_params *op_params) +{ + uint16_t iter; + uint64_t enq_total_time = 0, deq_total_time = 0; + uint16_t burst_sz = op_params->burst_sz; + const uint16_t num_to_process = op_params->num_to_process; + const enum rte_bbdev_op_type op_type = test_vector.op_type; + const uint16_t queue_id = ad->queue_ids[0]; + struct test_buffers *bufs = NULL; + struct rte_bbdev_info info; + + rte_bbdev_info_get(ad->dev_id, &info); + bufs = &op_params->q_bufs[GET_SOCKET(info.socket_id)][queue_id]; + + printf( + "Offload latency test: dev: %s, burst size: %u, num ops: %u, op type: %s\n", + info.dev_name, burst_sz, num_to_process, + rte_bbdev_op_type_str(op_type)); + + if (op_type == RTE_BBDEV_OP_TURBO_DEC) + iter = offload_latency_test_dec(op_params->mp, bufs, + op_params->ref_dec_op, ad->dev_id, queue_id, + num_to_process, burst_sz, &enq_total_time, + &deq_total_time); + else + iter = offload_latency_test_enc(op_params->mp, bufs, + op_params->ref_enc_op, ad->dev_id, queue_id, + num_to_process, burst_sz, &enq_total_time, + &deq_total_time); + + printf("\tenq offload avg. latency: %lg cycles, %lg us\n", + (double)enq_total_time / (double)iter, + (double)(enq_total_time * 1000000) / (double)iter / + (double)rte_get_tsc_hz()); + + printf("\tdeq offload avg. latency: %lg cycles, %lg us\n", + (double)deq_total_time / (double)iter, + (double)(deq_total_time * 1000000) / (double)iter / + (double)rte_get_tsc_hz()); + + return TEST_SUCCESS; +} + +static uint16_t +offload_latency_empty_q_test_dec(uint16_t dev_id, uint16_t queue_id, + const uint16_t num_to_process, uint16_t burst_sz, + uint64_t *deq_total_time) +{ + uint16_t i, deq_total; + struct rte_bbdev_dec_op *ops[MAX_BURST]; + uint64_t deq_start_time; + + /* Test deq offload latency from an empty queue */ + deq_start_time = rte_rdtsc_precise(); + for (i = 0, deq_total = 0; deq_total < num_to_process; + ++i, deq_total += burst_sz) { + if (unlikely(num_to_process - deq_total < burst_sz)) + burst_sz = num_to_process - deq_total; + rte_bbdev_dequeue_dec_ops(dev_id, queue_id, ops, burst_sz); + } + *deq_total_time = rte_rdtsc_precise() - deq_start_time; + + return i; +} + +static uint16_t +offload_latency_empty_q_test_enc(uint16_t dev_id, uint16_t queue_id, + const uint16_t num_to_process, uint16_t burst_sz, + uint64_t *deq_total_time) +{ + uint16_t i, deq_total; + struct rte_bbdev_enc_op *ops[MAX_BURST]; + uint64_t deq_start_time; + + /* Test deq offload latency from an empty queue */ + deq_start_time = rte_rdtsc_precise(); + for (i = 0, deq_total = 0; deq_total < num_to_process; + ++i, deq_total += burst_sz) { + if (unlikely(num_to_process - deq_total < burst_sz)) + burst_sz = num_to_process - deq_total; + rte_bbdev_dequeue_enc_ops(dev_id, queue_id, ops, burst_sz); + } + *deq_total_time = rte_rdtsc_precise() - deq_start_time; + + return i; +} + +static int +offload_latency_empty_q_test(struct active_device *ad, + struct test_op_params *op_params) +{ + uint16_t iter; + uint64_t deq_total_time = 0; + uint16_t burst_sz = op_params->burst_sz; + const uint16_t num_to_process = op_params->num_to_process; + const enum rte_bbdev_op_type op_type = test_vector.op_type; + const uint16_t queue_id = ad->queue_ids[0]; + struct rte_bbdev_info info; + + rte_bbdev_info_get(ad->dev_id, &info); + + printf( + "Offload latency empty dequeue test: dev: %s, burst size: %u, num ops: %u, op type: %s\n", + info.dev_name, burst_sz, num_to_process, + rte_bbdev_op_type_str(op_type)); + + if (op_type == RTE_BBDEV_OP_TURBO_DEC) + iter = offload_latency_empty_q_test_dec(ad->dev_id, queue_id, + num_to_process, burst_sz, &deq_total_time); + else + iter = offload_latency_empty_q_test_enc(ad->dev_id, queue_id, + num_to_process, burst_sz, &deq_total_time); + + printf("\tempty deq offload avg. latency: %lg cycles, %lg us\n", + (double)deq_total_time / (double)iter, + (double)(deq_total_time * 1000000) / (double)iter / + (double)rte_get_tsc_hz()); + + return TEST_SUCCESS; +} + +static int +throughput_tc(void) +{ + return run_test_case(throughput_test); +} + +static int +offload_latency_tc(void) +{ + return run_test_case(offload_latency_test); +} + +static int +offload_latency_empty_q_tc(void) +{ + return run_test_case(offload_latency_empty_q_test); +} + +static int +operation_latency_tc(void) +{ + return run_test_case(operation_latency_test); +} + +static int +interrupt_tc(void) +{ + return run_test_case(throughput_test); +} + +static struct unit_test_suite bbdev_throughput_testsuite = { + .suite_name = "BBdev Throughput Tests", + .setup = testsuite_setup, + .teardown = testsuite_teardown, + .unit_test_cases = { + TEST_CASE_ST(ut_setup, ut_teardown, throughput_tc), + TEST_CASES_END() /**< NULL terminate unit test array */ + } +}; + +static struct unit_test_suite bbdev_validation_testsuite = { + .suite_name = "BBdev Validation Tests", + .setup = testsuite_setup, + .teardown = testsuite_teardown, + .unit_test_cases = { + TEST_CASE_ST(ut_setup, ut_teardown, operation_latency_tc), + TEST_CASES_END() /**< NULL terminate unit test array */ + } +}; + +static struct unit_test_suite bbdev_latency_testsuite = { + .suite_name = "BBdev Latency Tests", + .setup = testsuite_setup, + .teardown = testsuite_teardown, + .unit_test_cases = { + TEST_CASE_ST(ut_setup, ut_teardown, offload_latency_tc), + TEST_CASE_ST(ut_setup, ut_teardown, offload_latency_empty_q_tc), + TEST_CASE_ST(ut_setup, ut_teardown, operation_latency_tc), + TEST_CASES_END() /**< NULL terminate unit test array */ + } +}; + +static struct unit_test_suite bbdev_interrupt_testsuite = { + .suite_name = "BBdev Interrupt Tests", + .setup = interrupt_testsuite_setup, + .teardown = testsuite_teardown, + .unit_test_cases = { + TEST_CASE_ST(ut_setup, ut_teardown, interrupt_tc), + TEST_CASES_END() /**< NULL terminate unit test array */ + } +}; + +REGISTER_TEST_COMMAND(throughput, bbdev_throughput_testsuite); +REGISTER_TEST_COMMAND(validation, bbdev_validation_testsuite); +REGISTER_TEST_COMMAND(latency, bbdev_latency_testsuite); +REGISTER_TEST_COMMAND(interrupt, bbdev_interrupt_testsuite); diff --git a/app/test-bbdev/test_bbdev_vector.c b/app/test-bbdev/test_bbdev_vector.c new file mode 100644 index 0000000..b9e1340 --- /dev/null +++ b/app/test-bbdev/test_bbdev_vector.c @@ -0,0 +1,884 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifdef RTE_EXEC_ENV_BSDAPP + #define _WITH_GETLINE +#endif +#include <stdio.h> +#include <stdbool.h> +#include <rte_malloc.h> + +#include "test_bbdev_vector.h" + +#define VALUE_DELIMITER "," +#define ENTRY_DELIMITER "=" + +const char *op_data_prefixes[] = { + "input", + "soft_output", + "hard_output", +}; + +/* trim leading and trailing spaces */ +static void +trim_space(char *str) +{ + char *start, *end; + + for (start = str; *start; start++) { + if (!isspace((unsigned char) start[0])) + break; + } + + for (end = start + strlen(start); end > start + 1; end--) { + if (!isspace((unsigned char) end[-1])) + break; + } + + *end = 0; + + /* Shift from "start" to the beginning of the string */ + if (start > str) + memmove(str, start, (end - start) + 1); +} + +static bool +starts_with(const char *str, const char *pre) +{ + return strncmp(pre, str, strlen(pre)) == 0; +} + +/* tokenization test values separated by a comma */ +static int +parse_values(char *tokens, uint32_t **data, uint32_t *data_length) +{ + uint32_t n_tokens = 0; + uint32_t data_size = 32; + + uint32_t *values, *values_resized; + char *tok, *error = NULL; + + tok = strtok(tokens, VALUE_DELIMITER); + if (tok == NULL) + return -1; + + values = (uint32_t *) + rte_zmalloc(NULL, sizeof(uint32_t) * data_size, 0); + if (values == NULL) + return -1; + + while (tok != NULL) { + values_resized = NULL; + + if (n_tokens >= data_size) { + data_size *= 2; + + values_resized = (uint32_t *) rte_realloc(values, + sizeof(uint32_t) * data_size, 0); + if (values_resized == NULL) { + rte_free(values); + return -1; + } + values = values_resized; + } + + values[n_tokens] = (uint32_t) strtoul(tok, &error, 0); + if ((error == NULL) || (*error != '\0')) { + printf("Failed with convert '%s'\n", tok); + rte_free(values); + return -1; + } + + *data_length = *data_length + (strlen(tok) - strlen("0x"))/2; + + tok = strtok(NULL, VALUE_DELIMITER); + if (tok == NULL) + break; + + n_tokens++; + } + + values_resized = (uint32_t *) rte_realloc(values, + sizeof(uint32_t) * (n_tokens + 1), 0); + + if (values_resized == NULL) { + rte_free(values); + return -1; + } + + *data = values_resized; + + return 0; +} + +/* convert turbo decoder flag from string to unsigned long int*/ +static int +op_decoder_flag_strtoul(char *token, uint32_t *op_flag_value) +{ + if (!strcmp(token, "RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE")) + *op_flag_value = RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE; + else if (!strcmp(token, "RTE_BBDEV_TURBO_CRC_TYPE_24B")) + *op_flag_value = RTE_BBDEV_TURBO_CRC_TYPE_24B; + else if (!strcmp(token, "RTE_BBDEV_TURBO_EQUALIZER")) + *op_flag_value = RTE_BBDEV_TURBO_EQUALIZER; + else if (!strcmp(token, "RTE_BBDEV_TURBO_SOFT_OUT_SATURATE")) + *op_flag_value = RTE_BBDEV_TURBO_SOFT_OUT_SATURATE; + else if (!strcmp(token, "RTE_BBDEV_TURBO_HALF_ITERATION_EVEN")) + *op_flag_value = RTE_BBDEV_TURBO_HALF_ITERATION_EVEN; + else if (!strcmp(token, "RTE_BBDEV_TURBO_CONTINUE_CRC_MATCH")) + *op_flag_value = RTE_BBDEV_TURBO_CONTINUE_CRC_MATCH; + else if (!strcmp(token, "RTE_BBDEV_TURBO_SOFT_OUTPUT")) + *op_flag_value = RTE_BBDEV_TURBO_SOFT_OUTPUT; + else if (!strcmp(token, "RTE_BBDEV_TURBO_EARLY_TERMINATION")) + *op_flag_value = RTE_BBDEV_TURBO_EARLY_TERMINATION; + else if (!strcmp(token, "RTE_BBDEV_TURBO_RAW_INPUT_DATA")) + *op_flag_value = RTE_BBDEV_TURBO_RAW_INPUT_DATA; + else { + printf("The given value is not a turbo decoder flag\n"); + return -1; + } + + return 0; +} + +/* convert turbo encoder flag from string to unsigned long int*/ +static int +op_encoder_flag_strtoul(char *token, uint32_t *op_flag_value) +{ + if (!strcmp(token, "RTE_BBDEV_TURBO_RV_INDEX_BYPASS")) + *op_flag_value = RTE_BBDEV_TURBO_RV_INDEX_BYPASS; + else if (!strcmp(token, "RTE_BBDEV_TURBO_RATE_MATCH")) + *op_flag_value = RTE_BBDEV_TURBO_RATE_MATCH; + else if (!strcmp(token, "RTE_BBDEV_TURBO_CRC_24B_ATTACH")) + *op_flag_value = RTE_BBDEV_TURBO_CRC_24B_ATTACH; + else if (!strcmp(token, "RTE_BBDEV_TURBO_CRC_24A_ATTACH")) + *op_flag_value = RTE_BBDEV_TURBO_CRC_24A_ATTACH; + else { + printf("The given value is not a turbo encoder flag\n"); + return -1; + } + + return 0; +} + +/* tokenization turbo decoder/encoder flags values separated by a comma */ +static int +parse_turbo_flags(char *tokens, uint32_t *op_flags, + enum rte_bbdev_op_type op_type) +{ + char *tok = NULL; + uint32_t op_flag_value = 0; + + tok = strtok(tokens, VALUE_DELIMITER); + if (tok == NULL) + return -1; + + while (tok != NULL) { + trim_space(tok); + if (op_type == RTE_BBDEV_OP_TURBO_DEC) { + if (op_decoder_flag_strtoul(tok, &op_flag_value) == -1) + return -1; + } else if (op_type == RTE_BBDEV_OP_TURBO_ENC) { + if (op_encoder_flag_strtoul(tok, &op_flag_value) == -1) + return -1; + } else { + return -1; + } + + *op_flags = *op_flags | op_flag_value; + + tok = strtok(NULL, VALUE_DELIMITER); + if (tok == NULL) + break; + } + + return 0; +} + +/* convert turbo encoder/decoder op_type from string to enum*/ +static int +op_turbo_type_strtol(char *token, enum rte_bbdev_op_type *op_type) +{ + trim_space(token); + if (!strcmp(token, "RTE_BBDEV_OP_TURBO_DEC")) + *op_type = RTE_BBDEV_OP_TURBO_DEC; + else if (!strcmp(token, "RTE_BBDEV_OP_TURBO_ENC")) + *op_type = RTE_BBDEV_OP_TURBO_ENC; + else if (!strcmp(token, "RTE_BBDEV_OP_NONE")) + *op_type = RTE_BBDEV_OP_NONE; + else { + printf("Not valid turbo op_type: '%s'\n", token); + return -1; + } + + return 0; +} + +/* tokenization expected status values separated by a comma */ +static int +parse_expected_status(char *tokens, int *status, enum rte_bbdev_op_type op_type) +{ + char *tok = NULL; + bool status_ok = false; + + tok = strtok(tokens, VALUE_DELIMITER); + if (tok == NULL) + return -1; + + while (tok != NULL) { + trim_space(tok); + if (!strcmp(tok, "OK")) + status_ok = true; + else if (!strcmp(tok, "DMA")) + *status = *status | (1 << RTE_BBDEV_DRV_ERROR); + else if (!strcmp(tok, "FCW")) + *status = *status | (1 << RTE_BBDEV_DATA_ERROR); + else if (!strcmp(tok, "CRC")) { + if (op_type == RTE_BBDEV_OP_TURBO_DEC) + *status = *status | (1 << RTE_BBDEV_CRC_ERROR); + else { + printf( + "CRC is only a valid value for turbo decoder\n"); + return -1; + } + } else { + printf("Not valid status: '%s'\n", tok); + return -1; + } + + tok = strtok(NULL, VALUE_DELIMITER); + if (tok == NULL) + break; + } + + if (status_ok && *status != 0) { + printf( + "Not valid status values. Cannot be OK and ERROR at the same time.\n"); + return -1; + } + + return 0; +} + +/* parse ops data entry (there can be more than 1 input entry, each will be + * contained in a separate op_data_buf struct) + */ +static int +parse_data_entry(const char *key_token, char *token, + struct test_bbdev_vector *vector, enum op_data_type type, + const char *prefix) +{ + int ret; + uint32_t data_length = 0; + uint32_t *data = NULL; + unsigned int id; + struct op_data_buf *op_data; + unsigned int *nb_ops; + + if (type > DATA_NUM_TYPES) { + printf("Unknown op type: %d!\n", type); + return -1; + } + + op_data = vector->entries[type].segments; + nb_ops = &vector->entries[type].nb_segments; + + if (*nb_ops >= RTE_BBDEV_MAX_CODE_BLOCKS) { + printf("Too many segments (code blocks defined): %u, max %d!\n", + *nb_ops, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + + if (sscanf(key_token + strlen(prefix), "%u", &id) != 1) { + printf("Missing ID of %s\n", prefix); + return -1; + } + if (id != *nb_ops) { + printf( + "Please order data entries sequentially, i.e. %s0, %s1, ...\n", + prefix, prefix); + return -1; + } + + /* Clear new op data struct */ + memset(op_data + *nb_ops, 0, sizeof(struct op_data_buf)); + + ret = parse_values(token, &data, &data_length); + if (!ret) { + op_data[*nb_ops].addr = data; + op_data[*nb_ops].length = data_length; + ++(*nb_ops); + } + + return ret; +} + +/* parses turbo decoder parameters and assigns to global variable */ +static int +parse_decoder_params(const char *key_token, char *token, + struct test_bbdev_vector *vector) +{ + int ret = 0, status = 0; + uint32_t op_flags = 0; + char *err = NULL; + + struct rte_bbdev_op_turbo_dec *turbo_dec = &vector->turbo_dec; + + /* compare keys */ + if (starts_with(key_token, op_data_prefixes[DATA_INPUT])) + ret = parse_data_entry(key_token, token, vector, + DATA_INPUT, op_data_prefixes[DATA_INPUT]); + + else if (starts_with(key_token, op_data_prefixes[DATA_SOFT_OUTPUT])) + ret = parse_data_entry(key_token, token, vector, + DATA_SOFT_OUTPUT, + op_data_prefixes[DATA_SOFT_OUTPUT]); + + else if (starts_with(key_token, op_data_prefixes[DATA_HARD_OUTPUT])) + ret = parse_data_entry(key_token, token, vector, + DATA_HARD_OUTPUT, + op_data_prefixes[DATA_HARD_OUTPUT]); + else if (!strcmp(key_token, "e")) { + vector->mask |= TEST_BBDEV_VF_E; + turbo_dec->cb_params.e = (uint32_t) strtoul(token, &err, 0); + } else if (!strcmp(key_token, "ea")) { + vector->mask |= TEST_BBDEV_VF_EA; + turbo_dec->tb_params.ea = (uint32_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "eb")) { + vector->mask |= TEST_BBDEV_VF_EB; + turbo_dec->tb_params.eb = (uint32_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "k")) { + vector->mask |= TEST_BBDEV_VF_K; + turbo_dec->cb_params.k = (uint16_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "k_pos")) { + vector->mask |= TEST_BBDEV_VF_K_POS; + turbo_dec->tb_params.k_pos = (uint16_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "k_neg")) { + vector->mask |= TEST_BBDEV_VF_K_NEG; + turbo_dec->tb_params.k_neg = (uint16_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "c")) { + vector->mask |= TEST_BBDEV_VF_C; + turbo_dec->tb_params.c = (uint16_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "c_neg")) { + vector->mask |= TEST_BBDEV_VF_C_NEG; + turbo_dec->tb_params.c_neg = (uint16_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "cab")) { + vector->mask |= TEST_BBDEV_VF_CAB; + turbo_dec->tb_params.cab = (uint8_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "rv_index")) { + vector->mask |= TEST_BBDEV_VF_RV_INDEX; + turbo_dec->rv_index = (uint8_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "iter_max")) { + vector->mask |= TEST_BBDEV_VF_ITER_MAX; + turbo_dec->iter_max = (uint8_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "iter_min")) { + vector->mask |= TEST_BBDEV_VF_ITER_MIN; + turbo_dec->iter_min = (uint8_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "expected_iter_count")) { + vector->mask |= TEST_BBDEV_VF_EXPECTED_ITER_COUNT; + turbo_dec->iter_count = (uint8_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "ext_scale")) { + vector->mask |= TEST_BBDEV_VF_EXT_SCALE; + turbo_dec->ext_scale = (uint8_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "num_maps")) { + vector->mask |= TEST_BBDEV_VF_NUM_MAPS; + turbo_dec->num_maps = (uint8_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "code_block_mode")) { + vector->mask |= TEST_BBDEV_VF_CODE_BLOCK_MODE; + turbo_dec->code_block_mode = (uint8_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "op_flags")) { + vector->mask |= TEST_BBDEV_VF_OP_FLAGS; + ret = parse_turbo_flags(token, &op_flags, + vector->op_type); + if (!ret) + turbo_dec->op_flags = op_flags; + } else if (!strcmp(key_token, "expected_status")) { + vector->mask |= TEST_BBDEV_VF_EXPECTED_STATUS; + ret = parse_expected_status(token, &status, vector->op_type); + if (!ret) + vector->expected_status = status; + } else { + printf("Not valid dec key: '%s'\n", key_token); + return -1; + } + + if (ret != 0) { + printf("Failed with convert '%s\t%s'\n", key_token, token); + return -1; + } + + return 0; +} + +/* parses turbo encoder parameters and assigns to global variable */ +static int +parse_encoder_params(const char *key_token, char *token, + struct test_bbdev_vector *vector) +{ + int ret = 0, status = 0; + uint32_t op_flags = 0; + char *err = NULL; + + + struct rte_bbdev_op_turbo_enc *turbo_enc = &vector->turbo_enc; + + if (starts_with(key_token, op_data_prefixes[DATA_INPUT])) + ret = parse_data_entry(key_token, token, vector, + DATA_INPUT, op_data_prefixes[DATA_INPUT]); + else if (starts_with(key_token, "output")) + ret = parse_data_entry(key_token, token, vector, + DATA_HARD_OUTPUT, "output"); + else if (!strcmp(key_token, "e")) { + vector->mask |= TEST_BBDEV_VF_E; + turbo_enc->cb_params.e = (uint32_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "ea")) { + vector->mask |= TEST_BBDEV_VF_EA; + turbo_enc->tb_params.ea = (uint32_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "eb")) { + vector->mask |= TEST_BBDEV_VF_EB; + turbo_enc->tb_params.eb = (uint32_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "k")) { + vector->mask |= TEST_BBDEV_VF_K; + turbo_enc->cb_params.k = (uint16_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "k_neg")) { + vector->mask |= TEST_BBDEV_VF_K_NEG; + turbo_enc->tb_params.k_neg = (uint16_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "k_pos")) { + vector->mask |= TEST_BBDEV_VF_K_POS; + turbo_enc->tb_params.k_pos = (uint16_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "c_neg")) { + vector->mask |= TEST_BBDEV_VF_C_NEG; + turbo_enc->tb_params.c_neg = (uint8_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "c")) { + vector->mask |= TEST_BBDEV_VF_C; + turbo_enc->tb_params.c = (uint8_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "cab")) { + vector->mask |= TEST_BBDEV_VF_CAB; + turbo_enc->tb_params.cab = (uint8_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "rv_index")) { + vector->mask |= TEST_BBDEV_VF_RV_INDEX; + turbo_enc->rv_index = (uint8_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "ncb")) { + vector->mask |= TEST_BBDEV_VF_NCB; + turbo_enc->cb_params.ncb = (uint16_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "ncb_neg")) { + vector->mask |= TEST_BBDEV_VF_NCB_NEG; + turbo_enc->tb_params.ncb_neg = + (uint16_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "ncb_pos")) { + vector->mask |= TEST_BBDEV_VF_NCB_POS; + turbo_enc->tb_params.ncb_pos = + (uint16_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "code_block_mode")) { + vector->mask |= TEST_BBDEV_VF_CODE_BLOCK_MODE; + turbo_enc->code_block_mode = (uint8_t) strtoul(token, &err, 0); + ret = ((err == NULL) || (*err != '\0')) ? -1 : 0; + } else if (!strcmp(key_token, "op_flags")) { + vector->mask |= TEST_BBDEV_VF_OP_FLAGS; + ret = parse_turbo_flags(token, &op_flags, + vector->op_type); + if (!ret) + turbo_enc->op_flags = op_flags; + } else if (!strcmp(key_token, "expected_status")) { + vector->mask |= TEST_BBDEV_VF_EXPECTED_STATUS; + ret = parse_expected_status(token, &status, vector->op_type); + if (!ret) + vector->expected_status = status; + } else { + printf("Not valid enc key: '%s'\n", key_token); + return -1; + } + + if (ret != 0) { + printf("Failed with convert '%s\t%s'\n", key_token, token); + return -1; + } + + return 0; +} + +/* checks the type of key and assigns data */ +static int +parse_entry(char *entry, struct test_bbdev_vector *vector) +{ + int ret = 0; + char *token, *key_token; + enum rte_bbdev_op_type op_type = RTE_BBDEV_OP_NONE; + + if (entry == NULL) { + printf("Expected entry value\n"); + return -1; + } + + /* get key */ + token = strtok(entry, ENTRY_DELIMITER); + key_token = token; + /* get values for key */ + token = strtok(NULL, ENTRY_DELIMITER); + + if (key_token == NULL || token == NULL) { + printf("Expected 'key = values' but was '%.40s'..\n", entry); + return -1; + } + trim_space(key_token); + + /* first key_token has to specify type of operation */ + if (vector->op_type == RTE_BBDEV_OP_NONE) { + if (!strcmp(key_token, "op_type")) { + ret = op_turbo_type_strtol(token, &op_type); + if (!ret) + vector->op_type = op_type; + return (!ret) ? 0 : -1; + } + printf("First key_token (%s) does not specify op_type\n", + key_token); + return -1; + } + + /* compare keys */ + if (vector->op_type == RTE_BBDEV_OP_TURBO_DEC) { + if (parse_decoder_params(key_token, token, vector) == -1) + return -1; + } else if (vector->op_type == RTE_BBDEV_OP_TURBO_ENC) { + if (parse_encoder_params(key_token, token, vector) == -1) + return -1; + } + + return 0; +} + +/* checks decoder parameters */ +static int +check_decoder(struct test_bbdev_vector *vector, const int mask) +{ + unsigned char i; + struct rte_bbdev_op_turbo_dec *turbo_dec = &vector->turbo_dec; + + if (vector->entries[DATA_INPUT].nb_segments == 0) + return -1; + + for (i = 0; i < vector->entries[DATA_INPUT].nb_segments; i++) + if (vector->entries[DATA_INPUT]. + segments[i].addr == NULL) + return -1; + + if (vector->entries[DATA_HARD_OUTPUT].nb_segments == 0) + return -1; + + for (i = 0; i < vector->entries[DATA_HARD_OUTPUT].nb_segments; + i++) + if (vector->entries[DATA_HARD_OUTPUT]. + segments[i].addr == NULL) + return -1; + + if ((turbo_dec->op_flags & RTE_BBDEV_TURBO_SOFT_OUTPUT) && + (vector->entries[DATA_SOFT_OUTPUT].nb_segments == 0)) + return -1; + + for (i = 0; i < vector->entries[DATA_SOFT_OUTPUT].nb_segments; + i++) + if (vector->entries[DATA_SOFT_OUTPUT]. + segments[i].addr == NULL) + return -1; + + if (!(mask & TEST_BBDEV_VF_CODE_BLOCK_MODE)) { + printf( + "WARNING: code_block_mode was not specified in vector file and will be set to 1 (0 - TB Mode, 1 - CB mode)\n"); + turbo_dec->code_block_mode = 1; + } + if (turbo_dec->code_block_mode == 0) { + if (!(mask & TEST_BBDEV_VF_EA)) + printf( + "WARNING: ea was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_EB)) + printf( + "WARNING: eb was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_K_NEG)) + printf( + "WARNING: k_neg was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_K_POS)) + printf( + "WARNING: k_pos was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_C_NEG)) + printf( + "WARNING: c_neg was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_C)) { + printf( + "WARNING: c was not specified in vector file and will be set to 1\n"); + turbo_dec->tb_params.c = 1; + } + if (!(mask & TEST_BBDEV_VF_CAB)) + printf( + "WARNING: cab was not specified in vector file and will be set to 0\n"); + } else { + if (!(mask & TEST_BBDEV_VF_E)) + printf( + "WARNING: e was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_K)) + printf( + "WARNING: k was not specified in vector file and will be set to 0\n"); + } + if (!(mask & TEST_BBDEV_VF_RV_INDEX)) + printf( + "WARNING: rv_index was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_ITER_MIN)) + printf( + "WARNING: iter_min was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_ITER_MAX)) + printf( + "WARNING: iter_max was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_EXPECTED_ITER_COUNT)) + printf( + "WARNING: expected_iter_count was not specified in vector file and iter_count will not be validated\n"); + if (!(mask & TEST_BBDEV_VF_EXT_SCALE)) + printf( + "WARNING: ext_scale was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_NUM_MAPS)) + printf( + "WARNING: num_maps was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_OP_FLAGS)) + printf( + "WARNING: op_flags was not specified in vector file and capabilities will not be validated\n"); + if (!(mask & TEST_BBDEV_VF_EXPECTED_STATUS)) + printf( + "WARNING: expected_status was not specified in vector file and will be set to 0\n"); + return 0; +} + +/* checks encoder parameters */ +static int +check_encoder(struct test_bbdev_vector *vector, const int mask) +{ + unsigned char i; + + if (vector->entries[DATA_INPUT].nb_segments == 0) + return -1; + + for (i = 0; i < vector->entries[DATA_INPUT].nb_segments; i++) + if (vector->entries[DATA_INPUT]. + segments[i].addr == NULL) + return -1; + + if (vector->entries[DATA_HARD_OUTPUT].nb_segments == 0) + return -1; + + for (i = 0; i < vector->entries[DATA_HARD_OUTPUT].nb_segments; + i++) + if (vector->entries[DATA_HARD_OUTPUT]. + segments[i].addr == NULL) + return -1; + + if (!(mask & TEST_BBDEV_VF_CODE_BLOCK_MODE)) { + printf( + "WARNING: code_block_mode was not specified in vector file and will be set to 1\n"); + vector->turbo_enc.code_block_mode = 1; + } + if (vector->turbo_enc.code_block_mode == 0) { + if (!(mask & TEST_BBDEV_VF_EA)) + printf( + "WARNING: ea was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_EB)) + printf( + "WARNING: eb was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_K_NEG)) + printf( + "WARNING: k_neg was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_K_POS)) + printf( + "WARNING: k_pos was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_C_NEG)) + printf( + "WARNING: c_neg was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_C)) { + printf( + "WARNING: c was not specified in vector file and will be set to 1\n"); + vector->turbo_enc.tb_params.c = 1; + } + if (!(mask & TEST_BBDEV_VF_CAB)) + printf( + "WARNING: cab was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_NCB_NEG)) + printf( + "WARNING: ncb_neg was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_NCB_POS)) + printf( + "WARNING: ncb_pos was not specified in vector file and will be set to 0\n"); + } else { + if (!(mask & TEST_BBDEV_VF_E)) + printf( + "WARNING: e was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_K)) + printf( + "WARNING: k was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_NCB)) + printf( + "WARNING: ncb was not specified in vector file and will be set to 0\n"); + } + if (!(mask & TEST_BBDEV_VF_RV_INDEX)) + printf( + "WARNING: rv_index was not specified in vector file and will be set to 0\n"); + if (!(mask & TEST_BBDEV_VF_OP_FLAGS)) + printf( + "WARNING: op_flags was not specified in vector file and capabilities will not be validated\n"); + if (!(mask & TEST_BBDEV_VF_EXPECTED_STATUS)) + printf( + "WARNING: expected_status was not specified in vector file and will be set to 0\n"); + + return 0; +} + +static int +bbdev_check_vector(struct test_bbdev_vector *vector) +{ + if (vector->op_type == RTE_BBDEV_OP_TURBO_DEC) { + if (check_decoder(vector, vector->mask) == -1) + return -1; + } else if (vector->op_type == RTE_BBDEV_OP_TURBO_ENC) { + if (check_encoder(vector, vector->mask) == -1) + return -1; + } else if (vector->op_type != RTE_BBDEV_OP_NONE) { + printf("Vector was not filled\n"); + return -1; + } + + return 0; +} + +int +test_bbdev_vector_read(const char *filename, + struct test_bbdev_vector *vector) +{ + int ret = 0; + size_t len = 0; + + FILE *fp = NULL; + char *line = NULL; + char *entry = NULL; + + fp = fopen(filename, "r"); + if (fp == NULL) { + printf("File %s does not exist\n", filename); + return -1; + } + + while (getline(&line, &len, fp) != -1) { + + /* ignore comments and new lines */ + if (line[0] == '#' || line[0] == '/' || line[0] == '\n' + || line[0] == '\r') + continue; + + trim_space(line); + + /* buffer for multiline */ + entry = realloc(entry, strlen(line) + 1); + if (entry == NULL) { + printf("Fail to realloc %zu bytes\n", strlen(line) + 1); + ret = -ENOMEM; + goto exit; + } + + memset(entry, 0, strlen(line) + 1); + strncpy(entry, line, strlen(line)); + + /* check if entry ends with , or = */ + if (entry[strlen(entry) - 1] == ',' + || entry[strlen(entry) - 1] == '=') { + while (getline(&line, &len, fp) != -1) { + trim_space(line); + + /* extend entry about length of new line */ + char *entry_extended = realloc(entry, + strlen(line) + + strlen(entry) + 1); + + if (entry_extended == NULL) { + printf("Fail to allocate %zu bytes\n", + strlen(line) + + strlen(entry) + 1); + ret = -ENOMEM; + goto exit; + } + + entry = entry_extended; + strncat(entry, line, strlen(line)); + + if (entry[strlen(entry) - 1] != ',') + break; + } + } + ret = parse_entry(entry, vector); + if (ret != 0) { + printf("An error occurred while parsing!\n"); + goto exit; + } + } + + ret = bbdev_check_vector(vector); + if (ret != 0) + printf("An error occurred while checking!\n"); + +exit: + fclose(fp); + free(line); + free(entry); + + return ret; +} diff --git a/app/test-bbdev/test_bbdev_vector.h b/app/test-bbdev/test_bbdev_vector.h new file mode 100644 index 0000000..75de330 --- /dev/null +++ b/app/test-bbdev/test_bbdev_vector.h @@ -0,0 +1,98 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef TEST_BBDEV_VECTOR_H_ +#define TEST_BBDEV_VECTOR_H_ + +#include <rte_bbdev_op.h> + +/* Flags which are set when specific parameter is define in vector file */ +enum { + TEST_BBDEV_VF_E = (1ULL << 0), + TEST_BBDEV_VF_EA = (1ULL << 1), + TEST_BBDEV_VF_EB = (1ULL << 2), + TEST_BBDEV_VF_K = (1ULL << 3), + TEST_BBDEV_VF_K_NEG = (1ULL << 4), + TEST_BBDEV_VF_K_POS = (1ULL << 5), + TEST_BBDEV_VF_C_NEG = (1ULL << 6), + TEST_BBDEV_VF_C = (1ULL << 7), + TEST_BBDEV_VF_CAB = (1ULL << 8), + TEST_BBDEV_VF_RV_INDEX = (1ULL << 9), + TEST_BBDEV_VF_ITER_MAX = (1ULL << 10), + TEST_BBDEV_VF_ITER_MIN = (1ULL << 11), + TEST_BBDEV_VF_EXPECTED_ITER_COUNT = (1ULL << 12), + TEST_BBDEV_VF_EXT_SCALE = (1ULL << 13), + TEST_BBDEV_VF_NUM_MAPS = (1ULL << 14), + TEST_BBDEV_VF_NCB = (1ULL << 15), + TEST_BBDEV_VF_NCB_NEG = (1ULL << 16), + TEST_BBDEV_VF_NCB_POS = (1ULL << 17), + TEST_BBDEV_VF_CODE_BLOCK_MODE = (1ULL << 18), + TEST_BBDEV_VF_OP_FLAGS = (1ULL << 19), + TEST_BBDEV_VF_EXPECTED_STATUS = (1ULL << 20), +}; + +enum op_data_type { + DATA_INPUT = 0, + DATA_SOFT_OUTPUT, + DATA_HARD_OUTPUT, + DATA_NUM_TYPES, +}; + +struct op_data_buf { + uint32_t *addr; + uint32_t length; +}; + +struct op_data_entries { + struct op_data_buf segments[RTE_BBDEV_MAX_CODE_BLOCKS]; + unsigned int nb_segments; +}; + +struct test_bbdev_vector { + enum rte_bbdev_op_type op_type; + int expected_status; + int mask; + union { + struct rte_bbdev_op_turbo_dec turbo_dec; + struct rte_bbdev_op_turbo_enc turbo_enc; + }; + /* Additional storage for op data entries */ + struct op_data_entries entries[DATA_NUM_TYPES]; +}; + +/* fills test vector paramaters based on test file */ +int +test_bbdev_vector_read(const char *filename, + struct test_bbdev_vector *vector); + + +#endif /* TEST_BBDEV_VECTOR_H_ */ diff --git a/app/test-bbdev/test_vectors/bbdev_vector_null.data b/app/test-bbdev/test_vectors/bbdev_vector_null.data new file mode 100644 index 0000000..91aea62 --- /dev/null +++ b/app/test-bbdev/test_vectors/bbdev_vector_null.data @@ -0,0 +1,32 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +op_type = +RTE_BBDEV_OP_NONE \ No newline at end of file diff --git a/app/test-bbdev/test_vectors/bbdev_vector_td_default.data b/app/test-bbdev/test_vectors/bbdev_vector_td_default.data new file mode 100644 index 0000000..db20640 --- /dev/null +++ b/app/test-bbdev/test_vectors/bbdev_vector_td_default.data @@ -0,0 +1,80 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +op_type = +RTE_BBDEV_OP_TURBO_DEC + +input0 = +0x7f007f00, 0x7f817f00, 0x767f8100, 0x817f8100, 0x81008100, 0x7f818100, 0x81817f00, 0x7f818100, +0x86007f00, 0x7f818100, 0x887f8100, 0x81815200, 0x81008100, 0x817f7f00, 0x7f7f8100, 0x9e817f00, +0x7f7f0000, 0xb97f0000, 0xa7810000, 0x7f7f4a7f, 0x7f810000, 0x7f7f7f7f, 0x81720000, 0x40658181, +0x84810000, 0x817f0000, 0x81810000, 0x7f818181, 0x7f810000, 0x81815a81, 0x817f0000, 0x7a867f7b, +0x817f0000, 0x6b7f0000, 0x7f810000, 0x81818181, 0x817f0000, 0x7f7f817f, 0x7f7f0000, 0xab7f4f7f, +0x817f0000, 0x817f6c00, 0x81810000, 0x817f8181, 0x7f810000, 0x81816981, 0x7f7f0000, 0x007f8181 + +hard_output0 = +0xa7d6732e, 0x61 + +soft_output0 = +0x7f7f7f7f, 0x81817f7f, 0x7f817f81, 0x817f7f81, 0x81817f81, 0x81817f81, 0x8181817f, 0x7f81817f, +0x7f81817f, 0x7f817f7f, 0x81817f7f, 0x817f8181, 0x81818181, 0x817f7f7f, 0x7f818181, 0x817f817f, +0x81818181, 0x81817f7f, 0x7f817f81, 0x7f81817f, 0x817f7f7f, 0x817f7f7f, 0x7f81817f, 0x817f817f, +0x81817f7f, 0x81817f7f, 0x81817f7f, 0x7f817f7f, 0x817f7f81, 0x7f7f8181, 0x81817f81, 0x817f7f7f, +0x7f7f8181 + +e = +17280 + +k = +40 + +rv_index = +1 + +iter_max = +8 + +iter_min = +4 + +expected_iter_count = +8 + +ext_scale = +15 + +num_maps = +0 + +op_flags = +RTE_BBDEV_TURBO_SOFT_OUTPUT, RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE + +expected_status = +OK diff --git a/app/test-bbdev/test_vectors/bbdev_vector_te_default.data b/app/test-bbdev/test_vectors/bbdev_vector_te_default.data new file mode 100644 index 0000000..b5cecf4 --- /dev/null +++ b/app/test-bbdev/test_vectors/bbdev_vector_te_default.data @@ -0,0 +1,60 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +op_type = +RTE_BBDEV_OP_TURBO_ENC + +input0 = +0x11d2bcac, 0x4d + +output0 = +0xd2399179, 0x640eb999, 0x2cbaf577, 0xaf224ae2, 0x9d139927, 0xe6909b29, 0xa25b7f47, 0x2aa224ce, +0x79f2 + +e = +272 + +k = +40 + +ncb = +192 + +rv_index = +0 + +code_block_mode = +1 + +op_flags = +RTE_BBDEV_TURBO_RATE_MATCH + +expected_status = +OK -- 2.7.4 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [dpdk-dev] [PATCH v2 4/5] bbdev: sample app 2017-10-18 2:14 [dpdk-dev] [PATCH v2 1/5] bbdev: librte_bbdev library Amr Mokhtar 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 2/5] bbdev: PMD drivers (null/turbo_sw) Amr Mokhtar 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 3/5] bbdev: test applications Amr Mokhtar @ 2017-10-18 2:14 ` Amr Mokhtar 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 5/5] bbdev: documentation Amr Mokhtar 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 0/5] Wireless Base Band Device (bbdev) Amr Mokhtar 4 siblings, 0 replies; 6+ messages in thread From: Amr Mokhtar @ 2017-10-18 2:14 UTC (permalink / raw) To: dev Cc: thomas, anatoly.burakov, pablo.de.lara.guarch, niall.power, chris.macnamara, Amr Mokhtar - Sample application performing a loop-back over ethernet using a bbbdev device - A packet is received on an Rx ethdev port -> enqueued for baseband operation -> dequeued -> looped-back to a Tx ethdev port - 'Turbo_sw' PMD must be enabled for the app to be functional Signed-off-by: Amr Mokhtar <amr.mokhtar@intel.com> --- examples/Makefile | 1 + examples/bbdev_app/Makefile | 50 ++ examples/bbdev_app/main.c | 1396 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 1447 insertions(+) create mode 100644 examples/bbdev_app/Makefile create mode 100644 examples/bbdev_app/main.c diff --git a/examples/Makefile b/examples/Makefile index d27eddd..535f36f 100644 --- a/examples/Makefile +++ b/examples/Makefile @@ -59,6 +59,7 @@ ifneq ($(PQOS_INSTALL_PATH),) DIRS-y += l2fwd-cat endif DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += l2fwd-crypto +DIRS-$(CONFIG_RTE_LIBRTE_BBDEV) += bbdev_app DIRS-$(CONFIG_RTE_LIBRTE_JOBSTATS) += l2fwd-jobstats DIRS-y += l2fwd-keepalive DIRS-y += l2fwd-keepalive/ka-agent diff --git a/examples/bbdev_app/Makefile b/examples/bbdev_app/Makefile new file mode 100644 index 0000000..1583251 --- /dev/null +++ b/examples/bbdev_app/Makefile @@ -0,0 +1,50 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +ifeq ($(RTE_SDK),) +$(error "Please define RTE_SDK environment variable") +endif + +# Default target, can be overridden by command line or environment +RTE_TARGET ?= x86_64-native-linuxapp-gcc + +include $(RTE_SDK)/mk/rte.vars.mk + +# binary name +APP = bbdev + +# all source are stored in SRCS-y +SRCS-y := main.c + +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) + +include $(RTE_SDK)/mk/rte.extapp.mk diff --git a/examples/bbdev_app/main.c b/examples/bbdev_app/main.c new file mode 100644 index 0000000..55bf23f --- /dev/null +++ b/examples/bbdev_app/main.c @@ -0,0 +1,1396 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <stdint.h> +#include <inttypes.h> +#include <sys/types.h> +#include <sys/unistd.h> +#include <sys/queue.h> +#include <stdarg.h> +#include <ctype.h> +#include <errno.h> +#include <math.h> +#include <assert.h> +#include <getopt.h> +#include <signal.h> +#include <inttypes.h> + +#include "rte_common.h" +#include "rte_eal.h" +#include "rte_cycles.h" +#include "rte_eal.h" +#include "rte_ether.h" +#include "rte_ethdev.h" +#include "rte_ip.h" +#include "rte_lcore.h" +#include "rte_malloc.h" +#include "rte_mbuf.h" +#include "rte_memory.h" +#include "rte_mempool.h" +#include "rte_log.h" +#include "rte_bbdev.h" +#include "rte_bbdev_op.h" + +#define MAX_PKT_BURST 32 +#define NB_MBUF 8163 +#define MAX_RX_QUEUE_PER_LCORE 16 +#define MAX_TX_QUEUE_PER_LCORE 16 +#define MEMPOOL_CACHE_SIZE 256 + +/*Configurable number of RX/TX ring descriptors */ +#define RTE_TEST_RX_DESC_DEFAULT 128 +#define RTE_TEST_TX_DESC_DEFAULT 512 + +#define BBDEV_ASSERT(a) do { \ + if (!(a)) { \ + usage(prgname); \ + return -1; \ + } \ +} while (0) + +static struct rte_mempool *ethdev_mbuf_mempool; + +struct dev_qs { + unsigned int port_id; + unsigned int queues_list[RTE_MAX_QUEUES_PER_PORT]; + unsigned int queues; +}; + +static struct rte_mempool *bbdev_op_pool[RTE_BBDEV_OP_TYPE_COUNT]; + +/**all lcores used*/ +struct lcore_setup { + unsigned int rxte_lcores; + unsigned int rxte_lcore_list[RTE_MAX_LCORE]; + unsigned int tetx_lcores; + unsigned int tetx_lcore_list[RTE_MAX_LCORE]; + unsigned int rxtd_lcores; + unsigned int rxtd_lcore_list[RTE_MAX_LCORE]; + unsigned int tdtx_lcores; + unsigned int tdtx_lcore_list[RTE_MAX_LCORE]; +}; + +/** each lcore configuration */ +struct lcore_queue_conf { + unsigned int nb_ports; + struct dev_qs port_list[RTE_MAX_ETHPORTS]; + /* ethernet addresses of ports */ + struct ether_addr ports_eth_addr[RTE_MAX_ETHPORTS]; + + unsigned int bbdev_id; + + unsigned int bbdev_qs[128]; + unsigned int nb_bbdev_qs; + + struct rte_mempool *mbuf_pool; +} __rte_cache_aligned; + +static struct lcore_queue_conf lcore_queue_conf[RTE_MAX_LCORE]; + +static const struct rte_eth_conf port_conf = { + .rxmode = { + .mq_mode = ETH_MQ_RX_NONE, + .max_rx_pkt_len = ETHER_MAX_LEN, + .split_hdr_size = 0, + .header_split = 0, /**< Header Split disabled */ + .hw_ip_checksum = 0, /**< IP checksum offload disabled */ + .hw_vlan_filter = 0, /**< VLAN filtering disabled */ + .jumbo_frame = 0, /**< Jumbo Frame Support disabled */ + .hw_strip_crc = 0, /**< CRC stripped by hardware */ + }, + .txmode = { + .mq_mode = ETH_MQ_TX_NONE, + }, +}; + +struct rte_bbdev_op_turbo_enc def_op_enc = { + /* These values are arbitrarily put, and does not map to the real + * values for the data received from ethdev ports + */ + .rv_index = 0, + .code_block_mode = 1, + .cb_params.e = 272, + .cb_params.ncb = 192, + .op_flags = RTE_BBDEV_TURBO_CRC_24B_ATTACH +}; + +struct rte_bbdev_op_turbo_dec def_op_dec = { + /* These values are arbitrarily put, and does not map to the real + * values for the data received from ethdev ports + */ + .cb_params.e = 44, + .rv_index = 0, + .iter_max = 8, + .iter_min = 4, + .ext_scale = 15, + .num_maps = 0, + .op_flags = RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE | + RTE_BBDEV_TURBO_EQUALIZER | RTE_BBDEV_TURBO_SOFT_OUTPUT +}; + +struct bbdev_config_params { + uint8_t downlink_lcores; + uint8_t uplink_lcores; + + uint8_t downlink_rx_ports; + uint8_t downlink_tx_ports; + + uint8_t uplink_rx_ports; + uint8_t uplink_tx_ports; + + unsigned int rx_port_list[MAX_RX_QUEUE_PER_LCORE]; + unsigned int tx_port_list[MAX_TX_QUEUE_PER_LCORE]; +}; + +struct lcore_statistics { + unsigned int enqueued; + unsigned int dequeued; + unsigned int lost_packets; +} __rte_cache_aligned; + +static struct lcore_statistics lcore_stats[RTE_MAX_LCORE]; + +static volatile int global_exit_flag; + +struct port_queues { + unsigned int nb_rx_qs; + unsigned int nb_tx_qs; +}; + +static struct port_queues ports_qs[RTE_MAX_ETHPORTS]; + +/* display usage */ +static inline void +usage(const char *prgname) +{ + printf("%s [EAL options] " + " --\n" + " --downlink_cores - downlink cores mask\n" + " --uplink_cores - uplink cores mask\n" + " --downlink_rx_ports - downlink Rx ports mask\n" + " --downlink_tx_ports - downlink Tx ports mask\n" + " --uplink_tx_ports -uplink Tx ports mask\n" + " --uplink_rx_ports -uplink Rx ports msk\n" + "\n", prgname); +} + +/* parse port or core mask */ +static inline +uint8_t bbdev_parse_mask(const char *mask) +{ + char *end = NULL; + unsigned long pm; + + /* parse hexadecimal string */ + pm = strtoul(mask, &end, 16); + if ((mask[0] == '\0') || (end == NULL) || (*end != '\0')) + return 0; + + if (pm == 0) + return 0; + + return pm; +} + +static int +bbdev_parse_args(int argc, char **argv, + struct bbdev_config_params *bbdev_params) +{ + int optind = 0; + int opt; + int opt_indx = 0; + char *prgname = argv[0]; + + memset(bbdev_params, 0, sizeof(*bbdev_params)); + + static struct option lgopts[] = { + { "downlink_cores", required_argument, 0, 'c' }, + { "uplink_cores", required_argument, 0, 'C' }, + { "downlink_rx_ports", required_argument, 0, 'r' }, + { "downlink_tx_ports", required_argument, 0, 't' }, + { "uplink_rx_ports", required_argument, 0, 'R' }, + { "uplink_tx_ports", required_argument, 0, 'T' }, + { NULL, 0, 0, 0 } + }; + + BBDEV_ASSERT(argc != 0); + BBDEV_ASSERT(argv != NULL); + BBDEV_ASSERT(bbdev_params != NULL); + + while ((opt = getopt_long(argc, argv, "c:C:r:t:R:T:", lgopts, + &opt_indx)) != EOF) { + + switch (opt) { + case 'c': + bbdev_params->downlink_lcores = + bbdev_parse_mask(optarg); + if (bbdev_params->downlink_lcores == 0) { + usage(prgname); + return -1; + } + break; + + case 'C': + bbdev_params->uplink_lcores = + bbdev_parse_mask(optarg); + if (bbdev_params->uplink_lcores == 0) { + usage(prgname); + return -1; + } + break; + + case 'r': + bbdev_params->downlink_rx_ports = + bbdev_parse_mask(optarg); + if (bbdev_params->downlink_rx_ports == 0) { + usage(prgname); + return -1; + } + break; + + case 't': + bbdev_params->downlink_tx_ports = + bbdev_parse_mask(optarg); + if (bbdev_params->downlink_tx_ports == 0) { + usage(prgname); + return -1; + } + break; + + case 'R': + bbdev_params->uplink_rx_ports = + bbdev_parse_mask(optarg); + if (bbdev_params->uplink_rx_ports == 0) { + usage(prgname); + return -1; + } + break; + + case 'T': + bbdev_params->uplink_tx_ports = + bbdev_parse_mask(optarg); + if (bbdev_params->uplink_tx_ports == 0) { + usage(prgname); + return -1; + } + break; + + default: + usage(prgname); + return -1; + } + } + optind = 0; + return optind; +} + +static void +signal_handler(int signum) +{ + printf("\nSignal %d received", signum); + global_exit_flag = 1; +} + +static void +print_mac(unsigned int portid, struct ether_addr *bbdev_ports_eth_address) +{ + printf("Port %u, MAC address: %02X:%02X:%02X:%02X:%02X:%02X\n\n", + (unsigned int) portid, + bbdev_ports_eth_address[portid].addr_bytes[0], + bbdev_ports_eth_address[portid].addr_bytes[1], + bbdev_ports_eth_address[portid].addr_bytes[2], + bbdev_ports_eth_address[portid].addr_bytes[3], + bbdev_ports_eth_address[portid].addr_bytes[4], + bbdev_ports_eth_address[portid].addr_bytes[5]); +} + +/* Check the link status of all ports in up to 9s, and print them finally */ +static void +check_all_ports_link_status(uint8_t port_num, uint32_t port_mask) +{ +#define CHECK_INTERVAL 100 /* 100ms */ +#define MAX_CHECK_TIME 90 /* 9s (90 * 100ms) in total */ + uint8_t portid, count, all_ports_up, print_flag = 0; + struct rte_eth_link link; + + printf("\nChecking link status"); + fflush(stdout); + + for (count = 0; count <= MAX_CHECK_TIME; count++) { + all_ports_up = 1; + for (portid = 0; portid < port_num; portid++) { + if ((port_mask & (1 << portid)) == 0) + continue; + memset(&link, 0, sizeof(link)); + rte_eth_link_get_nowait(portid, &link); + /* print link status if flag set */ + if (print_flag == 1) { + if (link.link_status) { + const char *dp = (link.link_duplex == + ETH_LINK_FULL_DUPLEX) ? + "full-duplex" : + "half-duplex"; + printf("Port %u Link Up - speed %u Mbps" + " - %s\n", portid, + link.link_speed, + dp); + } else + printf("Port %d Link Down\n", portid); + continue; + } + /* clear all_ports_up flag if any link down */ + if (link.link_status == 0) { + all_ports_up = 0; + break; + } + } + /* after finally printing all link status, get out */ + if (print_flag == 1) + break; + + if (all_ports_up == 0) { + printf("."); + fflush(stdout); + rte_delay_ms(CHECK_INTERVAL); + } + + /* set the print_flag if all ports up or timeout */ + if (all_ports_up == 1 || count == (MAX_CHECK_TIME - 1)) { + print_flag = 1; + printf("done\n"); + } + } +} + +static void +update_pkt_mac(uint16_t n, struct rte_mbuf *pkts_burst[], + struct ether_addr *addr, unsigned int dest_portid) +{ + struct ether_hdr *eth; + void *tmp; + + for (uint16_t i = 0; i < n; i++) { + eth = rte_pktmbuf_mtod(pkts_burst[i], struct ether_hdr *); + /* 02:00:00:00:00:xx */ + tmp = ð->d_addr.addr_bytes[0]; + *((uint64_t *)tmp) = 0x000000000002 + + ((uint64_t)dest_portid << 40); + /* src addr */ + ether_addr_copy(addr, ð->s_addr); + } +} + +static int +initialize_ports(uint8_t ports_mask, char link, char port_type, + uint32_t *bbdev_enabled_port_mask) +{ + uint8_t portid; + unsigned int enabled_portcount = 0, q; + int ret; + struct port_queues *port_qs; + /* ethernet addresses of ports */ + struct ether_addr bbdev_dl_ports_eth_addr[RTE_MAX_ETHPORTS]; + struct ether_addr bbdev_ul_ports_eth_addr[RTE_MAX_ETHPORTS]; + + uint8_t nb_ports = rte_eth_dev_count(); + if (nb_ports == 0) { + printf("No Ethernet ports available\n"); + return -1; + } + + for (portid = 0; portid < nb_ports; portid++) { + + /* Skip ports that are not enabled */ + if ((ports_mask & (1 << portid)) == 0) + continue; + + port_qs = &ports_qs[portid]; + /* initialize ports */ + printf("\nInitializing %cx port %u... ", + port_type, portid); + fflush(stdout); + ret = rte_eth_dev_configure(portid, port_qs->nb_rx_qs, + port_qs->nb_tx_qs, &port_conf); + + if (ret < 0) { + printf("Cannot configure device: err=%d, port=%u\n", + ret, portid); + return -1; + } + fflush(stdout); + + /* initialize one RX or TX queue on each port*/ + if (port_type == 'R') + for (q = 0; q < port_qs->nb_rx_qs; q++) + ret = rte_eth_rx_queue_setup(portid, q, + RTE_TEST_RX_DESC_DEFAULT, + rte_eth_dev_socket_id(portid), + NULL, ethdev_mbuf_mempool); + else if (port_type == 'T') + for (q = 0; q < port_qs->nb_rx_qs; q++) + ret = rte_eth_tx_queue_setup(portid, q, + RTE_TEST_TX_DESC_DEFAULT, + rte_eth_dev_socket_id(portid), + NULL); + + if (ret < 0) { + printf("%cL rte_eth_%cx_queue_setup:err=%d, port=%u\n", + port_type, link, ret, + (unsigned int) portid); + return -1; + } + + rte_eth_promiscuous_enable(portid); + + if (link == 'D') { + rte_eth_macaddr_get(portid, + &bbdev_dl_ports_eth_addr[portid]); + print_mac(portid, bbdev_dl_ports_eth_addr); + } else { + rte_eth_macaddr_get(portid, + &bbdev_ul_ports_eth_addr[portid]); + print_mac(portid, bbdev_ul_ports_eth_addr); + } + + *bbdev_enabled_port_mask |= (1 << portid); + enabled_portcount++; + } + return enabled_portcount; +} + +static int +start_ethdev_ports(uint8_t nb_ports, uint32_t bbdev_enabled_port_mask) +{ + unsigned int portid; + int ret, ports_started = 0; + + for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) { + + if ((bbdev_enabled_port_mask & (uint32_t)(1 << portid)) == 0) + continue; + ret = rte_eth_dev_start(portid); + if (ret < 0) { + printf("rte_eth_dev_start:err=%d, port=%u\n", ret, + (unsigned int) portid); + return -1; + } + + ports_started++; + if (ports_started == nb_ports) + break; + } + return ports_started; +} + +static int +lcore_queue_init(uint8_t nb_ports, struct bbdev_config_params *bbdev_params, + struct lcore_setup *l_setup) +{ + uint16_t port; + unsigned int lcore; + int lcount, total_count = 0; + struct lcore_queue_conf *qconf; + + /* + * for each core in the uplink populate the array of ports (rx or tx). + * core with odd id receives rx ports, core with even id receives all + * tx ports. + */ + for (lcore = 0, lcount = 0; lcore < sizeof(int) * 8; lcore++) { + + /* get the lcore_id */ + if ((bbdev_params->downlink_lcores & (1 << lcore)) == 0) + continue; + + lcount++; + for (port = 0; port <= nb_ports; port++) { + if ((lcount & 1) == 1) { + if (port < nb_ports) { + /*get the port id*/ + if ((bbdev_params->downlink_rx_ports & + (1 << port)) == 0) + continue; + + l_setup->rxte_lcore_list[ + l_setup->rxte_lcores] = + lcore; + printf("Lcore %d: DL RX port %d\n", + lcore, port); + } else { + l_setup->rxte_lcores++; + continue; + } + + } else { + if (port < nb_ports) { + if ((bbdev_params->downlink_tx_ports & + (1 << port)) == 0) + continue; + + l_setup->tdtx_lcore_list[ + l_setup->tdtx_lcores] = + lcore; + printf("Lcore %d: DL TX port %d\n", + lcore, port); + } else { + l_setup->tdtx_lcores++; + continue; + } + } + qconf = &lcore_queue_conf[lcore]; + qconf->port_list[qconf->nb_ports].queues++; + qconf->port_list[qconf->nb_ports].port_id = port; + rte_eth_macaddr_get(port, + &qconf->ports_eth_addr + [qconf->nb_ports]); + qconf->nb_ports++; + } + } + + if (lcount % 2) { + printf("\nNumber of DL lcores is not an even number"); + return -1; + } + + total_count += lcount; + + /* + * for each core in the downlink populate the array of ports (rx or tx). + * core with odd id receives rx ports, core with even id receives all + * tx ports + */ + for (lcore = 0, lcount = 0; lcore < (sizeof(int) * 8); lcore++) { + if ((bbdev_params->uplink_lcores & (1 << lcore)) == 0) + continue; + + lcount++; + for (port = 0; port <= nb_ports; port++) { + + if ((lcount & 1) == 1) { + if (port < nb_ports) { + if ((bbdev_params->uplink_rx_ports & + (1 << port)) == 0) + continue; + + l_setup->rxtd_lcore_list[ + l_setup->rxtd_lcores] = + lcore; + printf("Lcore %u: UL RX port %u\n", + lcore, port); + } else { + l_setup->rxtd_lcores++; + continue; + } + + } else { + if (port < nb_ports) { + if ((bbdev_params->uplink_tx_ports & + (1 << port)) == 0) + continue; + + l_setup->tetx_lcore_list[ + l_setup->tetx_lcores] = + lcore; + printf("Lcore %d: UL TX port %d\n", + lcore, port); + } else { + l_setup->tetx_lcores++; + continue; + } + } + qconf = &lcore_queue_conf[lcore]; + qconf->port_list[qconf->nb_ports].queues++; + qconf->port_list[qconf->nb_ports].port_id = port; + rte_eth_macaddr_get(port, + &qconf->ports_eth_addr + [qconf->nb_ports]); + qconf->nb_ports++; + } + } + if (lcount % 2) { + printf("\nNumber of DL lcores is not an even number"); + return -1; + } + total_count += lcount; + /*return number of lcores processed*/ + return total_count; +} + +static void +bbdev_dequeue_dec(void) +{ + struct rte_mbuf *pkts_burst[MAX_PKT_BURST]; + struct rte_bbdev_dec_op *ops_burst[MAX_PKT_BURST]; + unsigned int i, j, portid, nb_tx, lcoreid, qid = 0, q, nb_tx_sent = 0; + struct lcore_queue_conf *qconf; + struct lcore_statistics *lcore_stat; + + lcoreid = rte_lcore_id(); + lcore_stat = &lcore_stats[lcoreid]; + qconf = &lcore_queue_conf[lcoreid]; + + if (qconf->nb_ports == 0) + return; + + while (!global_exit_flag) { + for (i = 0; i < qconf->nb_ports; i++) { + portid = qconf->port_list[i].port_id; + + for (q = 0; q < qconf->port_list[i].queues; q++) { + qid = qconf->port_list[i].queues_list[q]; + /* Dequeue packets from bbdev device*/ + nb_tx = rte_bbdev_dequeue_dec_ops( + qconf->bbdev_id, + qconf->bbdev_qs[q], ops_burst, + MAX_PKT_BURST); + + if (!nb_tx) + continue; + + lcore_stat->dequeued += nb_tx; + + for (j = 0; j < nb_tx; j++) { + pkts_burst[j] = + ops_burst[j]->turbo_dec + .hard_output.data; + /* input mbufs are no longer needed, + * release! + */ + rte_pktmbuf_free(ops_burst[j]-> + turbo_dec.input.data); + } + + rte_bbdev_dec_op_free_bulk(ops_burst, nb_tx); + + update_pkt_mac(nb_tx, pkts_burst, + &(qconf->ports_eth_addr + [portid]), + portid); + + /*Enqueue packets to ethdev*/ + nb_tx_sent = rte_eth_tx_burst( + (uint8_t)portid, qid, + pkts_burst, nb_tx); + if (unlikely(nb_tx_sent < nb_tx)) { + lcore_stat->lost_packets += + nb_tx - nb_tx_sent; + for (j = nb_tx_sent; j < nb_tx; j++) + rte_pktmbuf_free(pkts_burst[j]); + } + } + } + } +} + +static void +bbdev_dequeue_enc(void) +{ + struct rte_mbuf *pkts_burst[MAX_PKT_BURST]; + struct rte_bbdev_enc_op *ops_burst[MAX_PKT_BURST]; + unsigned int i, j, portid, nb_tx, lcoreid, qid = 0, q, nb_tx_sent = 0; + struct lcore_queue_conf *qconf; + struct lcore_statistics *lcore_stat; + + lcoreid = rte_lcore_id(); + lcore_stat = &lcore_stats[lcoreid]; + qconf = &lcore_queue_conf[lcoreid]; + + if (qconf->nb_ports == 0) + return; + + while (!global_exit_flag) { + for (i = 0; i < qconf->nb_ports; i++) { + portid = qconf->port_list[i].port_id; + + for (q = 0; q < qconf->port_list[i].queues; q++) { + qid = qconf->port_list[i].queues_list[q]; + /* Dequeue packets from bbdev device*/ + nb_tx = rte_bbdev_dequeue_enc_ops( + qconf->bbdev_id, + qconf->bbdev_qs[q], ops_burst, + MAX_PKT_BURST); + + if (!nb_tx) + continue; + + lcore_stat->dequeued += nb_tx; + + for (j = 0; j < nb_tx; j++) { + pkts_burst[j] = + ops_burst[j]-> + turbo_enc.output.data; + /* input mbufs are no longer needed, + * release! + */ + rte_pktmbuf_free(ops_burst[j]-> + turbo_enc.input.data); + } + + rte_bbdev_enc_op_free_bulk(ops_burst, nb_tx); + + update_pkt_mac(nb_tx, pkts_burst, + &(qconf->ports_eth_addr + [portid]), portid); + + /*Enqueue packets to ethdev*/ + nb_tx_sent = rte_eth_tx_burst( + (uint8_t)portid, qid, + pkts_burst, nb_tx); + if (unlikely(nb_tx_sent < nb_tx)) { + lcore_stat->lost_packets += + nb_tx - nb_tx_sent; + for (j = nb_tx_sent; j < nb_tx; j++) + rte_pktmbuf_free(pkts_burst[j]); + } + } + } + } +} + +static void +bbdev_enqueue_dec(void) +{ + unsigned int i, portid, nb_rx = 0, j, lcoreid, q, qid = 0; + int ret; + struct rte_mbuf *pkts_burst[MAX_PKT_BURST]; + struct rte_mbuf *bbdev_pkts[MAX_PKT_BURST]; + struct rte_bbdev_dec_op *ops_burst[MAX_PKT_BURST]; + struct lcore_statistics *lcore_stat; + struct lcore_queue_conf *qconf = &lcore_queue_conf[rte_lcore_id()]; + + if (qconf->nb_ports == 0) + return; + + lcoreid = rte_lcore_id(); + lcore_stat = &lcore_stats[lcoreid]; + + /* Read packet from RX queues*/ + while (!global_exit_flag) { + + for (i = 0; i < qconf->nb_ports; i++) { + portid = qconf->port_list[i].port_id; + + for (q = 0; q < qconf->port_list[i].queues; q++) { + + qid = qconf->port_list[i].queues_list[q]; + /* Dequeue packets from ethdev */ + nb_rx = rte_eth_rx_burst((uint8_t) portid, qid, + pkts_burst, MAX_PKT_BURST); + + if (!nb_rx) + continue; + + if (rte_bbdev_dec_op_alloc_bulk( + bbdev_op_pool + [RTE_BBDEV_OP_TURBO_DEC], + ops_burst, nb_rx) != 0) + continue; + + ret = rte_pktmbuf_alloc_bulk(qconf->mbuf_pool, + bbdev_pkts, MAX_PKT_BURST); + if (ret < 0) + continue; + + for (j = 0; j < nb_rx; j++) { + struct rte_bbdev_op_turbo_dec *td; + /* append the size of the etherneit + * header + */ + rte_pktmbuf_append(bbdev_pkts[j], + sizeof(struct ether_hdr)); + /* copy the ethernet header */ + void *in_buf = rte_pktmbuf_mtod( + pkts_burst[j], void *); + void *out_buf = rte_pktmbuf_mtod( + bbdev_pkts[j], void *); + rte_memcpy(out_buf, in_buf, + sizeof(struct ether_hdr)); + /* set op */ + ops_burst[j]->turbo_dec = def_op_dec; + td = &ops_burst[j]->turbo_dec; + td->cb_params.k = + rte_pktmbuf_pkt_len( + bbdev_pkts[j]) + * 8; + td->input.offset = + sizeof( + struct + ether_hdr); + td->input.length = + rte_pktmbuf_pkt_len( + bbdev_pkts[j]); + td->input.data = + pkts_burst[j]; + td->hard_output.offset = + sizeof( + struct + ether_hdr); + td->hard_output.data = + bbdev_pkts[j]; + } + + /* Enqueue packets on BBDEV device */ + nb_rx = rte_bbdev_enqueue_dec_ops( + qconf->bbdev_id, + qconf->bbdev_qs[q], ops_burst, + nb_rx); + lcore_stat->enqueued += nb_rx; + } + } + } +} + +static void +bbdev_enqueue_enc(void) +{ + unsigned int i, portid, nb_rx = 0, j, lcoreid, q, qid = 0; + int ret; + struct rte_mbuf *pkts_burst[MAX_PKT_BURST]; + struct rte_mbuf *bbdev_pkts[MAX_PKT_BURST]; + struct rte_bbdev_enc_op *ops_burst[MAX_PKT_BURST]; + struct lcore_statistics *lcore_stat; + struct lcore_queue_conf *qconf = &lcore_queue_conf[rte_lcore_id()]; + + lcoreid = rte_lcore_id(); + lcore_stat = &lcore_stats[lcoreid]; + + /* Read packet from RX queues*/ + while (!global_exit_flag) { + + for (i = 0; i < qconf->nb_ports; i++) { + portid = qconf->port_list[i].port_id; + + for (q = 0; q < qconf->port_list[i].queues; q++) { + + qid = qconf->port_list[i].queues_list[q]; + /* Dequeue packets from ethdev */ + nb_rx = rte_eth_rx_burst((uint8_t)portid, qid, + pkts_burst, MAX_PKT_BURST); + + if (!nb_rx) + continue; + + if (rte_bbdev_enc_op_alloc_bulk( + bbdev_op_pool + [RTE_BBDEV_OP_TURBO_ENC], + ops_burst, nb_rx) != 0) + continue; + + ret = rte_pktmbuf_alloc_bulk(qconf->mbuf_pool, + bbdev_pkts, MAX_PKT_BURST); + if (ret < 0) + continue; + + for (j = 0; j < nb_rx; j++) { + /* append the size of the ethernet + * header + */ + rte_pktmbuf_append(bbdev_pkts[j], + sizeof( + struct + ether_hdr)); + /* copy the ethernet header */ + void *in_buf = rte_pktmbuf_mtod( + pkts_burst[j], void *); + void *out_buf = rte_pktmbuf_mtod( + bbdev_pkts[j], void *); + rte_memcpy(out_buf, in_buf, + sizeof( + struct + ether_hdr)); + /* set op */ + ops_burst[j]->turbo_enc = def_op_enc; + ops_burst[j]->turbo_enc.cb_params.k = + rte_pktmbuf_pkt_len( + bbdev_pkts[j]) + * 8; + ops_burst[j]->turbo_enc.input.offset = + sizeof( + struct + ether_hdr); + ops_burst[j]->turbo_enc.input.length = + rte_pktmbuf_pkt_len( + bbdev_pkts[j]); + ops_burst[j]->turbo_enc.input.data = + pkts_burst[j]; + ops_burst[j]->turbo_enc.output.offset = + sizeof( + struct + ether_hdr); + ops_burst[j]->turbo_enc.output.data = + bbdev_pkts[j]; + } + + /* Enqueue packets on BBDEV device */ + nb_rx = rte_bbdev_enqueue_enc_ops( + qconf->bbdev_id, + qconf->bbdev_qs[q], ops_burst, + nb_rx); + lcore_stat->enqueued += nb_rx; + } + } + } +} + +static void +print_lcore_stats(unsigned int lcore_id) +{ + struct lcore_statistics *lcore_stat; + static const char *stats_border = "_______"; + + lcore_stat = &lcore_stats[lcore_id]; + printf("\nLcore %d: %s enqueued count:\t\t%u\n", + lcore_id, stats_border, lcore_stat->enqueued); + printf("Lcore %d: %s dequeued count:\t\t%u\n", + lcore_id, stats_border, lcore_stat->dequeued); +} + +static void +print_stats(struct lcore_setup *lcore_setup) +{ + unsigned int l_id, dev_id; + unsigned char x, nb_ports; + int len, ret, i; + + struct rte_eth_xstat *xstats; + struct rte_eth_xstat_name *xstats_names; + struct rte_bbdev_stats bbstats; + static const char *stats_border = "_______"; + + const char clr[] = { 27, '[', '2', 'J', '\0' }; + const char topLeft[] = { 27, '[', '1', ';', '1', 'H', '\0' }; + + /* Clear screen and move to top left */ + printf("%s%s", clr, topLeft); + + nb_ports = rte_eth_dev_count(); + printf("PORT STATISTICS:\n================\n"); + for (x = 0; x < nb_ports; x++) { + + len = rte_eth_xstats_get(x, NULL, 0); + if (len < 0) + rte_exit(EXIT_FAILURE, + "rte_eth_xstats_get(%hhu) failed: %d", + x, len); + + xstats = calloc(len, sizeof(*xstats)); + if (xstats == NULL) + rte_exit(EXIT_FAILURE, + "Failed to calloc memory for xstats"); + + ret = rte_eth_xstats_get(x, xstats, len); + if (ret < 0 || ret > len) + rte_exit(EXIT_FAILURE, + "rte_eth_xstats_get(%hhu) len%i failed: %d", + x, len, ret); + + xstats_names = calloc(len, sizeof(*xstats_names)); + if (xstats_names == NULL) + rte_exit(EXIT_FAILURE, + "Failed to calloc memory for xstats_names"); + + ret = rte_eth_xstats_get_names(x, xstats_names, len); + if (ret < 0 || ret > len) + rte_exit(EXIT_FAILURE, + "rte_eth_xstats_get_names(%hhu) len%i failed: %d", + x, len, ret); + + for (i = 0; i < len; i++) { + if (xstats[i].value > 0) + printf("Port %u: %s %s:\t\t%"PRIu64"\n", + x, stats_border, + xstats_names[i].name, + xstats[i].value); + } + } + + printf("\nBBDEV STATISTICS:\n=================\n"); + RTE_BBDEV_FOREACH(dev_id) { + rte_bbdev_stats_get(dev_id, &bbstats); + printf("BBDEV %u: %s enqueue count:\t\t%"PRIu64"\n", + dev_id, stats_border, + bbstats.enqueued_count); + printf("BBDEV %u: %s dequeue count:\t\t%"PRIu64"\n", + dev_id, stats_border, + bbstats.dequeued_count); + printf("BBDEV %u: %s enqueue error count:\t\t%"PRIu64"\n", + dev_id, stats_border, + bbstats.enqueue_err_count); + printf("BBDEV %u: %s dequeue error count:\t\t%"PRIu64"\n\n", + dev_id, stats_border, + bbstats.dequeue_err_count); + } + + printf("LCORE STATISTICS:\n=================\n"); + for (l_id = 0; l_id < lcore_setup->tdtx_lcores; l_id++) + print_lcore_stats(lcore_setup->tdtx_lcore_list[l_id]); + for (l_id = 0; l_id < lcore_setup->tetx_lcores; l_id++) + print_lcore_stats(lcore_setup->tetx_lcore_list[l_id]); + for (l_id = 0; l_id < lcore_setup->rxtd_lcores; l_id++) + print_lcore_stats(lcore_setup->rxtd_lcore_list[l_id]); + for (l_id = 0; l_id < lcore_setup->rxte_lcores; l_id++) + print_lcore_stats(lcore_setup->rxte_lcore_list[l_id]); +} + +static void +qs_to_lcores(unsigned int nb_lcores, unsigned int lcores[]) +{ + unsigned int port = 0, lcore, nb_ports = 0, q_id; + unsigned int l = 0, y, x; + struct lcore_queue_conf *qconf; + struct dev_qs temp[RTE_MAX_LCORE]; + int count = 0; + struct dev_qs *p; + struct port_queues *port_qs; + + /* + * for each core copy array of ports (holding all ports available to + * this lcore) from qconf into temp array and empty the one in qconf + */ + for (x = 0; x < nb_lcores; x++) { + + lcore = lcores[x]; + qconf = &lcore_queue_conf[lcore]; + nb_ports = qconf->nb_ports; + qconf->nb_ports = 0; + p = qconf->port_list; + + for (y = 0; y < nb_ports; y++, p++) { + temp[y] = *p; + p->port_id = 0; + } + } + /* + * re-populate array of ports and queues in qconf according to number + * of available lcores, ports, and queues so only the required ports + * and queues stay in the configuration of each particular lcore + */ + while (port < nb_ports || l < nb_lcores) { + + if (port < nb_ports && l == nb_lcores) { + l = 0; + count++; + } + if (port == nb_ports && l < nb_lcores) + port = 0; + + lcore = lcores[l]; + qconf = &lcore_queue_conf[lcore]; + port_qs = &ports_qs[temp[port].port_id]; + + if ((lcore & 1) == 1) { + qconf->port_list[count].port_id = + temp[port].port_id; + + for (q_id = 0; q_id < qconf->port_list[count].queues; + q_id++) { + qconf->port_list[count].queues_list[q_id] = + port_qs->nb_tx_qs; + port_qs->nb_tx_qs++; + } + + qconf->nb_ports++; + } else { + qconf->port_list[count].port_id = + temp[port].port_id; + for (q_id = 0; q_id < qconf->port_list[count].queues; + q_id++) { + qconf->port_list[count].queues_list[q_id] = + port_qs->nb_rx_qs; + port_qs->nb_rx_qs++; + } + qconf->nb_ports++; + } + printf("Lcore: %u connected to port: %u\n", + lcores[l], temp[port].port_id); + port++; + l++; + } +} + +static void +main_loop(struct lcore_setup *lcore_setup) +{ + unsigned int i = 0, this_lcore; + this_lcore = rte_lcore_id(); + + /* print stats on master core */ + if (rte_get_master_lcore() == this_lcore) { + while (!global_exit_flag) { + print_stats(lcore_setup); + rte_delay_ms(500); + } + } + + for (i = 0; i < lcore_setup->rxte_lcores; i++) { + if (lcore_setup->rxte_lcore_list[i] == this_lcore) + bbdev_enqueue_enc(); + } + + for (i = 0; i < lcore_setup->tetx_lcores; i++) { + if (lcore_setup->tetx_lcore_list[i] == this_lcore) + bbdev_dequeue_enc(); + } + + for (i = 0; i < lcore_setup->rxtd_lcores; i++) { + if (lcore_setup->rxtd_lcore_list[i] == this_lcore) + bbdev_enqueue_dec(); + } + + for (i = 0; i < lcore_setup->tdtx_lcores; i++) { + if (lcore_setup->tdtx_lcore_list[i] == this_lcore) + bbdev_dequeue_dec(); + } +} + +static int +launch_one_lcore(void *arg) +{ + main_loop((struct lcore_setup *)arg); + return 0; +} + +static int +prepare_bbdev_device(unsigned int dev_id, uint8_t qs_nb) +{ + int ret; + unsigned int q_id; + struct rte_bbdev_info info; + struct rte_bbdev_queue_conf qconf = {0}; + + rte_bbdev_info_get(dev_id, &info); + + ret = rte_bbdev_setup_queues(dev_id, qs_nb, info.socket_id); + if (ret < 0) + rte_exit(EXIT_FAILURE, + "ERROR(%d): BBDEV %u not configured properly\n", + ret, dev_id); + + /* setup device queues */ + qconf.socket = info.socket_id; + qconf.queue_size = info.drv.queue_size_lim; + qconf.op_type = (dev_id & 0x1) ? RTE_BBDEV_OP_TURBO_DEC : + RTE_BBDEV_OP_TURBO_ENC; + + for (q_id = 0; q_id < qs_nb; q_id++) { + + ret = rte_bbdev_queue_configure(dev_id, q_id, &qconf); + if (ret < 0) + rte_exit(EXIT_FAILURE, + "ERROR(%d): BBDEV %u queue %u not configured properly\n", + ret, dev_id, q_id); + } + + ret = rte_bbdev_start(dev_id); + + if (ret != 0) + rte_exit(EXIT_FAILURE, "ERROR(%d): BBDEV %u not started\n", + ret, dev_id); + + printf("BBdev %u started\n", dev_id); + + return 0; +} + +static void +enable_bbdev(unsigned int lcores[], unsigned int nb_lcores, unsigned int dev_id) +{ + unsigned int i, nb_qs, tot_nb_bbdev_qs = 1, lcore, lcoreid; + struct lcore_queue_conf *qconf, *qconf_next; + unsigned int nb_bbdev_qs = 0; + char pool_name[RTE_MEMPOOL_NAMESIZE]; + + /* set num of port queues and bbdev queues for each lcore on the link */ + for (lcore = 0; lcore < nb_lcores; lcore++) { + nb_qs = 0; + lcoreid = lcores[lcore]; + qconf = &lcore_queue_conf[lcoreid]; + + for (i = 0; i < qconf->nb_ports; i++) + nb_qs += qconf->port_list->queues; + + for (i = 0; i < nb_qs; i++) { + qconf->bbdev_qs[i] = nb_bbdev_qs; + nb_bbdev_qs++; + } + + /* create the mbuf mempool for ethdev pkts */ + snprintf(pool_name, sizeof(pool_name), "bbdev%d_mbuf_pool%d", + dev_id, lcore); + qconf->mbuf_pool = rte_pktmbuf_pool_create(pool_name, + NB_MBUF, MEMPOOL_CACHE_SIZE, 0, + RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id()); + if (qconf->mbuf_pool == NULL) + rte_exit(EXIT_FAILURE, + "Unable to create '%s' pool\n", + pool_name); + + qconf->nb_bbdev_qs = nb_qs; + qconf->bbdev_id = dev_id; + printf("BBdev: %u, lcore %u\n", dev_id, lcoreid); + + qconf_next = &lcore_queue_conf[lcoreid + 1]; + qconf_next->bbdev_id = dev_id; + + for (i = 0; i < nb_qs; i++) + qconf_next->bbdev_qs[i] = qconf->bbdev_qs[i]; + + qconf_next->nb_bbdev_qs = nb_qs; + printf("BBdev: %u, lcore %u\n", dev_id, lcoreid + 1); + } + + /* count the number of queues needed on the bbdev device on this link */ + for (lcore = 0; lcore < nb_lcores; lcore++) { + lcoreid = lcores[lcore]; + qconf = &lcore_queue_conf[lcoreid]; + tot_nb_bbdev_qs += qconf->nb_bbdev_qs; + } + + prepare_bbdev_device(dev_id, tot_nb_bbdev_qs); +} + +int +main(int argc, char **argv) +{ + int ret, nb_bbdevs, nb_ports; + int enabled_portcount = 0; + uint8_t lcore_id; + char rx = 'R', tx = 'T', up_link = 'U', down_link = 'D'; + void *sigret; + uint32_t bbdev_enabled_port_mask = 0; + struct lcore_setup lcore_setup = { 0 }; + struct bbdev_config_params bbdev_params = { 0 }; + + sigret = signal(SIGTERM, signal_handler); + if (sigret == SIG_ERR) + rte_exit(EXIT_FAILURE, "signal(%d, ...) failed", SIGTERM); + + sigret = signal(SIGINT, signal_handler); + if (sigret == SIG_ERR) + rte_exit(EXIT_FAILURE, "signal(%d, ...) failed", SIGINT); + + ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n"); + + argc -= ret; + argv += ret; + + /* parse application arguments (after the EAL ones) */ + ret = bbdev_parse_args(argc, argv, &bbdev_params); + + if (ret < 0) + rte_exit(EXIT_FAILURE, "Invalid BBDEV arguments\n"); + + /* Get number of available bbdev devices */ + nb_bbdevs = rte_bbdev_count(); + if (nb_bbdevs == 0) + rte_exit(EXIT_FAILURE, "No bbdevs detected!\n"); + printf("Number of bbdevs detected: %d\n", nb_bbdevs); + + /* Get number of available ethdev devices */ + nb_ports = rte_eth_dev_count(); + /* Initialize the port/queue configuration of each logical core */ + ret = lcore_queue_init(nb_ports, &bbdev_params, &lcore_setup); + if (ret < nb_bbdevs * 2) + rte_exit(EXIT_FAILURE, "Failed to initialize queues.\n"); + + /* reconfigure ports to lcores assignment to allow for multiple lcores + * usage + */ + qs_to_lcores(lcore_setup.rxte_lcores, lcore_setup.rxte_lcore_list); + qs_to_lcores(lcore_setup.rxtd_lcores, lcore_setup.rxtd_lcore_list); + qs_to_lcores(lcore_setup.tdtx_lcores, lcore_setup.tdtx_lcore_list); + qs_to_lcores(lcore_setup.tetx_lcores, lcore_setup.tetx_lcore_list); + + /* create the mbuf mempool for ethdev pkts */ + ethdev_mbuf_mempool = rte_pktmbuf_pool_create("ethdev_mbuf_pool", + NB_MBUF, MEMPOOL_CACHE_SIZE, 0, + RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id()); + if (ethdev_mbuf_mempool == NULL) + rte_exit(EXIT_FAILURE, "Cannot create ethdev mbuf mempool\n"); + + /*initialize ports*/ + enabled_portcount = initialize_ports(bbdev_params.downlink_rx_ports, + down_link, rx, &bbdev_enabled_port_mask); + enabled_portcount += initialize_ports(bbdev_params.downlink_tx_ports, + down_link, tx, &bbdev_enabled_port_mask); + enabled_portcount += initialize_ports(bbdev_params.uplink_rx_ports, + up_link, rx, &bbdev_enabled_port_mask); + enabled_portcount += initialize_ports(bbdev_params.uplink_tx_ports, + up_link, tx, &bbdev_enabled_port_mask); + + if (enabled_portcount < 1) + rte_exit(EXIT_FAILURE, "Failed to initialize Ethernet ports\n"); + + /*start Ethernet devices*/ + ret = start_ethdev_ports(nb_ports, bbdev_enabled_port_mask); + if (ret <= 0) + rte_exit(EXIT_FAILURE, + "Failed to start Ethernet devices on ports\n"); + check_all_ports_link_status(nb_ports, bbdev_enabled_port_mask); + + /*create bbdev op pools*/ + bbdev_op_pool[RTE_BBDEV_OP_TURBO_DEC] = + rte_bbdev_op_pool_create("bbdev_op_pool_dec", + RTE_BBDEV_OP_TURBO_DEC, NB_MBUF, 128, rte_socket_id()); + bbdev_op_pool[RTE_BBDEV_OP_TURBO_ENC] = + rte_bbdev_op_pool_create("bbdev_op_pool_enc", + RTE_BBDEV_OP_TURBO_ENC, NB_MBUF, 128, rte_socket_id()); + + if ((bbdev_op_pool[RTE_BBDEV_OP_TURBO_DEC] == NULL) || + (bbdev_op_pool[RTE_BBDEV_OP_TURBO_ENC] == NULL)) + rte_exit(EXIT_FAILURE, "Cannot create bbdev op pools\n"); + + /*start DL bbdev device*/ + if (lcore_setup.rxte_lcores) + enable_bbdev(lcore_setup.rxte_lcore_list, + lcore_setup.rxte_lcores, 0); + /*start UL bbdev device*/ + if (lcore_setup.rxtd_lcores) + enable_bbdev(lcore_setup.rxtd_lcore_list, + lcore_setup.rxtd_lcores, 1); + + /*launch per-lcore init on every lcore*/ + rte_eal_mp_remote_launch(launch_one_lcore, (void *)&lcore_setup, + CALL_MASTER); + + RTE_LCORE_FOREACH_SLAVE(lcore_id) { + if (rte_eal_wait_lcore(lcore_id) < 0) + return -1; + } + + return 0; +} -- 2.7.4 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [dpdk-dev] [PATCH v2 5/5] bbdev: documentation 2017-10-18 2:14 [dpdk-dev] [PATCH v2 1/5] bbdev: librte_bbdev library Amr Mokhtar ` (2 preceding siblings ...) 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 4/5] bbdev: sample app Amr Mokhtar @ 2017-10-18 2:14 ` Amr Mokhtar 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 0/5] Wireless Base Band Device (bbdev) Amr Mokhtar 4 siblings, 0 replies; 6+ messages in thread From: Amr Mokhtar @ 2017-10-18 2:14 UTC (permalink / raw) To: dev Cc: thomas, anatoly.burakov, pablo.de.lara.guarch, niall.power, chris.macnamara, Amr Mokhtar - Wireless Baseband Device Library Programmer’s Guide - test-bbdev User Guide - BBDEV Sample Application User Guides - Baseband Device Drivers Guides - Doxygen API Signed-off-by: Amr Mokhtar <amr.mokhtar@intel.com> --- doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf | 1 + doc/guides/bbdevs/index.rst | 40 +++ doc/guides/bbdevs/null.rst | 77 ++++ doc/guides/bbdevs/turbo_sw.rst | 169 +++++++++ doc/guides/index.rst | 1 + doc/guides/prog_guide/bbdev.rst | 621 +++++++++++++++++++++++++++++++++ doc/guides/prog_guide/index.rst | 1 + doc/guides/sample_app_ug/bbdev_app.rst | 160 +++++++++ doc/guides/sample_app_ug/index.rst | 1 + doc/guides/tools/index.rst | 1 + doc/guides/tools/testbbdev.rst | 546 +++++++++++++++++++++++++++++ 12 files changed, 1619 insertions(+) create mode 100644 doc/guides/bbdevs/index.rst create mode 100644 doc/guides/bbdevs/null.rst create mode 100644 doc/guides/bbdevs/turbo_sw.rst create mode 100644 doc/guides/prog_guide/bbdev.rst create mode 100644 doc/guides/sample_app_ug/bbdev_app.rst create mode 100644 doc/guides/tools/testbbdev.rst diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md index 990815f..0e2227b 100644 --- a/doc/api/doxy-api-index.md +++ b/doc/api/doxy-api-index.md @@ -48,6 +48,7 @@ The public API headers are grouped by topics: [bitrate] (@ref rte_bitrate.h), [latency] (@ref rte_latencystats.h), [devargs] (@ref rte_devargs.h), + [bbdev] (@ref rte_bbdev.h), [PCI] (@ref rte_pci.h) - **device specific**: diff --git a/doc/api/doxy-api.conf b/doc/api/doxy-api.conf index 9e9fa56..d4a0890 100644 --- a/doc/api/doxy-api.conf +++ b/doc/api/doxy-api.conf @@ -39,6 +39,7 @@ INPUT = doc/api/doxy-api-index.md \ lib/librte_eal/common/include \ lib/librte_eal/common/include/generic \ lib/librte_acl \ + lib/librte_bbdev \ lib/librte_bitratestats \ lib/librte_cfgfile \ lib/librte_cmdline \ diff --git a/doc/guides/bbdevs/index.rst b/doc/guides/bbdevs/index.rst new file mode 100644 index 0000000..c9aa1b0 --- /dev/null +++ b/doc/guides/bbdevs/index.rst @@ -0,0 +1,40 @@ +.. + BSD LICENSE + + Copyright(c) 2017 Intel Corporation. All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Intel Corporation nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Baseband Device Drivers +======================= + +.. toctree:: + :maxdepth: 2 + :numbered: + + null + turbo_sw diff --git a/doc/guides/bbdevs/null.rst b/doc/guides/bbdevs/null.rst new file mode 100644 index 0000000..0f40232 --- /dev/null +++ b/doc/guides/bbdevs/null.rst @@ -0,0 +1,77 @@ +.. + BSD LICENSE + + Copyright(c) 2017 Intel Corporation. All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Intel Corporation nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +BBDEV null Poll Mode Driver +============================ + +The (**bbdev_null**) is a bbdev poll mode driver which provides a minimal +implementation of a software bbdev device. As a null device it does not modify +the data in the mbuf on which the bbdev operation is to operate and it only +works for operation type ``RTE_BBDEV_OP_NONE``. + +When a burst of mbufs is submitted to a *bbdev null PMD* for processing then +each mbuf in the burst will be enqueued in an internal buffer ring to be +collected on a dequeue call. + + +Limitations +----------- + +* In-place operations for Turbo encode and decode are not supported + +Installation +------------ + +The *bbdev null PMD* is enabled and built by default in both the Linux and +FreeBSD builds. + +Initialization +-------------- + +To use the PMD in an application, user must: + +- Call ``rte_vdev_init("bbdev_null")`` within the application. + +- Use ``--vdev="bbdev_null"`` in the EAL options, which will call ``rte_vdev_init()`` internally. + +The following parameters (all optional) can be provided in the previous two calls: + +* ``socket_id``: Specify the socket where the memory for the device is going to be allocated + (by default, *socket_id* will be the socket where the core that is creating the PMD is running on). + +* ``max_nb_queues``: Specify the maximum number of queues in the device (default is ``RTE_MAX_LCORE``). + +Example: +~~~~~~~~ + +.. code-block:: console + + ./test-bbdev.py -e="--vdev=bbdev_null,socket_id=0,max_nb_queues=8" diff --git a/doc/guides/bbdevs/turbo_sw.rst b/doc/guides/bbdevs/turbo_sw.rst new file mode 100644 index 0000000..46833b5 --- /dev/null +++ b/doc/guides/bbdevs/turbo_sw.rst @@ -0,0 +1,169 @@ +.. + BSD LICENSE + + Copyright(c) 2017 Intel Corporation. All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Intel Corporation nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +SW Turbo Poll Mode Driver +========================= + +The SW Turbo PMD (**turbo_sw**) provides a poll mode bbdev driver that utilizes +Intel optimized libraries for LTE Layer 1 workloads acceleration. This PMD +supports the functions: Turbo FEC, Rate Matching and CRC functions. + +Features +-------- + +SW Turbo PMD has support for the following capabilities: + +For the encode operation: + +* ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` +* ``RTE_BBDEV_TURBO_RAW_INPUT_DATA`` + +For the decode operation: + +* ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` +* ``RTE_BBDEV_TURBO_RATE_MATCH`` +* ``RTE_BBDEV_TURBO_RV_INDEX_BYPASS`` + + +Limitations +----------- + +* In-place operations for Turbo encode and decode are not supported + +Installation +------------ + +To build DPDK with the *turbo_sw* the user is required to download +the export controlled ``FlexRAN SDK`` Libraries, by requesting them from +`<https://networkbuilders.intel.com/network-technologies/dpdk>`_. +Once approval has been granted, the user needs to log in +`<https://networkbuilders.intel.com/dpdklogin>`_ +and click on "FlexRAN SDK Libraries" link, to download the libraries. +After downloading the libraries, the user needs to unpack and compile on their +system before building DPDK. + +FlexRAN SDK Installation +~~~~~~~~~~~~~~~~~~~~~~~~ + +The following are pre-requisites for building FlexRAN SDK Libraries: + (a) An AVX2 supporting machine + (b) Windriver TS 2 or CentOS 7 operating systems + (c) Intel ICC compiler installed + +The following instructions should be followed in this exact order: + +#. Set the environment variables: + + .. code-block:: console + + export ICC_LOCATION=<path-to-icc-compiler-install-folder> + export CPATH=$ICC_LOCATION/linux/mkl/include/:$ICC_LOCATION/linux/ipp/include/ + + +#. Extract the SDK-R1.3.0 package: + + .. code-block:: console + + cd <path-to-workspace>/ + ./SDK-R1.3.0.sh + + +#. To allow FlexRAN SDK R1.3.0 to work with bbdev properly, the following + hotfix is required. Change the return of function ``bit_selection_complete_avx2()`` + located in file + ``<path-to-workspace>/SDK-R1.3.0/sdk/source/phy/lib_rate_matching/phy_rate_match_avx2.cpp`` + to return 0 instead of 1. + + .. code-block:: c + + - return 1; + + return 0; + +#. Install SDK-R1.3.0 package: + + .. code-block:: console + + cd SDK-R1.3.0/sdk/ + ./create-makefiles-linux.sh + cd build-avx2-icc/ + make install + + +Initialization +-------------- + +In order to enable this virtual bbdev PMD, the user must: + +* Build the ``FLEXRAN SDK`` libraries (explained in Installation section). + +* Export ICC environment variables: + +.. code-block:: console + + export LIBRARY_PATH=$ICC_LOCATION/linux/compiler/lib/intel64_lin/:$ICC_LOCATION/linux/ipp/lib/intel64_lin/ + export LD_LIBRARY_PATH=$ICC_LOCATION/linux/compiler/lib/intel64_lin/:$ICC_LOCATION/linux/ipp/lib/intel64_lin/ + + +* Export the environmental variables ``FLEXRAN_SDK`` to the path where the + FlexRAN library was installed. And ``DIR_WIRELESS_SDK`` to the path where the + library was extracted. + +Example: + +.. code-block:: console + + export FLEXRAN_SDK=<path-to-workspace>/SDK-R1.3.0/sdk/build-avx2-icc/install + export DIR_WIRELESS_SDK=<path-to-workspace>/SDK-R1.3.0/sdk/ + + +* Set ``CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y`` in DPDK common configuration + file ``config/common_base``. + +To use the PMD in an application, user must: + +- Call ``rte_vdev_init("turbo_sw")`` within the application. + +- Use ``--vdev="turbo_sw"`` in the EAL options, which will call ``rte_vdev_init()`` internally. + +The following parameters (all optional) can be provided in the previous two calls: + +* ``socket_id``: Specify the socket where the memory for the device is going to be allocated + (by default, *socket_id* will be the socket where the core that is creating the PMD is running on). + +* ``max_nb_queues``: Specify the maximum number of queues in the device (default is ``RTE_MAX_LCORE``). + +Example: +~~~~~~~~ + +.. code-block:: console + + ./test-bbdev.py -e="--vdev=turbo_sw,socket_id=0,max_nb_queues=8" \ + -c validation -v ./test_vectors/bbdev_vector_t?_default.data diff --git a/doc/guides/index.rst b/doc/guides/index.rst index 5b6eb7e..e3df7ae 100644 --- a/doc/guides/index.rst +++ b/doc/guides/index.rst @@ -44,6 +44,7 @@ DPDK documentation nics/index cryptodevs/index eventdevs/index + bbdevs/index contributing/index rel_notes/index faq/index diff --git a/doc/guides/prog_guide/bbdev.rst b/doc/guides/prog_guide/bbdev.rst new file mode 100644 index 0000000..7e7cde6 --- /dev/null +++ b/doc/guides/prog_guide/bbdev.rst @@ -0,0 +1,621 @@ +.. + BSD LICENSE + + Copyright(c) 2017 Intel Corporation. All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Intel Corporation nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Wireless Baseband Device Library +================================ + +The Wireless Baseband library provides a common programming framework that +abstracts HW accelerators based on FPGA and/or Fixed Function Accelerators that +assist with 3gpp Physical Layer processing. Furthermore, it decouples the +application from the compute-intensive wireless functions by abstracting their +optimized libraries to appear as virtual bbdev devices. + +The functional scope of the BBDEV library are those functions in relation to +the 3gpp Layer 1 signal processing (channel coding, modulation, ...). + +The framework currently only supports Turbo Code FEC function. + + +Design Principles +----------------- + +The Wireless Baseband library follows the same ideology of DPDK's Ethernet +Device and Crypto Device frameworks. Wireless Baseband provides a generic +acceleration abstraction framework which supports both physical (hardware) and +virtual (software) wireless acceleration functions. + +Device Management +----------------- + +Device Creation +~~~~~~~~~~~~~~~ + +Physical bbdev devices are discovered during the PCI probe/enumeration of the +EAL function which is executed at DPDK initialization, based on +their PCI device identifier, each unique PCI BDF (bus/bridge, device, +function). + +Virtual devices can be created by two mechanisms, either using the EAL command +line options or from within the application using an EAL API directly. + +From the command line using the --vdev EAL option + +.. code-block:: console + + --vdev 'turbo_sw,max_nb_queues=8,socket_id=0' + +Our using the rte_vdev_init API within the application code. + +.. code-block:: c + + rte_vdev_init("turbo_sw", "max_nb_queues=2,socket_id=0") + +All virtual bbdev devices support the following initialization parameters: + +- ``max_nb_queues`` - maximum number of queues supported by the device. + +- ``socket_id`` - socket on which to allocate the device resources on. + + +Device Identification +~~~~~~~~~~~~~~~~~~~~~ + +Each device, whether virtual or physical is uniquely designated by two +identifiers: + +- A unique device index used to designate the bbdev device in all functions + exported by the bbdev API. + +- A device name used to designate the bbdev device in console messages, for + administration or debugging purposes. For ease of use, the port name includes + the port index. + + +Device Configuration +~~~~~~~~~~~~~~~~~~~~ + +From the application point of view, each instance of a bbdev device consists of +one or more queues identified by queue IDs. While different devices may have +different capabilities (e.g. support different operation types), all queues on +a device support identical configuration possibilities. A queue is configured +for only one type of operation and is configured at initializations time. +When an operation is enqueued to a specific queue ID, the result is dequeued +from the same queue ID. + +Configuration of a device has two different levels: configuration that applies +to the whole device, and configuration that applies to a single queue. + +Device configuration is applied with +``rte_bbdev_setup_queues(dev_id,num_queues,socket_id)`` +and queue configuration is applied with +``rte_bbdev_queue_configure(dev_id,queue_id,conf)``. Note that, although all +queues on a device support same capabilities, they can be configured differently +and will then behave differently. +Devices supporting interrupts can enable them by using +``rte_bbdev_intr_enable(dev_id)``. + +The configuration of each bbdev device includes the following operations: + +- Allocation of resources, including hardware resources if a physical device. +- Resetting the device into a well-known default state. +- Initialization of statistics counters. + +The ``rte_bbdev_setup_queues`` API is used to setup queues for a bbdev device. + +.. code-block:: c + + int rte_bbdev_setup_queues(uint16_t dev_id, uint16_t num_queues, + int socket_id); + +- ``num_queues`` argument identifies the total number of queues to setup for + this device. + +- ``socket_id`` specifies which socket will be used to allocate the memory. + + +The ``rte_bbdev_intr_enable`` API is used to enable interrupts for a bbdev +device, if supported by the driver. Should be called before starting the device. + +.. code-block:: c + + int rte_bbdev_intr_enable(uint16_t dev_id); + + +Queues Configuration +~~~~~~~~~~~~~~~~~~~~ + +Each bbdev devices queue is individually configured through the +``rte_bbdev_queue_configure()`` API. +Each queue resources may be allocated on a specified socket. + +.. code-block:: c + + struct rte_bbdev_queue_conf { + int socket; /**< NUMA socket used for memory allocation */ + uint32_t queue_size; /**< Size of queue */ + uint8_t priority; /**< Queue priority */ + bool deferred_start; /**< Do not start queue when device is started. */ + enum rte_bbdev_op_type op_type; /**< Operation type */ + }; + +Device & Queues Management +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +After initialization, devices are in a stopped state, so must be started by the +application. If an application is finished using a device it can close the +device. Once closed, it cannot be restarted. + +.. code-block:: c + + int rte_bbdev_start(uint16_t dev_id) + int rte_bbdev_stop(uint16_t dev_id) + int rte_bbdev_close(uint16_t dev_id) + int rte_bbdev_queue_start(uint16_t dev_id, uint16_t queue_id) + int rte_bbdev_queue_stop(uint16_t dev_id, uint16_t queue_id) + + +By default, all queues are started when the device is started, but they can be +stopped individually. + +.. code-block:: c + + int rte_bbdev_queue_start(uint16_t dev_id, uint16_t queue_id) + int rte_bbdev_queue_stop(uint16_t dev_id, uint16_t queue_id) + + +Logical Cores, Memory and Queues Relationships +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The bbdev device Library as the Poll Mode Driver library support NUMA for when +a processor’s logical cores and interfaces utilize its local memory. Therefore +baseband operations, the mbuf being operated on should be allocated from memory +pools created in the local memory. The buffers should, if possible, remain on +the local processor to obtain the best performance results and buffer +descriptors should be populated with mbufs allocated from a mempool allocated +from local memory. + +The run-to-completion model also performs better, especially in the case of +virtual bbdev devices, if the baseband operation and data buffers are in local +memory instead of a remote processor's memory. This is also true for the +pipe-line model provided all logical cores used are located on the same processor. + +Multiple logical cores should never share the same queue for enqueuing +operations or dequeuing operations on the same bbdev device since this would +require global locks and hinder performance. It is however possible to use a +different logical core to dequeue an operation on a queue pair from the logical +core which it was enqueued on. This means that a baseband burst enqueue/dequeue +APIs are a logical place to transition from one logical core to another in a +packet processing pipeline. + + +Device Operation Capabilities +----------------------------- + +Capabilities (in terms of operations supported, max number of queues, etc.) +identify what a bbdev is capable of performing that differs from one device to +another. For the full scope of the bbdev capability see the definition of the +structure in the *DPDK API Reference*. + +.. code-block:: c + + struct rte_bbdev_op_cap; + +A device reports its capabilities when registering itself in the bbdev framework. +With the aid of this capabilities mechanism, an application can query devices to +discover which operations within the 3gpp physical layer they are capable of +performing. Below is an example of the capabilities for a PMD it supports in +relation to Turbo Encoding and Decoding operations. + +.. code-block:: c + + static const struct rte_bbdev_op_cap bbdev_capabilities[] = { + { + .type = RTE_BBDEV_OP_TURBO_DEC, + .cap.turbo_dec = { + .capability_flags = + RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE | + RTE_BBDEV_TURBO_RAW_INPUT_DATA, + .num_buffers_src = 1, + .num_buffers_hard_out = 1, + .num_buffers_soft_out = 0, + } + }, + { + .type = RTE_BBDEV_OP_TURBO_ENC, + .cap.turbo_enc = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_24B_ATTACH | + RTE_BBDEV_TURBO_RATE_MATCH | + RTE_BBDEV_TURBO_RV_INDEX_BYPASS, + .num_buffers_src = 1, + .num_buffers_dst = 1, + } + }, + RTE_BBDEV_END_OF_CAPABILITIES_LIST() + }; + +Capabilities Discovery +~~~~~~~~~~~~~~~~~~~~~~ + +Discovering the features and capabilities of a bbdev device poll mode driver +is achieved through the ``rte_bbdev_info_get()`` function. + +.. code-block:: c + + int rte_bbdev_info_get(uint16_t dev_id, struct rte_bbdev_info *dev_info) + +This allows the user to query a specific bbdev PMD and get all the device +capabilities. The ``rte_bbdev_info`` structure provides two levels of +information: + +- Device relevant information, like: name and related rte_bus. + +- Driver specific information, as defined by the ``struct rte_bbdev_driver_info`` + structure, this is where capabilities reside along with other specifics like: + maximum queue sizes and priority level. + +.. code-block:: c + + struct rte_bbdev_info { + int socket_id; /**< NUMA socket that device is on */ + const char *dev_name; /**< Unique device name */ + const struct rte_bus *bus; /**< Bus information */ + uint16_t num_queues; /**< Number of queues currently configured */ + bool started; /**< Set if device is currently started */ + struct rte_bbdev_driver_info drv; /**< Info from device driver */ + }; + +Operation Processing +-------------------- + +Scheduling of baseband operations on DPDK's application data path is +performed using a burst oriented asynchronous API set. A queue on a bbdev +device accepts a burst of baseband operations using enqueue burst API. On physical +bbdev devices the enqueue burst API will place the operations to be processed +on the device's hardware input queue, for virtual devices the processing of the +baseband operations is usually completed during the enqueue call to the bbdev +device. The dequeue burst API will retrieve any processed operations available +from the queue on the bbdev device, from physical devices this is usually +directly from the device's processed queue, and for virtual device's from a +``rte_ring`` where processed operations are place after being processed on the +enqueue call. + + +Enqueue / Dequeue Burst APIs +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The burst enqueue API uses a bbdev device identifier and a queue +identifier to specify the bbdev device queue to schedule the processing on. +The ``num_ops`` parameter is the number of operations to process which are +supplied in the ``ops`` array of ``rte_bbdev_*_op`` structures. +The enqueue function returns the number of operations it actually enqueued for +processing, a return value equal to ``num_ops`` means that all packets have been +enqueued. + +.. code-block:: c + + uint16_t rte_bbdev_enqueue_enc_ops(uint16_t dev_id, uint16_t queue_id, + struct rte_bbdev_enc_op **ops, uint16_t num_ops) + + uint16_t rte_bbdev_enqueue_dec_ops(uint16_t dev_id, uint16_t queue_id, + struct rte_bbdev_dec_op **ops, uint16_t num_ops) + +The dequeue API uses the same format as the enqueue API of processed but +the ``num_ops`` and ``ops`` parameters are now used to specify the max processed +operations the user wishes to retrieve and the location in which to store them. +The API call returns the actual number of processed operations returned, this +can never be larger than ``num_ops``. + +.. code-block:: c + + uint16_t rte_bbdev_dequeue_enc_ops(uint16_t dev_id, uint16_t queue_id, + struct rte_bbdev_enc_op **ops, uint16_t num_ops) + + uint16_t rte_bbdev_dequeue_dec_ops(uint16_t dev_id, uint16_t queue_id, + struct rte_bbdev_dec_op **ops, uint16_t num_ops) + +Operation Representation +~~~~~~~~~~~~~~~~~~~~~~~~ + +An encode bbdev operation is represented by ``rte_bbdev_enc_op`` structure, +and by ``rte_bbdev_dec_op`` for decode. These structures act as metadata +containers for all necessary information required for the bbdev operation to be +processed on a particular bbdev device poll mode driver. + +.. code-block:: c + + struct rte_bbdev_enc_op { + int status; + struct rte_mempool *mempool; + void *opaque_data; + /** Contains encoder specific parameters */ + struct rte_bbdev_op_turbo_enc turbo_enc; + }; + + struct rte_bbdev_dec_op { + int status; + struct rte_mempool *mempool; + void *opaque_data; + /** Contains decoder specific parameters */ + struct rte_bbdev_op_turbo_dec turbo_dec; + }; + +The operation structure by itself defines the operation type. It includes an +operation status, a reference to the operation specific data, which can vary in +size and content depending on the operation being provisioned. It also contains +the source mempool for the operation, if it is allocated from a mempool. + +If bbdev operations are allocated from a bbdev operation mempool, see next +section, there is also the ability to allocate private memory with the +operation for applications purposes. + +Application software is responsible for specifying all the operation specific +fields in the ``rte_bbdev_*_op`` structure which are then used by the bbdev PMD +to process the requested operation. + + +Operation Management and Allocation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The bbdev library provides an API set for managing bbdev operations which +utilize the Mempool Library to allocate operation buffers. Therefore, it ensures +that the bbdev operation is interleaved optimally across the channels and +ranks for optimal processing. + +.. code-block:: c + + struct rte_mempool * + rte_bbdev_op_pool_create(const char *name, enum rte_bbdev_op_type type, + unsigned int num_elements, unsigned int cache_size, + int socket_id) + +``rte_bbdev_*_op_alloc_bulk()`` and ``rte_bbdev_*_op_free_bulk()`` are used to +allocate bbdev operations of a specific type from a given bbdev operation mempool. + +.. code-block:: c + + int rte_bbdev_enc_op_alloc_bulk(struct rte_mempool *mempool, + struct rte_bbdev_enc_op **ops, uint16_t num_ops) + + int rte_bbdev_dec_op_alloc_bulk(struct rte_mempool *mempool, + struct rte_bbdev_dec_op **ops, uint16_t num_ops) + +``rte_bbdev_*_op_free_bulk()`` is called by the application to return an +operation to its allocating pool. + +.. code-block:: c + + void rte_bbdev_dec_op_free_bulk(struct rte_bbdev_dec_op **ops, + unsigned int num_ops) + void rte_bbdev_enc_op_free_bulk(struct rte_bbdev_enc_op **ops, + unsigned int num_ops) + +BBDEV Operations +~~~~~~~~~~~~~~~~ + +The bbdev operation structure contains all the mutable data relating to +performing Turbo code processing on a referenced mbuf data buffer. It is used +for either encode or decode operations. + +Turbo Encode operation accepts one input and one output. + +Turbo Decode operation accepts one input and two outputs, called *hard-decision* +and *soft-decision* outputs. *Soft-decision* output is optional. + +It is expected that the application provides input and output ``mbuf`` pointers +allocated and ready to use. The baseband framework supports turbo coding on +Code Blocks (CB) and Transport Blocks (TB). + +For the output buffer(s), the application needs only to provide an allocated and +free mbuf (containing only one mbuf segment), so that bbdev can write the +operation outcome. + +**Turbo Encode Op structure** + +.. code-block:: c + + struct rte_bbdev_op_turbo_enc { + struct rte_bbdev_op_data input; /**< input src data */ + struct rte_bbdev_op_data output; /**< output buffer */ + + uint32_t op_flags; + int32_t n_soft; + int32_t k_mimo; + int32_t mdl_harq; + + int32_t g; + int32_t nl; + int32_t qm; + uint8_t rv_index; + + uint8_t code_block_mode; /**< 0 - transpot block, 1 - code block */ + union { + struct rte_bbdev_op_enc_cb_params cb_params; + struct rte_bbdev_op_enc_tb_params tb_params; + }; + }; + + +**Turbo Decode Op structure** + +.. code-block:: c + + struct rte_bbdev_op_turbo_dec { + struct rte_bbdev_op_data input; /**< input src data */ + struct rte_bbdev_op_data hard_output; /**< hard output buffer */ + struct rte_bbdev_op_data soft_output; /**< soft output buffer */ + + uint32_t op_flags; + uint8_t rv_index; + uint8_t iter_min:4; + uint8_t iter_max:4; + uint8_t iter_count; + uint8_t ext_scale; + uint8_t num_maps; + + uint8_t code_block_mode; /**< 0 - transpot block, 1 - code block */ + union { + struct rte_bbdev_op_dec_cb_params cb_params; + struct rte_bbdev_op_dec_tb_params tb_params; + }; + }; + +Input and output data buffers are identified by ``rte_bbdev_op_data`` structure. +This strucutre has three elements: + +- ``data`` - This is the mbuf reference + +- ``offset`` - The starting point for the Turbo input/output, in bytes, from the + start of the data in the data buffer. It must be smaller than data_len of the + mbuf's first segment + +- ``length`` - The length, in bytes, of the buffer on which the Turbo operation + will or has been computed. For the input, the length is set by the application. + For the output(s), the length is computed by the bbdev PMD driver. + +Sample code +----------- + +The baseband device sample application gives an introduction on how to use the +bbdev framework, by giving a sample code performing a loop-back operation with a +baseband processor capable of transceiving data packets. + +The following sample pseudo-code shows the basic steps to encode several buffers +using (**sw_trubo**) bbdev PMD. + +.. code-block:: c + + /* EAL Init */ + ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n"); + + /* Get number of available bbdev devices */ + nb_bbdevs = rte_bbdev_count(); + if (nb_bbdevs == 0) + rte_exit(EXIT_FAILURE, "No bbdevs detected!\n"); + + /* Create bbdev op pools */ + bbdev_op_pool[RTE_BBDEV_OP_TURBO_ENC] = + rte_bbdev_op_pool_create("bbdev_op_pool_enc", + RTE_BBDEV_OP_TURBO_ENC, NB_MBUF, 128, rte_socket_id()); + + /* Get information for this device */ + rte_bbdev_info_get(dev_id, &info); + + /* Setup BBDEV device queues */ + ret = rte_bbdev_setup_queues(dev_id, qs_nb, info.socket_id); + if (ret < 0) + rte_exit(EXIT_FAILURE, + "ERROR(%d): BBDEV %u not configured properly\n", + ret, dev_id); + + /* setup device queues */ + qconf.socket = info.socket_id; + qconf.queue_size = info.drv.queue_size_lim; + qconf.op_type = RTE_BBDEV_OP_TURBO_ENC; + + for (q_id = 0; q_id < qs_nb; q_id++) { + /* Configure all queues belonging to this bbdev device */ + ret = rte_bbdev_queue_configure(dev_id, q_id, &qconf); + if (ret < 0) + rte_exit(EXIT_FAILURE, + "ERROR(%d): BBDEV %u queue %u not configured properly\n", + ret, dev_id, q_id); + } + + /* Start bbdev device */ + ret = rte_bbdev_start(dev_id); + + /* Create the mbuf mempool for pkts */ + mbuf_pool = rte_pktmbuf_pool_create("bbdev_mbuf_pool", + NB_MBUF, MEMPOOL_CACHE_SIZE, 0, + RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id()); + if (mbuf_pool == NULL) + rte_exit(EXIT_FAILURE, + "Unable to create '%s' pool\n", pool_name); + + while (!global_exit_flag) { + + /* Allocate burst of op structures in preparation for enqueue */ + if (rte_bbdev_enc_op_alloc_bulk(bbdev_op_pool[RTE_BBDEV_OP_TURBO_ENC], + op_type, ops_burst, op_num) != 0) + continue; + + /* Allocate input mbuf pkts */ + ret = rte_pktmbuf_alloc_bulk(mbuf_pool, input_pkts_burst, MAX_PKT_BURST); + if (ret < 0) + continue; + + /* Allocate output mbuf pkts */ + ret = rte_pktmbuf_alloc_bulk(mbuf_pool, output_pkts_burst, MAX_PKT_BURST); + if (ret < 0) + continue; + + for (j = 0; j < op_num; j++) { + /* Append the size of the ethernet header */ + rte_pktmbuf_append(input_pkts_burst[j], + sizeof(struct ether_hdr)); + + /* set op */ + + ops_burst[j]->turbo_enc->input.offset = + sizeof(struct ether_hdr); + + ops_burst[j]->turbo_enc->input.length = + rte_pktmbuf_pkt_len(bbdev_pkts[j]); + + ops_burst[j]->turbo_enc->input.data = + input_pkts_burst[j]; + + ops_burst[j]->turbo_enc->output.offset = + sizeof(struct ether_hdr); + + ops_burst[j]->turbo_enc->output.data = + output_pkts_burst[j]; + } + + /* Enqueue packets on BBDEV device */ + op_num = rte_bbdev_enqueue_ops(qconf->bbdev_id, + qconf->bbdev_qs[q], ops_burst, + MAX_PKT_BURST); + + /* Dequeue packets from BBDEV device*/ + op_num = rte_bbdev_dequeue_ops(qconf->bbdev_id, + qconf->bbdev_qs[q], ops_burst, + MAX_PKT_BURST); + } + + +BBDEV Device API +~~~~~~~~~~~~~~~~ + +The bbdev Library API is described in the *DPDK API Reference* document. diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst index b5ad6b8..421c0f6 100644 --- a/doc/guides/prog_guide/index.rst +++ b/doc/guides/prog_guide/index.rst @@ -63,6 +63,7 @@ Programmer's Guide kernel_nic_interface thread_safety_dpdk_functions eventdev + bbdev qos_framework power_man packet_classif_access_ctrl diff --git a/doc/guides/sample_app_ug/bbdev_app.rst b/doc/guides/sample_app_ug/bbdev_app.rst new file mode 100644 index 0000000..dee37ad --- /dev/null +++ b/doc/guides/sample_app_ug/bbdev_app.rst @@ -0,0 +1,160 @@ +.. BSD LICENSE + Copyright(c) 2017 Intel Corporation. All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Intel Corporation nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +.. bbdev_app: + +Loop-back Sample Application using Baseband Device (bbdev) +========================================================== + +The baseband sample application is a simple example of packet processing using +the Data Plane Development Kit (DPDK) for baseband workloads using the bbdev +library. + +Overview +-------- + +The Baseband device sample application performs a loop-back operation using a +baseband device capable of transceiving data packets. +A packet is received on a DOWNLINK_RX_PORT or UPLINK_RX_PORTS -> enqueued for +baseband operation -> dequeued from the baseband device then looped back to +DOWNLINK_TX_PORTS or UPLINK_TX_PORTS, respectively. + +* The source MAC address is replaced by the TX_PORT MAC address + +* The destination MAC address is replaced by 02:00:00:00:00:TX_PORT_ID + +Limitations +----------- + +* Although, the baseband sample application is designed to work in full-duplex + mode, but due to theplain format of the received packets from pkt-gen, the + Turbo decode (uplink) is not operable. Only the Turbo encode direction + (downlink) is currently supported in the bbdev sample application. + +Compiling the Application +------------------------- + +#. DPDK needs to be built with ``turbo_sw`` PMD driver enabled along with + ``FLEXRAN SDK`` Libraries. Refer to *SW Turbo Poll Mode Driver* + documentation for more details on this. + +#. Go to the example directory: + + .. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + cd ${RTE_SDK}/examples/bbdev_app + +#. Set the target (a default target is used if not specified). For example: + + .. code-block:: console + + export RTE_TARGET=x86_64-native-linuxapp-gcc + + See the *DPDK Getting Started Guide* for possible RTE_TARGET values. + +#. Build the application: + + .. code-block:: console + + make + +Running the Application +----------------------- + +The application requires a number of command line options: + +.. code-block:: console + + $ ./build/bbdev [EAL options] -- [-r DOWNLINK_RX_PORTS] [-R UPLINK_RX_PORTS] / + [-t DOWNLINK_TX_PORTS ] [-T UPLINK_TX_PORTS] / + [-c DOWNLINK_CORES] [-C UPLINK_CORES] + +where: + +* ``r DOWNLINK_RX_PORTS``: decimal number of downlink receiver ports +* ``R UPLINK_RX_PORTS``: decimal number of uplink receiver ports +* ``t DOWNLINK_TX_PORTS``: decimal number of downlink transceiver ports +* ``T UPLINK_TX_PORTS``: decimal number of uplink transceiver ports +* ``c DOWNLINK_CORES``: hexmask for downlink cores number +* ``C UPLINK_CORES``: hexmask for uplink cores number + +The application requires that baseband devices is capable of performing +the specified baseband operation are available on application initialization. +This means that HW baseband device/s must be bound to a DPDK driver or +a SW baseband device/s (virtual BBdev) must be created (using --vdev). + +To run the application in linuxapp environment with one baseband device for +uplink, running on 1 port and 2 logical cores, issue the command: + +.. code-block:: console + + $ ./build/bbdev --vdev='turbo_sw' -w <NIC0PCIADDR> -c 0x38 --socket-mem=2,2 \ + --file-prefix=bbdev -- -r 1 -t 1 -c 0x30 + +where, NIC0PCIADDR is the PCI addresse of the Rx port + +This command creates one virtual bbdev devices ``turbo_sw`` where the device +gets linked to a corresponding ethernet port as whitelisted by the parameter -w. +3 cores are allocated to the application, and assigned as: + + - core 3 is the master and used to print the stats live on screen, + + - cores 4 & 5 are the downlink cores, Tx & Rx + + +Refer to the *DPDK Getting Started Guide* for general information on running +applications and the Environment Abstraction Layer (EAL) options. + +Using Packet Generator with baseband device sample application +-------------------------------------------------------------- + +To allow the bbdev sample app to do the loopback, an influx of traffic is required. +This can be done by using DPDK Pktgen to burst traffic on two ethernet ports, and +it will print the transmitted along with the looped-back traffic on Rx ports. +Executing the command below will generate traffic on the two whitelisted ethernet +ports. + +.. code-block:: console + + $ ./pktgen-3.4.0/app/x86_64-native-linuxapp-gcc/pktgen -c 0x3 \ + --socket-mem=1,1 --file-prefix=pg -w <NIC1PCIADDR> -- -m 1.0 -P + +where: + +* ``-c COREMASK``: A hexadecimal bitmask of cores to run on +* ``--socket-mem``: Memory to allocate on specific sockets (use comma separated values) +* ``--file-prefix``: Prefix for hugepage filenames +* ``-w <NIC1PCIADDR>``: Add a PCI device in white list. The argument format is <[domain:]bus:devid.func>. +* ``-m <string>``: Matrix for mapping ports to logical cores. +* ``-P``: PROMISCUOUS mode + + +Refer to *The Pktgen Application* documents for general information on running +Pktgen with DPDK applications. diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst index 069d4f1..cc568a9 100644 --- a/doc/guides/sample_app_ug/index.rst +++ b/doc/guides/sample_app_ug/index.rst @@ -77,6 +77,7 @@ Sample Applications User Guides ptpclient performance_thread ipsec_secgw + bbdev_app **Figures** diff --git a/doc/guides/tools/index.rst b/doc/guides/tools/index.rst index c9133ec..a390fe7 100644 --- a/doc/guides/tools/index.rst +++ b/doc/guides/tools/index.rst @@ -41,3 +41,4 @@ DPDK Tools User Guides devbind cryptoperf testeventdev + testbbdev diff --git a/doc/guides/tools/testbbdev.rst b/doc/guides/tools/testbbdev.rst new file mode 100644 index 0000000..d257b22 --- /dev/null +++ b/doc/guides/tools/testbbdev.rst @@ -0,0 +1,546 @@ +.. + BSD LICENSE + + Copyright(c) 2017 Intel Corporation. All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Intel Corporation nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +dpdk-test-bbdev Application +=========================== + +The ``dpdk-test-bbdev`` tool is a Data Plane Development Kit (DPDK) utility that +allows measuring performance parameters of PMDs available in the bbdev framework. +Available tests available for execution are: latency, throughput, validation and +sanity tests. Execution of tests can be customized using various parameters +passed to a python running script. + +Compiling the Application +------------------------- + +**Step 1: PMD setting** + +The ``dpdk-test-bbdev`` tool depends on crypto device drivers PMD which +are disabled by default in the build configuration file ``common_base``. +The bbdevice drivers PMD which should be tested can be enabled by setting + + ``CONFIG_RTE_LIBRTE_PMD_<name>=y`` + +Setting example for (*turbo_sw*) PMD + + ``CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y`` + +**Step 2: Build the application** + +Execute the ``dpdk-setup.sh`` script to build the DPDK library together with the +``dpdk-test-bbdev`` application. + +Initially, the user must select a DPDK target to choose the correct target type +and compiler options to use when building the libraries. +The user must have all libraries, modules, updates and compilers installed +in the system prior to this, as described in the earlier chapters in this +Getting Started Guide. + +Running the Application +----------------------- + +The tool application has a number of command line options: + +.. code-block:: console + + python test-bbdev.py [-h] [-p TESTAPP_PATH] [-e EAL_PARAMS] [-t TIMEOUT] + [-c TEST_CASE [TEST_CASE ...]] + [-v TEST_VECTOR [TEST_VECTOR...]] [-n NUM_OPS] + [-b BURST_SIZE [BURST_SIZE ...]] + +command-line Options +~~~~~~~~~~~~~~~~~~~~ + +The following are the command-line options: + +``-h, --help`` + Shows help message and exit. + +``-p TESTAPP_PATH, --testapp_path TESTAPP_PATH`` + Indicates the path to the bbdev test app. If not specified path is set based + on *$RTE_SDK* environment variable concatenated with "*/build/app/testbbdev*". + +``-e EAL_PARAMS, --eal_params EAL_PARAMS`` + Specifies EAL arguments which are passed to the test app. For more details, + refer to DPDK documentation at http://dpdk.org/doc. + +``-t TIMEOUT, --timeout TIMEOUT`` + Specifies timeout in seconds. If not specified timeout is set to 300 seconds. + +``-c TEST_CASE [TEST_CASE ...], --test_cases TEST_CASE [TEST_CASE ...]`` + Defines test cases to run. If not specified all available tests are run. + + The following tests can be run: + * unittest + Small unit tests witch check basic functionality of bbdev library. + * latency + Test calculates three latency metrics: + * offload_latency_tc + measures the cost of offloading enqueue and dequeue operations. + * offload_latency_empty_q_tc + measures the cost of offloading a dequeue operation from an empty queue. + checks how long last dequeueing if there is no operations to dequeue + * operation_latency_tc + measures the time difference from the first attempt to enqueue till the + first successful dequeue. + * validation + Test do enqueue on given vector and compare output after dequeueing. + * throughput + Test measures the achieved throughput on the available lcores. + Results are printed in million operations per second and million bits per second. + * interrupt + The same test as 'throughput' but uses interrupts instead of PMD to perform + the dequeue. + + **Example usage:** + + ``./test-bbdev.py -c validation`` + Runs validation test suite + + ``./test-bbdev.py -c latency throughput`` + Runs latency and throughput test suites + +``-v TEST_VECTOR [TEST_VECTOR ...], --test_vector TEST_VECTOR [TEST_VECTOR ...]`` + Specifies paths to the test vector files. If not specified path is set based + on *$RTE_SDK* environment variable concatenated with + "*/app/test-bbdev/test_vectors/bbdev_vector_null.data*" and indicates default + data file. + + **Example usage:** + + ``./test-bbdev.py -v app/test-bbdev/test_vectors/bbdev_vector_td_test1.data`` + Fills vector based on bbdev_vector_td_test1.data file and runs all tests + + ``./test-bbdev.py -v bbdev_vector_td_test1.data bbdev_vector_te_test2.data`` + The bbdev test app is executed twice. First time vector is filled based on + *bbdev_vector_td_test1.data* file and second time based on + *bbdev_vector_te_test2.data* file. For both executions all tests are run. + +``-n NUM_OPS, --num_ops NUM_OPS`` + Specifies number of operations to process on device. If not specified num_ops + is set to 32 operations. + +``-v BURST_SIZE [BURST_SIZE ...], --burst-size BURST_SIZE [BURST_SIZE ...]`` + Specifies operations enqueue/dequeue burst size. If not specified burst_size is + set to 32. + + +Parameter globbing +~~~~~~~~~~~~~~~~~~ + +Thanks to the globbing functionality in python test-bbdev.py script allows to +run tests with different set of vector files without giving all of them explicitly. + +**Example usage:** + +.. code-block:: console + + ./test-bbdev.py -v app/test-bbdev/test_vectors/bbdev_vector_*.data + +It runs all tests with following vectors: + +- ``bbdev_vector_null.data`` + +- ``bbdev_vector_te_default.data`` + +- ``bbdev_vector_td_default.data`` + + +.. code-block:: console + + ./test-bbdev.py -v app/test-bbdev/test_vectors/bbdev_vector_t?_default.data + +It runs all tests with "default" vectors: + +- ``bbdev_vector_te_default.data`` + +- ``bbdev_vector_td_default.data`` + + +Running Tests +------------- + +Shortened tree of isg_cid-wireless_dpdk_ae with dpdk compiled for +x86_64-native-linuxapp-icc target: + +:: + + |-- app + |-- test-bbdev + |-- test_vectors + |-- bbdev_vector_null.data + |-- bbdev_vector_te_default.data + |-- bbdev_vector_td_default.data + + |-- x86_64-native-linuxapp-icc + |-- app + |-- testbbdev + +All bbdev devices +~~~~~~~~~~~~~~~~~ + +.. code-block:: console + + ./test-bbdev.py -p ../../x86_64-native-linuxapp-icc/app/testbbdev + -v ./test_vectors/bbdev_vector_td_default.data + +It runs all available tests using the test vector filled based on +*bbdev_vector_td_default.data* file. +By default number of operations to process on device is set to 32, timeout is +set to 300s and operations enqueue/dequeue burst size is set to 32. +Moreover a bbdev (*bbdev_null*) device will be created. + +bbdev turbo_sw device +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: console + + ./test-bbdev.py -p ../../x86_64-native-linuxapp-icc/app/testbbdev + -e="--vdev=turbo_sw" -t 120 -c validation + -v ./test_vectors/bbdev_vector_t?_default.data -n 64 -b 8 32 + +It runs **validation** test for each vector file that matches the given pattern. +Number of operations to process on device is set to 64 and operations timeout is +set to 120s and enqueue/dequeue burst size is set to 8 and to 32. +Moreover a bbdev (*turbo_sw*) device will be created. + + +bbdev null device +~~~~~~~~~~~~~~~~~ + +Executing bbdev null device with *bbdev_vector_null.data* helps in measuring the +overhead introduced by the bbdev framework. + +.. code-block:: console + + ./test-bbdev.py -e="--vdev=bbdev_null0" + -v ./test_vectors/bbdev_vector_null.data + +**Note:** + +bbdev_null device does not have to be defined explicitly as it is created by default. + + + +Test Vector files +================= + +Test Vector files contain the data which is used to set turbo decoder/encoder +parameters and buffers for validation purpose. New test vector files should be +stored in ``app/test-bbdev/test_vectors/`` directory. Detailed description of +the systax of the test vector files is in the following section. + + +Basic principles for test vector files +-------------------------------------- +Line started with ``#`` is treated as a comment and is ignored. + +If variable is a chain of values, values should be separated by a comma. If +assignment is split into several lines, each line (except the last one) has to +be ended with a comma. +There is no comma after last value in last line. Correct assignment should +look like the following: + +.. parsed-literal:: + + variable = + value, value, value, value, + value, value + +In case where variable is a single value correct assignment looks like the +following: + +.. parsed-literal:: + + variable = + value + +Length of chain variable is calculated by parser. Can not be defined +explicitly. + +Variable op_type has to be defined as a first variable in file. It specifies +what type of operations will be executed. For decoder op_type has to be set to +``RTE_BBDEV_OP_TURBO_DEC`` and for encoder to ``RTE_BBDEV_OP_TURBO_ENC``. + +Full details of the meaning and valid values for the below fields are +documented in *rte_bbdev_op.h* + + +Turbo decoder test vectors template +----------------------------------- + +For turbo decoder it has to be always set to ``RTE_BBDEV_OP_TURBO_DEC`` + +.. parsed-literal:: + + op_type = + RTE_BBDEV_OP_TURBO_DEC + +Chain of uint32_t values. Note that it is possible to define more than one +input/output entries which will result in chaining two or more data structures +for *segmented Transport Blocks* + +.. parsed-literal:: + + input0 = + 0x00000000, 0x7f817f00, 0x7f7f8100, 0x817f8100, 0x81008100, 0x7f818100, 0x81817f00, 0x7f818100, + 0x81007f00, 0x7f818100, 0x817f8100, 0x81817f00, 0x81008100, 0x817f7f00, 0x7f7f8100, 0x81817f00 + +Chain of uint32_t values + +.. parsed-literal:: + + input1 = + 0x7f7f0000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, + 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000 + +Chain of uint32_t values + +.. parsed-literal:: + + input2 = + 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, + 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000 + +Chain of uint32_t values + +.. parsed-literal:: + + hard_output0 = + 0xa7d6732e + +Chain of uint32_t values + +.. parsed-literal:: + + hard_output1 = + 0xa61 + +Chain of uint32_t values + +.. parsed-literal:: + + soft_output0 = + 0x817f817f, 0x7f817f7f, 0x81818181, 0x817f7f81, 0x7f818181, 0x8181817f, 0x817f817f, 0x8181817f + +Chain of uint32_t values + +.. parsed-literal:: + + soft_output1 = + 0x817f7f81, 0x7f7f7f81, 0x7f7f8181 + +uint32_t value + +.. parsed-literal:: + + e = + 44 + +uint16_t value + +.. parsed-literal:: + + k = + 40 + +uint8_t value + +.. parsed-literal:: + + rv_index = + 0 + +uint8_t value + +.. parsed-literal:: + + iter_max = + 8 + +uint8_t value + +.. parsed-literal:: + + iter_min = + 4 + +uint8_t value + +.. parsed-literal:: + + expected_iter_count = + 8 + +uint8_t value + +.. parsed-literal:: + + ext_scale = + 15 + +uint8_t value + +.. parsed-literal:: + + num_maps = + 0 + +Chain of flags for turbo decoder operation. Following flags can be used: + +- ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` + +- ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` + +- ``RTE_BBDEV_TURBO_EQUALIZER`` + +- ``RTE_BBDEV_TURBO_SOFT_OUT_SATURATE`` + +- ``RTE_BBDEV_TURBO_HALF_ITERATION_EVEN`` + +- ``RTE_BBDEV_TURBO_CONTINUE_CRC_MATCH`` + +- ``RTE_BBDEV_TURBO_SOFT_OUTPUT`` + +- ``RTE_BBDEV_TURBO_EARLY_TERMINATION`` + +If ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` is set input has to be increased by three +bytes for CRC24B purposes + +.. parsed-literal:: + + op_flags = + RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE, RTE_BBDEV_TURBO_EQUALIZER, + RTE_BBDEV_TURBO_SOFT_OUTPUT + +Chain of operation statuses that are expected after operation is performed. +Following statuses can be used: + +- ``DMA`` + +- ``FCW`` + +- ``CRC`` + +- ``OK`` + +``OK`` means no errors are expected. Cannot be used with other values. + +.. parsed-literal:: + + expected_status = + FCW, CRC + + +Turbo encoder test vectors template +----------------------------------- + +For turbo encoder it has to be always set to ``RTE_BBDEV_OP_TURBO_ENC`` + +.. parsed-literal:: + + op_type = + RTE_BBDEV_OP_TURBO_ENC + +Chain of uint32_t values + +.. parsed-literal:: + + input0 = + 0x11d2bcac, 0x4d + +Chain of uint32_t values + +.. parsed-literal:: + + output0 = + 0xd2399179, 0x640eb999, 0x2cbaf577, 0xaf224ae2, 0x9d139927, 0xe6909b29, + 0xa25b7f47, 0x2aa224ce, 0x79f2 + +uint32_t value + +.. parsed-literal:: + + e = + 272 + +uint16_t value + +.. parsed-literal:: + + k = + 40 + +uint16_t value + +.. parsed-literal:: + + ncb = + 192 + +uint8_t value + +.. parsed-literal:: + + rv_index = + 0 + +Chain of flags for turbo encoder operation. Following flags can be used: + +- ``RTE_BBDEV_TURBO_RV_INDEX_BYPASS`` + +- ``RTE_BBDEV_TURBO_RATE_MATCH`` + +- ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` + +- ``RTE_BBDEV_TURBO_CRC_24A_ATTACH`` + + +.. parsed-literal:: + + op_flags = + RTE_BBDEV_TURBO_RATE_MATCH + +Chain of operation statuses that are expected after operation is performed. +Following statuses can be used: + +- ``DMA`` + +- ``FCW`` + +- ``OK`` + +``OK`` means no errors are expected. Cannot be used with other values. + +.. parsed-literal:: + + expected_status = + OK -- 2.7.4 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [dpdk-dev] [PATCH v2 0/5] Wireless Base Band Device (bbdev) 2017-10-18 2:14 [dpdk-dev] [PATCH v2 1/5] bbdev: librte_bbdev library Amr Mokhtar ` (3 preceding siblings ...) 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 5/5] bbdev: documentation Amr Mokhtar @ 2017-10-18 2:14 ` Amr Mokhtar 4 siblings, 0 replies; 6+ messages in thread From: Amr Mokhtar @ 2017-10-18 2:14 UTC (permalink / raw) To: dev Cc: thomas, anatoly.burakov, pablo.de.lara.guarch, niall.power, chris.macnamara, Amr Mokhtar Hello again, A v2 patch of the Wireless Base Band Device (bbdev) RFC is enclosed. Addressing the feedback received from the community and more crafting for the application interface. --- v2: * Split the functionality of rte_bbdev_configure() into smaller portions -> rte_bbdev_setup_queues() and rte_bbdev_enable_intr() * Split rte_bbdev_enqueue() -> rte_bbdev_enc_enqueue() and rte_bbdev_dec_enqueue() * Split rte_bbdev_dequeue() -> rte_bbdev_enc_dequeue() and rte_bbdev_dec_dequeue() * Removed attached flag until hotplug is properly supported in DPDK * More details on the installation of FlexRAN SDK libraries in accordance with Turbo_sw PMD * Minor build fixes for other targets: bsdapp-gcc, bsdapp-clang and linuxapp-clang. * Better-organized patchwork v1: * Initial release of BBDEV library. * Support Turbo Code FEC with two virtual devices (vdev): - Null Turbo PMD - Turbo_sw PMD * A complete Test suite for Turbo Encode/Decode and None operations * Test Vectors parsing and testing functionality * Sample App for a looped-back bbdev with ethdev * Documentation in rst format for all new components [1] http://dpdk.org/dev/patchwork/patch/29447/ [2] http://dpdk.org/dev/patchwork/patch/29448/ [3] http://dpdk.org/dev/patchwork/patch/29450/ [4] http://dpdk.org/dev/patchwork/patch/29449/ [5] http://dpdk.org/dev/patchwork/patch/29452/ [6] http://dpdk.org/dev/patchwork/patch/29451/ RFC: [1] http://dpdk.org/ml/archives/dev/2017-August/073585.html [2] http://dpdk.org/ml/archives/dev/2017-August/073584.html Amr Mokhtar (5): bbdev: librte_bbdev library bbdev: PMD drivers (null/turbo_sw) bbdev: test applications bbdev: sample app bbdev: documentation MAINTAINERS | 10 + app/Makefile | 4 + app/test-bbdev/Makefile | 53 + app/test-bbdev/main.c | 317 +++ app/test-bbdev/main.h | 144 ++ app/test-bbdev/test-bbdev.py | 132 ++ app/test-bbdev/test_bbdev.c | 1406 +++++++++++++ app/test-bbdev/test_bbdev_perf.c | 2090 ++++++++++++++++++++ app/test-bbdev/test_bbdev_vector.c | 884 +++++++++ app/test-bbdev/test_bbdev_vector.h | 98 + app/test-bbdev/test_vectors/bbdev_vector_null.data | 32 + .../test_vectors/bbdev_vector_td_default.data | 80 + .../test_vectors/bbdev_vector_te_default.data | 60 + config/common_base | 23 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf | 1 + doc/guides/bbdevs/index.rst | 40 + doc/guides/bbdevs/null.rst | 77 + doc/guides/bbdevs/turbo_sw.rst | 169 ++ doc/guides/index.rst | 1 + doc/guides/prog_guide/bbdev.rst | 621 ++++++ doc/guides/prog_guide/index.rst | 1 + doc/guides/rel_notes/release_17_11.rst | 10 + doc/guides/sample_app_ug/bbdev_app.rst | 160 ++ doc/guides/sample_app_ug/index.rst | 1 + doc/guides/tools/index.rst | 1 + doc/guides/tools/testbbdev.rst | 546 +++++ drivers/Makefile | 2 + drivers/bbdev/Makefile | 41 + drivers/bbdev/null/Makefile | 49 + drivers/bbdev/null/bbdev_null.c | 377 ++++ drivers/bbdev/null/rte_pmd_bbdev_null_version.map | 3 + drivers/bbdev/turbo_sw/Makefile | 59 + drivers/bbdev/turbo_sw/bbdev_turbo_software.c | 1235 ++++++++++++ .../bbdev/turbo_sw/bbdev_turbo_software_tables.h | 1344 +++++++++++++ .../turbo_sw/rte_pmd_bbdev_turbo_sw_version.map | 3 + examples/Makefile | 1 + examples/bbdev_app/Makefile | 50 + examples/bbdev_app/main.c | 1396 +++++++++++++ lib/Makefile | 3 + lib/librte_bbdev/Makefile | 56 + lib/librte_bbdev/rte_bbdev.c | 1095 ++++++++++ lib/librte_bbdev/rte_bbdev.h | 741 +++++++ lib/librte_bbdev/rte_bbdev_op.h | 514 +++++ lib/librte_bbdev/rte_bbdev_pci.h | 288 +++ lib/librte_bbdev/rte_bbdev_pmd.h | 223 +++ lib/librte_bbdev/rte_bbdev_vdev.h | 102 + lib/librte_bbdev/rte_bbdev_version.map | 37 + mk/rte.app.mk | 13 + 49 files changed, 14594 insertions(+) create mode 100644 app/test-bbdev/Makefile create mode 100644 app/test-bbdev/main.c create mode 100644 app/test-bbdev/main.h create mode 100755 app/test-bbdev/test-bbdev.py create mode 100644 app/test-bbdev/test_bbdev.c create mode 100644 app/test-bbdev/test_bbdev_perf.c create mode 100644 app/test-bbdev/test_bbdev_vector.c create mode 100644 app/test-bbdev/test_bbdev_vector.h create mode 100644 app/test-bbdev/test_vectors/bbdev_vector_null.data create mode 100644 app/test-bbdev/test_vectors/bbdev_vector_td_default.data create mode 100644 app/test-bbdev/test_vectors/bbdev_vector_te_default.data create mode 100644 doc/guides/bbdevs/index.rst create mode 100644 doc/guides/bbdevs/null.rst create mode 100644 doc/guides/bbdevs/turbo_sw.rst create mode 100644 doc/guides/prog_guide/bbdev.rst create mode 100644 doc/guides/sample_app_ug/bbdev_app.rst create mode 100644 doc/guides/tools/testbbdev.rst create mode 100644 drivers/bbdev/Makefile create mode 100644 drivers/bbdev/null/Makefile create mode 100644 drivers/bbdev/null/bbdev_null.c create mode 100644 drivers/bbdev/null/rte_pmd_bbdev_null_version.map create mode 100644 drivers/bbdev/turbo_sw/Makefile create mode 100644 drivers/bbdev/turbo_sw/bbdev_turbo_software.c create mode 100644 drivers/bbdev/turbo_sw/bbdev_turbo_software_tables.h create mode 100644 drivers/bbdev/turbo_sw/rte_pmd_bbdev_turbo_sw_version.map create mode 100644 examples/bbdev_app/Makefile create mode 100644 examples/bbdev_app/main.c create mode 100644 lib/librte_bbdev/Makefile create mode 100644 lib/librte_bbdev/rte_bbdev.c create mode 100644 lib/librte_bbdev/rte_bbdev.h create mode 100644 lib/librte_bbdev/rte_bbdev_op.h create mode 100644 lib/librte_bbdev/rte_bbdev_pci.h create mode 100644 lib/librte_bbdev/rte_bbdev_pmd.h create mode 100644 lib/librte_bbdev/rte_bbdev_vdev.h create mode 100644 lib/librte_bbdev/rte_bbdev_version.map -- 2.7.4 ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-10-18 2:15 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-10-18 2:14 [dpdk-dev] [PATCH v2 1/5] bbdev: librte_bbdev library Amr Mokhtar 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 2/5] bbdev: PMD drivers (null/turbo_sw) Amr Mokhtar 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 3/5] bbdev: test applications Amr Mokhtar 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 4/5] bbdev: sample app Amr Mokhtar 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 5/5] bbdev: documentation Amr Mokhtar 2017-10-18 2:14 ` [dpdk-dev] [PATCH v2 0/5] Wireless Base Band Device (bbdev) Amr Mokhtar
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).