* [dpdk-dev] [PATCH] BBDEV turbo_sw PMD compilation fix @ 2019-06-05 17:38 Nicolas Chautru 2019-06-05 17:38 ` [dpdk-dev] [PATCH] baseband/turbo_sw: Option to build turbosw PMD without SDK Nicolas Chautru 0 siblings, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-05 17:38 UTC (permalink / raw) To: amr.mokhtar, ferruh.yigit, thomas, akhil.goyal, dev; +Cc: Nicolas Chautru Based on discussion with maintainer, pushing first a patch to help maintenance of the baseband_turbo_sw which had been lacking. The documentation is clarified to point to steps on building the SDK libraries which are now publicly available. (Cosmetic changes to the webpage containing these SDK will happen in parallel based on feedback from maintainer). A compile flag is added to be able to build the turbo_sw PMD when the SDK libraries for AVX2 are installed or not. In both cases this can be compiled with gcc RTE_TARGET. Missing meson build support is also added. Note that additional BBDEV changes pushed in previous v1 https://patches.dpdk.org/project/dpdk/list/?series=4657 will be added in a separate v2 patchset which will depend on this very patchset. Nicolas Chautru (1): baseband/turbo_sw: Option to build turbosw PMD without SDK config/common_base | 3 +- doc/guides/bbdevs/turbo_sw.rst | 339 ++++++++++++----------- drivers/baseband/meson.build | 14 +- drivers/baseband/turbo_sw/Makefile | 13 +- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 39 ++- drivers/baseband/turbo_sw/meson.build | 30 ++ mk/rte.app.mk | 3 + 7 files changed, 268 insertions(+), 173 deletions(-) create mode 100644 drivers/baseband/turbo_sw/meson.build -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH] baseband/turbo_sw: Option to build turbosw PMD without SDK 2019-06-05 17:38 [dpdk-dev] [PATCH] BBDEV turbo_sw PMD compilation fix Nicolas Chautru @ 2019-06-05 17:38 ` Nicolas Chautru 2019-06-05 17:47 ` Thomas Monjalon ` (3 more replies) 0 siblings, 4 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-05 17:38 UTC (permalink / raw) To: amr.mokhtar, ferruh.yigit, thomas, akhil.goyal, dev; +Cc: Nicolas Chautru Adding compile flag to allow to build the turbo_sw PMD without SDK libraries installed with limited capability. Update of documentation for buidling steps. Meson build support. Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 3 +- doc/guides/bbdevs/turbo_sw.rst | 339 ++++++++++++----------- drivers/baseband/meson.build | 14 +- drivers/baseband/turbo_sw/Makefile | 13 +- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 39 ++- drivers/baseband/turbo_sw/meson.build | 30 ++ mk/rte.app.mk | 3 + 7 files changed, 268 insertions(+), 173 deletions(-) create mode 100644 drivers/baseband/turbo_sw/meson.build diff --git a/config/common_base b/config/common_base index 6b96e0e..bc80209 100644 --- a/config/common_base +++ b/config/common_base @@ -523,6 +523,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y CONFIG_RTE_LIBRTE_BBDEV=y CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y +CONFIG_RTE_BBDEV_SDK_AVX2=n # # Compile PMD for NULL bbdev device @@ -532,7 +533,7 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y # # Compile PMD for turbo software bbdev device # -CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n +CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y # # Compile generic crypto device library diff --git a/doc/guides/bbdevs/turbo_sw.rst b/doc/guides/bbdevs/turbo_sw.rst index 29f7ec9..97ff888 100644 --- a/doc/guides/bbdevs/turbo_sw.rst +++ b/doc/guides/bbdevs/turbo_sw.rst @@ -1,158 +1,181 @@ -.. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2017 Intel Corporation - -SW Turbo Poll Mode Driver -========================= - -The SW Turbo PMD (**baseband_turbo_sw**) provides a poll mode bbdev driver that utilizes -Intel optimized libraries for LTE Layer 1 workloads acceleration. This PMD -supports the functions: Turbo FEC, Rate Matching and CRC functions. - -Features --------- - -SW Turbo PMD has support for the following capabilities: - -For the encode operation: - -* ``RTE_BBDEV_TURBO_CRC_24A_ATTACH`` -* ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` -* ``RTE_BBDEV_TURBO_RATE_MATCH`` -* ``RTE_BBDEV_TURBO_RV_INDEX_BYPASS`` - -For the decode operation: - -* ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` -* ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` -* ``RTE_BBDEV_TURBO_POS_LLR_1_BIT_IN`` -* ``RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN`` -* ``RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP`` -* ``RTE_BBDEV_TURBO_EARLY_TERMINATION`` - - -Limitations ------------ - -* In-place operations for Turbo encode and decode are not supported - -Installation ------------- - -FlexRAN SDK Download -~~~~~~~~~~~~~~~~~~~~ - -To build DPDK with the *baseband_turbo_sw* PMD the user is required to download -the export controlled ``FlexRAN SDK`` Libraries. An account at `Intel Resource -Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_ -needs to be registered. - -Once registered, the user needs to log in, and look for -*Intel FlexRAN Software Release Package -18-09* to download or directly through -this `link <https://cdrdv2.intel.com/v1/dl/getContent/605167>`_. - -After download is complete, the user needs to unpack and compile on their -system before building DPDK. - -The following table maps DPDK versions with past FlexRAN SDK releases: - -.. _table_flexran_releases: - -.. table:: DPDK and FlexRAN SDK releases compliance - - ===================== ============================ - DPDK version FlexRAN SDK release - ===================== ============================ - 18.02 1.3.0 - 18.05 1.4.0 - 18.08 1.6.0 - 19.02 18.09 - ===================== ============================ - -FlexRAN SDK Installation -~~~~~~~~~~~~~~~~~~~~~~~~ - -The following are pre-requisites for building FlexRAN SDK Libraries: - (a) An AVX2 supporting machine - (b) CentOS Linux release 7.2.1511 (Core) operating system - (c) Intel ICC 18.0.1 20171018 compiler installed - -The following instructions should be followed in this exact order: - -#. Set the environment variables: - - .. code-block:: console - - source <path-to-icc-compiler-install-folder>/linux/bin/compilervars.sh intel64 -platform linux - -#. Extract the ``605167-flexran-18-09-tar.gz`` package: - - .. code-block:: console - - mkdir FlexRAN-18.09 - tar xvzf 605167-flexran-18-09-tar.gz -C FlexRAN-18.09/ - -#. Run the SDK extractor script and accept the license: - - .. code-block:: console - - cd <path-to-workspace>/FlexRAN-18.09/ - ./SDK-18.09.sh - -#. Generate makefiles based on system configuration: - - .. code-block:: console - - cd <path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ - ./create-makefiles-linux.sh - -#. A build folder is generated in this form ``build-<ISA>-<CC>``, enter that - folder and install: - - .. code-block:: console - - cd build-avx2-icc/ - make && make install - - -Initialization --------------- - -In order to enable this virtual bbdev PMD, the user must: - -* Build the ``FLEXRAN SDK`` libraries (explained in Installation section). - -* Export the environmental variables ``FLEXRAN_SDK`` to the path where the - FlexRAN SDK libraries were installed. And ``DIR_WIRELESS_SDK`` to the path - where the libraries were extracted. - -Example: - -.. code-block:: console - - export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/build-avx2-icc/install - export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ - - -* Set ``CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y`` in DPDK common configuration - file ``config/common_base``. - -To use the PMD in an application, user must: - -- Call ``rte_vdev_init("baseband_turbo_sw")`` within the application. - -- Use ``--vdev="baseband_turbo_sw"`` in the EAL options, which will call ``rte_vdev_init()`` internally. - -The following parameters (all optional) can be provided in the previous two calls: - -* ``socket_id``: Specify the socket where the memory for the device is going to be allocated - (by default, *socket_id* will be the socket where the core that is creating the PMD is running on). - -* ``max_nb_queues``: Specify the maximum number of queues in the device (default is ``RTE_MAX_LCORE``). - -Example: -~~~~~~~~ - -.. code-block:: console - - ./test-bbdev.py -e="--vdev=baseband_turbo_sw,socket_id=0,max_nb_queues=8" \ - -c validation -v ./turbo_*_default.data +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2017 Intel Corporation + +SW FEC Poll Mode Driver +========================= + +The SW FEC PMD (**baseband_turbo_sw**) provides a software only poll mode bbdev +driver that can optionally utilize Intel optimized libraries for LTE and 5GNR +Layer 1 workloads acceleration. + +Note that the driver can also be built without any dependency with reduced +functionality for maintenance purpose. + +To enable linking to the SDK libraries see detailed installation section below. +Two flags can be enabled depending on whether the target machine can support +AVX2 and AVX512 instructions sets and the related SDK libraries for vectorized +signal processing functions are installed : +- CONFIG_RTE_BBDEV_SDK_AVX2 +- CONFIG_RTE_BBDEV_SDK_AVX512 +By default these 2 flags are disabled by default. For AVX2 machine and SDK +library installed then the first flag can be enabled. For AVX512 machine and +SDK library installed then both flags can be enabled for full real time capability. + +This PMD supports the functions: FEC, Rate Matching and CRC functions detailed +in the Features section. + +Features +-------- + +SW FEC PMD can support for the following capabilities when the SDK libraries +are used: + +For the LTE encode operation: + +* ``RTE_BBDEV_TURBO_CRC_24A_ATTACH`` +* ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` +* ``RTE_BBDEV_TURBO_RATE_MATCH`` +* ``RTE_BBDEV_TURBO_RV_INDEX_BYPASS`` + +For the LTE decode operation: + +* ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` +* ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` +* ``RTE_BBDEV_TURBO_POS_LLR_1_BIT_IN`` +* ``RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN`` +* ``RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP`` +* ``RTE_BBDEV_TURBO_EARLY_TERMINATION`` + +For the 5G NR LDPC encode operation: + +* ``RTE_BBDEV_LDPC_RATE_MATCH`` +* ``RTE_BBDEV_LDPC_CRC_24A_ATTACH`` +* ``RTE_BBDEV_LDPC_CRC_24B_ATTACH`` + +For the 5G NR LDPC decode operation: + +* ``RTE_BBDEV_LDPC_CRC_TYPE_24B_CHECK`` +* ``RTE_BBDEV_LDPC_CRC_TYPE_24A_CHECK`` +* ``RTE_BBDEV_LDPC_CRC_TYPE_24B_DROP`` +* ``RTE_BBDEV_LDPC_HQ_COMBINE_IN_ENABLE`` +* ``RTE_BBDEV_LDPC_HQ_COMBINE_OUT_ENABLE`` +* ``RTE_BBDEV_LDPC_ITERATION_STOP_ENABLE`` + +Limitations +----------- + +* In-place operations for encode and decode are not supported + +Installation +------------ + +FlexRAN FEC SDK Download +~~~~~~~~~~~~~~~~~~~~ + +As an option it is possible to link this driver with FleXRAN SDK libraries +which can enable real time signal processing using AVX instructions. + +These libraries are available through this link `link <https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules>`_. + +After download is complete, the user needs to unpack and compile on their +system before building DPDK. + +The following table maps DPDK versions with past FlexRAN SDK releases: + +.. _table_flexran_releases: + +.. table:: DPDK and FlexRAN FEC SDK releases compliance + + ===================== ============================ + DPDK version FlexRAN FEC SDK release + ===================== ============================ + 19.08 19.04 + ===================== ============================ + +FlexRAN SDK Installation +~~~~~~~~~~~~~~~~~~~~~~~~ + +Note that the installation of these libraries is optional. + +The following are pre-requisites for building FlexRAN SDK Libraries: + (a) An AVX2 or AVX512 supporting machine + (b) CentOS Linux release 7.2.1511 (Core) operating system is advised + (c) Intel ICC 18.0.1 20171018 compiler or more recent and related libraries + ICC is available with a free community license `link <https://software.intel.com/en-us/system-studio/choose-download#technical>`_. + +The following instructions should be followed in this exact order: + +#. Set the environment variables: + + .. code-block:: console + + source <path-to-icc-compiler-install-folder>/linux/bin/compilervars.sh intel64 -platform linux + +#. Run the SDK extractor script and accept the license: + + .. code-block:: console + + cd <path-to-workspace> + ./FlexRAN-FEC-SDK-19-04.sh + +#. Generate makefiles based on system configuration: + + .. code-block:: console + + cd <path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/ + ./create-makefiles-linux.sh + +#. A build folder is generated in this form ``build-<ISA>-<CC>``, enter that + folder and install: + + .. code-block:: console + + cd build-avx512-icc/ + make && make install + +Initialization +-------------- + +In order to enable this virtual bbdev PMD, the user may: + +* Build the ``FLEXRAN SDK`` libraries (explained in Installation section). + +* Export the environmental variables ``FLEXRAN_SDK`` to the path where the + FlexRAN SDK libraries were installed. And ``DIR_WIRELESS_SDK`` to the path + where the libraries were extracted. + +Example: + +.. code-block:: console + + export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/install + export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/ + +* Set ``CONFIG_RTE_BBDEV_SDK_AVX2=y`` and ``CONFIG_RTE_BBDEV_SDK_AVX512=y`` + in DPDK common configuration file ``config/common_base`` to be able to use + the SDK libraries as mentioned above. + For AVX2 machine it is possible to only enable CONFIG_RTE_BBDEV_SDK_AVX2 + for limited 4G functionality. + If no flag are set the PMD driver will still build but its capabilities + will be limited accordingly. + +To use the PMD in an application, user must: + +- Call ``rte_vdev_init("baseband_turbo_sw")`` within the application. + +- Use ``--vdev="baseband_turbo_sw"`` in the EAL options, which will call ``rte_vdev_init()`` internally. + +The following parameters (all optional) can be provided in the previous two calls: + +* ``socket_id``: Specify the socket where the memory for the device is going to be allocated + (by default, *socket_id* will be the socket where the core that is creating the PMD is running on). + +* ``max_nb_queues``: Specify the maximum number of queues in the device (default is ``RTE_MAX_LCORE``). + +Example: +~~~~~~~~ + +.. code-block:: console + + ./test-bbdev.py -e="--vdev=baseband_turbo_sw,socket_id=0,max_nb_queues=8" \ + -c validation -v ./turbo_*_default.data diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build index 52489df..d2641d6 100644 --- a/drivers/baseband/meson.build +++ b/drivers/baseband/meson.build @@ -1,7 +1,7 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2018 Luca Boccassi <bluca@debian.org> - -drivers = ['null'] - -config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' -driver_name_fmt = 'rte_pmd_@0@' +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2018 Luca Boccassi <bluca@debian.org> + +drivers = ['null', 'turbo_sw'] + +config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' +driver_name_fmt = 'rte_pmd_@0@' diff --git a/drivers/baseband/turbo_sw/Makefile b/drivers/baseband/turbo_sw/Makefile index d364677..414d0d9 100644 --- a/drivers/baseband/turbo_sw/Makefile +++ b/drivers/baseband/turbo_sw/Makefile @@ -3,9 +3,6 @@ include $(RTE_SDK)/mk/rte.vars.mk -ifeq ($(FLEXRAN_SDK),) -$(error "Please define FLEXRAN_SDK environment variable") -endif # library name LIB = librte_pmd_bbdev_turbo_sw.a @@ -21,17 +18,21 @@ LDLIBS += -lrte_bus_vdev # versioning export map EXPORT_MAP := rte_pmd_bbdev_turbo_sw_version.map -# external library dependencies +# external library dependencies if available +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +ifeq ($(FLEXRAN_SDK),) +$(error "Please define FLEXRAN_SDK environment variable") +endif CFLAGS += -I$(FLEXRAN_SDK)/lib_common CFLAGS += -I$(FLEXRAN_SDK)/lib_turbo CFLAGS += -I$(FLEXRAN_SDK)/lib_crc CFLAGS += -I$(FLEXRAN_SDK)/lib_rate_matching - LDLIBS += -L$(FLEXRAN_SDK)/lib_turbo -lturbo LDLIBS += -L$(FLEXRAN_SDK)/lib_crc -lcrc LDLIBS += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching LDLIBS += -L$(FLEXRAN_SDK)/lib_common -lcommon -LDLIBS += -lstdc++ -lirc -limf -lipps +LDLIBS += -lstdc++ -lirc -limf -lipps -lsvml +endif # library version LIBABIVER := 1 diff --git a/drivers/baseband/turbo_sw/bbdev_turbo_software.c b/drivers/baseband/turbo_sw/bbdev_turbo_software.c index 5204a77..4781de0 100644 --- a/drivers/baseband/turbo_sw/bbdev_turbo_software.c +++ b/drivers/baseband/turbo_sw/bbdev_turbo_software.c @@ -14,10 +14,11 @@ #include <rte_bbdev.h> #include <rte_bbdev_pmd.h> +#ifdef RTE_BBDEV_SDK_AVX2 #include <phy_turbo.h> #include <phy_crc.h> #include <phy_rate_match.h> -#include <divide.h> +#endif #define DRIVER_NAME baseband_turbo_sw @@ -143,6 +144,7 @@ struct turbo_sw_queue { struct bbdev_private *internals = dev->data->dev_private; static const struct rte_bbdev_op_cap bbdev_capabilities[] = { +#ifdef RTE_BBDEV_SDK_AVX2 { .type = RTE_BBDEV_OP_TURBO_DEC, .cap.turbo_dec = { @@ -172,6 +174,7 @@ struct turbo_sw_queue { .num_buffers_dst = RTE_BBDEV_MAX_CODE_BLOCKS, } }, +#endif RTE_BBDEV_END_OF_CAPABILITIES_LIST() }; @@ -472,6 +475,7 @@ struct turbo_sw_queue { struct rte_mbuf *m_out, uint16_t in_offset, uint16_t out_offset, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int16_t k_idx; uint16_t m; @@ -724,6 +728,22 @@ struct turbo_sw_queue { } *tmp_out = 0; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(r); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(ncb); + RTE_SET_USED(e); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void @@ -856,6 +876,7 @@ struct turbo_sw_queue { uint16_t crc24_overlap, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int32_t k_idx; int32_t iter_cnt; @@ -972,6 +993,22 @@ struct turbo_sw_queue { rte_bbdev_log(ERR, "Turbo Decoder failed"); return; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(kw); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(check_crc_24b); + RTE_SET_USED(crc24_overlap); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void diff --git a/drivers/baseband/turbo_sw/meson.build b/drivers/baseband/turbo_sw/meson.build new file mode 100644 index 0000000..9b4fe34 --- /dev/null +++ b/drivers/baseband/turbo_sw/meson.build @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +path = get_option('flexran_sdk') + +if dpdk_conf.has('RTE_BBDEV_SDK_AVX2') + lib = cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: false) + if not lib.found() + build = false + else + ext_deps += cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: true) + ext_deps += cc.find_library('libcrc', dirs: [path + '/lib_crc'], required: true) + ext_deps += cc.find_library('librate_matching', dirs: [path + '/lib_rate_matching'], required: true) + ext_deps += cc.find_library('libcommon', dirs: [path + '/lib_common'], required: true) + ext_deps += cc.find_library('libstdc++', required: true) + ext_deps += cc.find_library('libirc', required: true) + ext_deps += cc.find_library('libimf', required: true) + ext_deps += cc.find_library('libipps', required: true) + ext_deps += cc.find_library('libsvml', required: true) + includes += include_directories(path + '/lib_turbo') + includes += include_directories(path + '/lib_crc') + includes += include_directories(path + '/lib_rate_matching') + includes += include_directories(path + '/lib_common') + endif +endif + +deps += ['bbdev', 'bus_vdev', 'ring'] +name = 'bbdev_turbo_sw' +allow_experimental_apis = true +sources = files('bbdev_turbo_software.c') diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 7c9b4b5..878cc31 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -221,11 +221,14 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +# Dependency on the FLEXRAN SDK library if available _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_crc -lcrc _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_turbo -lturbo _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_common -lcommon _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lirc -limf -lstdc++ -lipps +endif # CONFIG_RTE_BBDEV_SDK_AVX2 endif # CONFIG_RTE_LIBRTE_BBDEV ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y) -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [dpdk-dev] [PATCH] baseband/turbo_sw: Option to build turbosw PMD without SDK 2019-06-05 17:38 ` [dpdk-dev] [PATCH] baseband/turbo_sw: Option to build turbosw PMD without SDK Nicolas Chautru @ 2019-06-05 17:47 ` Thomas Monjalon 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru ` (2 subsequent siblings) 3 siblings, 0 replies; 62+ messages in thread From: Thomas Monjalon @ 2019-06-05 17:47 UTC (permalink / raw) To: Nicolas Chautru; +Cc: amr.mokhtar, ferruh.yigit, akhil.goyal, dev 05/06/2019 19:38, Nicolas Chautru: > Adding compile flag to allow to build the turbo_sw PMD > without SDK libraries installed with limited capability. > Update of documentation for buidling steps. > Meson build support. There are 3 different things which deserve 3 patches. About the doc, I am surprised it looks to be fully rewritten. I think you need to double check it. ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v2 0/3] BBDEV turbo_sw PMD compilation fix 2019-06-05 17:38 ` [dpdk-dev] [PATCH] baseband/turbo_sw: Option to build turbosw PMD without SDK Nicolas Chautru 2019-06-05 17:47 ` Thomas Monjalon @ 2019-06-05 20:06 ` Nicolas Chautru 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 1/3] version: 19.05-rc4 Nicolas Chautru ` (2 more replies) 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 0/3] baseband/fpga_lte_fec: adding driver for FEC on FPGA Nicolas Chautru 3 siblings, 3 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-05 20:06 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, Nicolas Chautru Updated in v2 : Split into 3 patches as recommended. Based on discussion with maintainer, pushing first a patch to help maintenance of the baseband_turbo_sw which had been lacking. The documentation is clarified to point to steps on building the SDK libraries which are now publicly available. (Cosmetic changes to the webpage containing these SDK will happen in parallel based on feedback from maintainer). A compile flag is added to be able to build the turbo_sw PMD when the SDK libraries for AVX2 are installed or not. In both cases this can be compiled with gcc RTE_TARGET. Missing meson build support is also added. Note that additional BBDEV changes pushed in previous v1 https://patches.dpdk.org/project/dpdk/list/?series=4657 will be added in a separate v2 patchset which will depend on this very patchset. Nicolas Chautru (2): baseband/turbo_sw: option to build turbosw PMD without SDK docs/guides: updating turbo_sw building steps Thomas Monjalon (1): version: 19.05-rc4 VERSION | 2 +- config/common_base | 3 +- doc/guides/bbdevs/turbo_sw.rst | 84 +++++++++++++----------- drivers/baseband/turbo_sw/Makefile | 13 ++-- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 39 ++++++++++- mk/rte.app.mk | 3 + 6 files changed, 97 insertions(+), 47 deletions(-) -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v2 1/3] version: 19.05-rc4 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru @ 2019-06-05 20:06 ` Nicolas Chautru 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 2/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 3/3] docs/guides: updating turbo_sw building steps Nicolas Chautru 2 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-05 20:06 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit From: Thomas Monjalon <thomas@monjalon.net> Signed-off-by: Thomas Monjalon <thomas@monjalon.net> --- VERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/VERSION b/VERSION index 5cfba53..746b4f3 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -19.05.0-rc3 +19.05.0-rc4 -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v2 2/3] baseband/turbo_sw: option to build turbosw PMD without SDK 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 1/3] version: 19.05-rc4 Nicolas Chautru @ 2019-06-05 20:06 ` Nicolas Chautru 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 3/3] docs/guides: updating turbo_sw building steps Nicolas Chautru 2 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-05 20:06 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, Nicolas Chautru Adding compile flag to allow to build the turbo_sw PMD without dependency to have the SDK libraries installed. Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 3 +- drivers/baseband/turbo_sw/Makefile | 13 ++++---- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 39 +++++++++++++++++++++++- mk/rte.app.mk | 3 ++ 4 files changed, 50 insertions(+), 8 deletions(-) diff --git a/config/common_base b/config/common_base index 6b96e0e..bc80209 100644 --- a/config/common_base +++ b/config/common_base @@ -523,6 +523,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y CONFIG_RTE_LIBRTE_BBDEV=y CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y +CONFIG_RTE_BBDEV_SDK_AVX2=n # # Compile PMD for NULL bbdev device @@ -532,7 +533,7 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y # # Compile PMD for turbo software bbdev device # -CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n +CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y # # Compile generic crypto device library diff --git a/drivers/baseband/turbo_sw/Makefile b/drivers/baseband/turbo_sw/Makefile index d364677..414d0d9 100644 --- a/drivers/baseband/turbo_sw/Makefile +++ b/drivers/baseband/turbo_sw/Makefile @@ -3,9 +3,6 @@ include $(RTE_SDK)/mk/rte.vars.mk -ifeq ($(FLEXRAN_SDK),) -$(error "Please define FLEXRAN_SDK environment variable") -endif # library name LIB = librte_pmd_bbdev_turbo_sw.a @@ -21,17 +18,21 @@ LDLIBS += -lrte_bus_vdev # versioning export map EXPORT_MAP := rte_pmd_bbdev_turbo_sw_version.map -# external library dependencies +# external library dependencies if available +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +ifeq ($(FLEXRAN_SDK),) +$(error "Please define FLEXRAN_SDK environment variable") +endif CFLAGS += -I$(FLEXRAN_SDK)/lib_common CFLAGS += -I$(FLEXRAN_SDK)/lib_turbo CFLAGS += -I$(FLEXRAN_SDK)/lib_crc CFLAGS += -I$(FLEXRAN_SDK)/lib_rate_matching - LDLIBS += -L$(FLEXRAN_SDK)/lib_turbo -lturbo LDLIBS += -L$(FLEXRAN_SDK)/lib_crc -lcrc LDLIBS += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching LDLIBS += -L$(FLEXRAN_SDK)/lib_common -lcommon -LDLIBS += -lstdc++ -lirc -limf -lipps +LDLIBS += -lstdc++ -lirc -limf -lipps -lsvml +endif # library version LIBABIVER := 1 diff --git a/drivers/baseband/turbo_sw/bbdev_turbo_software.c b/drivers/baseband/turbo_sw/bbdev_turbo_software.c index 5204a77..4781de0 100644 --- a/drivers/baseband/turbo_sw/bbdev_turbo_software.c +++ b/drivers/baseband/turbo_sw/bbdev_turbo_software.c @@ -14,10 +14,11 @@ #include <rte_bbdev.h> #include <rte_bbdev_pmd.h> +#ifdef RTE_BBDEV_SDK_AVX2 #include <phy_turbo.h> #include <phy_crc.h> #include <phy_rate_match.h> -#include <divide.h> +#endif #define DRIVER_NAME baseband_turbo_sw @@ -143,6 +144,7 @@ struct turbo_sw_queue { struct bbdev_private *internals = dev->data->dev_private; static const struct rte_bbdev_op_cap bbdev_capabilities[] = { +#ifdef RTE_BBDEV_SDK_AVX2 { .type = RTE_BBDEV_OP_TURBO_DEC, .cap.turbo_dec = { @@ -172,6 +174,7 @@ struct turbo_sw_queue { .num_buffers_dst = RTE_BBDEV_MAX_CODE_BLOCKS, } }, +#endif RTE_BBDEV_END_OF_CAPABILITIES_LIST() }; @@ -472,6 +475,7 @@ struct turbo_sw_queue { struct rte_mbuf *m_out, uint16_t in_offset, uint16_t out_offset, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int16_t k_idx; uint16_t m; @@ -724,6 +728,22 @@ struct turbo_sw_queue { } *tmp_out = 0; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(r); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(ncb); + RTE_SET_USED(e); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void @@ -856,6 +876,7 @@ struct turbo_sw_queue { uint16_t crc24_overlap, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int32_t k_idx; int32_t iter_cnt; @@ -972,6 +993,22 @@ struct turbo_sw_queue { rte_bbdev_log(ERR, "Turbo Decoder failed"); return; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(kw); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(check_crc_24b); + RTE_SET_USED(crc24_overlap); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 7c9b4b5..878cc31 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -221,11 +221,14 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +# Dependency on the FLEXRAN SDK library if available _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_crc -lcrc _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_turbo -lturbo _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_common -lcommon _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lirc -limf -lstdc++ -lipps +endif # CONFIG_RTE_BBDEV_SDK_AVX2 endif # CONFIG_RTE_LIBRTE_BBDEV ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y) -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v2 3/3] docs/guides: updating turbo_sw building steps 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 1/3] version: 19.05-rc4 Nicolas Chautru 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 2/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru @ 2019-06-05 20:06 ` Nicolas Chautru 2 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-05 20:06 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, Nicolas Chautru The documentation is clarified to point to steps on building the SDK libraries which are now publicly available. Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- doc/guides/bbdevs/turbo_sw.rst | 84 +++++++++++++++++++++++------------------- 1 file changed, 46 insertions(+), 38 deletions(-) diff --git a/doc/guides/bbdevs/turbo_sw.rst b/doc/guides/bbdevs/turbo_sw.rst index 29f7ec9..196b3d6 100644 --- a/doc/guides/bbdevs/turbo_sw.rst +++ b/doc/guides/bbdevs/turbo_sw.rst @@ -1,26 +1,43 @@ .. SPDX-License-Identifier: BSD-3-Clause Copyright(c) 2017 Intel Corporation -SW Turbo Poll Mode Driver +SW FEC Poll Mode Driver ========================= -The SW Turbo PMD (**baseband_turbo_sw**) provides a poll mode bbdev driver that utilizes -Intel optimized libraries for LTE Layer 1 workloads acceleration. This PMD -supports the functions: Turbo FEC, Rate Matching and CRC functions. +The SW FEC PMD (**baseband_turbo_sw**) provides a software only poll mode bbdev +driver that can optionally utilize Intel optimized libraries for LTE and 5GNR +Layer 1 workloads acceleration. + +Note that the driver can also be built without any dependency with reduced +functionality for maintenance purpose. + +To enable linking to the SDK libraries see detailed installation section below. +Two flags can be enabled depending on whether the target machine can support +AVX2 and AVX512 instructions sets and the related SDK libraries for vectorized +signal processing functions are installed : +- CONFIG_RTE_BBDEV_SDK_AVX2 +- CONFIG_RTE_BBDEV_SDK_AVX512 +By default these 2 flags are disabled by default. For AVX2 machine and SDK +library installed then the first flag can be enabled. For AVX512 machine and +SDK library installed then both flags can be enabled for full real time capability. + +This PMD supports the functions: FEC, Rate Matching and CRC functions detailed +in the Features section. Features -------- -SW Turbo PMD has support for the following capabilities: +SW FEC PMD can support for the following capabilities when the SDK libraries +are used: -For the encode operation: +For the LTE encode operation: * ``RTE_BBDEV_TURBO_CRC_24A_ATTACH`` * ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` * ``RTE_BBDEV_TURBO_RATE_MATCH`` * ``RTE_BBDEV_TURBO_RV_INDEX_BYPASS`` -For the decode operation: +For the LTE decode operation: * ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` * ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` @@ -41,14 +58,10 @@ Installation FlexRAN SDK Download ~~~~~~~~~~~~~~~~~~~~ -To build DPDK with the *baseband_turbo_sw* PMD the user is required to download -the export controlled ``FlexRAN SDK`` Libraries. An account at `Intel Resource -Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_ -needs to be registered. +As an option it is possible to link this driver with FleXRAN SDK libraries +which can enable real time signal processing using AVX instructions. -Once registered, the user needs to log in, and look for -*Intel FlexRAN Software Release Package -18-09* to download or directly through -this `link <https://cdrdv2.intel.com/v1/dl/getContent/605167>`_. +These libraries are available through this link `link <https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules>`_. After download is complete, the user needs to unpack and compile on their system before building DPDK. @@ -57,24 +70,24 @@ The following table maps DPDK versions with past FlexRAN SDK releases: .. _table_flexran_releases: -.. table:: DPDK and FlexRAN SDK releases compliance +.. table:: DPDK and FlexRAN FEC SDK releases compliance ===================== ============================ - DPDK version FlexRAN SDK release + DPDK version FlexRAN FEC SDK release ===================== ============================ - 18.02 1.3.0 - 18.05 1.4.0 - 18.08 1.6.0 - 19.02 18.09 + 19.08 19.04 ===================== ============================ FlexRAN SDK Installation ~~~~~~~~~~~~~~~~~~~~~~~~ +Note that the installation of these libraries is optional. + The following are pre-requisites for building FlexRAN SDK Libraries: - (a) An AVX2 supporting machine - (b) CentOS Linux release 7.2.1511 (Core) operating system - (c) Intel ICC 18.0.1 20171018 compiler installed + (a) An AVX2 or AVX512 supporting machine + (b) CentOS Linux release 7.2.1511 (Core) operating system is advised + (c) Intel ICC 18.0.1 20171018 compiler or more recent and related libraries + ICC is available with a free community license `link <https://software.intel.com/en-us/system-studio/choose-download#technical>`_. The following instructions should be followed in this exact order: @@ -84,25 +97,18 @@ The following instructions should be followed in this exact order: source <path-to-icc-compiler-install-folder>/linux/bin/compilervars.sh intel64 -platform linux -#. Extract the ``605167-flexran-18-09-tar.gz`` package: - - .. code-block:: console - - mkdir FlexRAN-18.09 - tar xvzf 605167-flexran-18-09-tar.gz -C FlexRAN-18.09/ - #. Run the SDK extractor script and accept the license: .. code-block:: console - cd <path-to-workspace>/FlexRAN-18.09/ - ./SDK-18.09.sh + cd <path-to-workspace> + ./FlexRAN-FEC-SDK-19-04.sh #. Generate makefiles based on system configuration: .. code-block:: console - cd <path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ + cd <path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/ ./create-makefiles-linux.sh #. A build folder is generated in this form ``build-<ISA>-<CC>``, enter that @@ -129,12 +135,14 @@ Example: .. code-block:: console - export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/build-avx2-icc/install - export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ - + export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/install + export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/ -* Set ``CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y`` in DPDK common configuration - file ``config/common_base``. +* Set ``CONFIG_RTE_BBDEV_SDK_AVX2=y`` + in DPDK common configuration file ``config/common_base`` to be able to use + the SDK libraries as mentioned above. + If no flag are set the PMD driver will still build but its capabilities + will be limited accordingly. To use the PMD in an application, user must: -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v2 0/3] BBDEV turbo_sw PMD compilation fix 2019-06-05 17:38 ` [dpdk-dev] [PATCH] baseband/turbo_sw: Option to build turbosw PMD without SDK Nicolas Chautru 2019-06-05 17:47 ` Thomas Monjalon 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru @ 2019-06-05 20:10 ` Nicolas Chautru 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru ` (2 more replies) 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 0/3] baseband/fpga_lte_fec: adding driver for FEC on FPGA Nicolas Chautru 3 siblings, 3 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-05 20:10 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Update v2 : Splitting into 3 patches as recommended (ignore previous v2 which had a typo) Based on discussion with maintainer, pushing first a patch to help maintenance of the baseband_turbo_sw which had been lacking. The documentation is clarified to point to steps on building the SDK libraries which are now publicly available. (Cosmetic changes to the webpage containing these SDK will happen in parallel based on feedback from maintainer). A compile flag is added to be able to build the turbo_sw PMD when the SDK libraries for AVX2 are installed or not. In both cases this can be compiled with gcc RTE_TARGET. Missing meson build support is also added. Note that additional BBDEV changes pushed in previous v1 https://patches.dpdk.org/project/dpdk/list/?series=4657 will be added in a separate v2 patchset which will depend on this very patchset. Nicolas Chautru (3): baseband/turbo_sw: option to build turbosw PMD without SDK docs/guides: updating turbo_sw building steps baseband/turbo_sw: meson build support for PMD driver config/common_base | 3 +- doc/guides/bbdevs/turbo_sw.rst | 84 +++++++++++++----------- drivers/baseband/meson.build | 2 +- drivers/baseband/turbo_sw/Makefile | 13 ++-- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 39 ++++++++++- drivers/baseband/turbo_sw/meson.build | 30 +++++++++ mk/rte.app.mk | 3 + 7 files changed, 127 insertions(+), 47 deletions(-) create mode 100644 drivers/baseband/turbo_sw/meson.build -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v2 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru @ 2019-06-05 20:10 ` Nicolas Chautru 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-05 20:10 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Adding compile flag to allow to build the turbo_sw PMD without dependency to have the SDK libraries installed. Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 3 +- drivers/baseband/turbo_sw/Makefile | 13 ++++---- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 39 +++++++++++++++++++++++- mk/rte.app.mk | 3 ++ 4 files changed, 50 insertions(+), 8 deletions(-) diff --git a/config/common_base b/config/common_base index 6b96e0e..bc80209 100644 --- a/config/common_base +++ b/config/common_base @@ -523,6 +523,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y CONFIG_RTE_LIBRTE_BBDEV=y CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y +CONFIG_RTE_BBDEV_SDK_AVX2=n # # Compile PMD for NULL bbdev device @@ -532,7 +533,7 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y # # Compile PMD for turbo software bbdev device # -CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n +CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y # # Compile generic crypto device library diff --git a/drivers/baseband/turbo_sw/Makefile b/drivers/baseband/turbo_sw/Makefile index d364677..414d0d9 100644 --- a/drivers/baseband/turbo_sw/Makefile +++ b/drivers/baseband/turbo_sw/Makefile @@ -3,9 +3,6 @@ include $(RTE_SDK)/mk/rte.vars.mk -ifeq ($(FLEXRAN_SDK),) -$(error "Please define FLEXRAN_SDK environment variable") -endif # library name LIB = librte_pmd_bbdev_turbo_sw.a @@ -21,17 +18,21 @@ LDLIBS += -lrte_bus_vdev # versioning export map EXPORT_MAP := rte_pmd_bbdev_turbo_sw_version.map -# external library dependencies +# external library dependencies if available +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +ifeq ($(FLEXRAN_SDK),) +$(error "Please define FLEXRAN_SDK environment variable") +endif CFLAGS += -I$(FLEXRAN_SDK)/lib_common CFLAGS += -I$(FLEXRAN_SDK)/lib_turbo CFLAGS += -I$(FLEXRAN_SDK)/lib_crc CFLAGS += -I$(FLEXRAN_SDK)/lib_rate_matching - LDLIBS += -L$(FLEXRAN_SDK)/lib_turbo -lturbo LDLIBS += -L$(FLEXRAN_SDK)/lib_crc -lcrc LDLIBS += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching LDLIBS += -L$(FLEXRAN_SDK)/lib_common -lcommon -LDLIBS += -lstdc++ -lirc -limf -lipps +LDLIBS += -lstdc++ -lirc -limf -lipps -lsvml +endif # library version LIBABIVER := 1 diff --git a/drivers/baseband/turbo_sw/bbdev_turbo_software.c b/drivers/baseband/turbo_sw/bbdev_turbo_software.c index 5204a77..4781de0 100644 --- a/drivers/baseband/turbo_sw/bbdev_turbo_software.c +++ b/drivers/baseband/turbo_sw/bbdev_turbo_software.c @@ -14,10 +14,11 @@ #include <rte_bbdev.h> #include <rte_bbdev_pmd.h> +#ifdef RTE_BBDEV_SDK_AVX2 #include <phy_turbo.h> #include <phy_crc.h> #include <phy_rate_match.h> -#include <divide.h> +#endif #define DRIVER_NAME baseband_turbo_sw @@ -143,6 +144,7 @@ struct turbo_sw_queue { struct bbdev_private *internals = dev->data->dev_private; static const struct rte_bbdev_op_cap bbdev_capabilities[] = { +#ifdef RTE_BBDEV_SDK_AVX2 { .type = RTE_BBDEV_OP_TURBO_DEC, .cap.turbo_dec = { @@ -172,6 +174,7 @@ struct turbo_sw_queue { .num_buffers_dst = RTE_BBDEV_MAX_CODE_BLOCKS, } }, +#endif RTE_BBDEV_END_OF_CAPABILITIES_LIST() }; @@ -472,6 +475,7 @@ struct turbo_sw_queue { struct rte_mbuf *m_out, uint16_t in_offset, uint16_t out_offset, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int16_t k_idx; uint16_t m; @@ -724,6 +728,22 @@ struct turbo_sw_queue { } *tmp_out = 0; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(r); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(ncb); + RTE_SET_USED(e); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void @@ -856,6 +876,7 @@ struct turbo_sw_queue { uint16_t crc24_overlap, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int32_t k_idx; int32_t iter_cnt; @@ -972,6 +993,22 @@ struct turbo_sw_queue { rte_bbdev_log(ERR, "Turbo Decoder failed"); return; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(kw); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(check_crc_24b); + RTE_SET_USED(crc24_overlap); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 7c9b4b5..878cc31 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -221,11 +221,14 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +# Dependency on the FLEXRAN SDK library if available _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_crc -lcrc _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_turbo -lturbo _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_common -lcommon _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lirc -limf -lstdc++ -lipps +endif # CONFIG_RTE_BBDEV_SDK_AVX2 endif # CONFIG_RTE_LIBRTE_BBDEV ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y) -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v2 2/3] docs/guides: updating turbo_sw building steps 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru @ 2019-06-05 20:10 ` Nicolas Chautru 2019-06-06 10:34 ` Ferruh Yigit 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2 siblings, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-05 20:10 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru The documentation is clarified to point to steps on building the SDK libraries which are now publicly available. Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- doc/guides/bbdevs/turbo_sw.rst | 84 +++++++++++++++++++++++------------------- 1 file changed, 46 insertions(+), 38 deletions(-) diff --git a/doc/guides/bbdevs/turbo_sw.rst b/doc/guides/bbdevs/turbo_sw.rst index 29f7ec9..196b3d6 100644 --- a/doc/guides/bbdevs/turbo_sw.rst +++ b/doc/guides/bbdevs/turbo_sw.rst @@ -1,26 +1,43 @@ .. SPDX-License-Identifier: BSD-3-Clause Copyright(c) 2017 Intel Corporation -SW Turbo Poll Mode Driver +SW FEC Poll Mode Driver ========================= -The SW Turbo PMD (**baseband_turbo_sw**) provides a poll mode bbdev driver that utilizes -Intel optimized libraries for LTE Layer 1 workloads acceleration. This PMD -supports the functions: Turbo FEC, Rate Matching and CRC functions. +The SW FEC PMD (**baseband_turbo_sw**) provides a software only poll mode bbdev +driver that can optionally utilize Intel optimized libraries for LTE and 5GNR +Layer 1 workloads acceleration. + +Note that the driver can also be built without any dependency with reduced +functionality for maintenance purpose. + +To enable linking to the SDK libraries see detailed installation section below. +Two flags can be enabled depending on whether the target machine can support +AVX2 and AVX512 instructions sets and the related SDK libraries for vectorized +signal processing functions are installed : +- CONFIG_RTE_BBDEV_SDK_AVX2 +- CONFIG_RTE_BBDEV_SDK_AVX512 +By default these 2 flags are disabled by default. For AVX2 machine and SDK +library installed then the first flag can be enabled. For AVX512 machine and +SDK library installed then both flags can be enabled for full real time capability. + +This PMD supports the functions: FEC, Rate Matching and CRC functions detailed +in the Features section. Features -------- -SW Turbo PMD has support for the following capabilities: +SW FEC PMD can support for the following capabilities when the SDK libraries +are used: -For the encode operation: +For the LTE encode operation: * ``RTE_BBDEV_TURBO_CRC_24A_ATTACH`` * ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` * ``RTE_BBDEV_TURBO_RATE_MATCH`` * ``RTE_BBDEV_TURBO_RV_INDEX_BYPASS`` -For the decode operation: +For the LTE decode operation: * ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` * ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` @@ -41,14 +58,10 @@ Installation FlexRAN SDK Download ~~~~~~~~~~~~~~~~~~~~ -To build DPDK with the *baseband_turbo_sw* PMD the user is required to download -the export controlled ``FlexRAN SDK`` Libraries. An account at `Intel Resource -Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_ -needs to be registered. +As an option it is possible to link this driver with FleXRAN SDK libraries +which can enable real time signal processing using AVX instructions. -Once registered, the user needs to log in, and look for -*Intel FlexRAN Software Release Package -18-09* to download or directly through -this `link <https://cdrdv2.intel.com/v1/dl/getContent/605167>`_. +These libraries are available through this link `link <https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules>`_. After download is complete, the user needs to unpack and compile on their system before building DPDK. @@ -57,24 +70,24 @@ The following table maps DPDK versions with past FlexRAN SDK releases: .. _table_flexran_releases: -.. table:: DPDK and FlexRAN SDK releases compliance +.. table:: DPDK and FlexRAN FEC SDK releases compliance ===================== ============================ - DPDK version FlexRAN SDK release + DPDK version FlexRAN FEC SDK release ===================== ============================ - 18.02 1.3.0 - 18.05 1.4.0 - 18.08 1.6.0 - 19.02 18.09 + 19.08 19.04 ===================== ============================ FlexRAN SDK Installation ~~~~~~~~~~~~~~~~~~~~~~~~ +Note that the installation of these libraries is optional. + The following are pre-requisites for building FlexRAN SDK Libraries: - (a) An AVX2 supporting machine - (b) CentOS Linux release 7.2.1511 (Core) operating system - (c) Intel ICC 18.0.1 20171018 compiler installed + (a) An AVX2 or AVX512 supporting machine + (b) CentOS Linux release 7.2.1511 (Core) operating system is advised + (c) Intel ICC 18.0.1 20171018 compiler or more recent and related libraries + ICC is available with a free community license `link <https://software.intel.com/en-us/system-studio/choose-download#technical>`_. The following instructions should be followed in this exact order: @@ -84,25 +97,18 @@ The following instructions should be followed in this exact order: source <path-to-icc-compiler-install-folder>/linux/bin/compilervars.sh intel64 -platform linux -#. Extract the ``605167-flexran-18-09-tar.gz`` package: - - .. code-block:: console - - mkdir FlexRAN-18.09 - tar xvzf 605167-flexran-18-09-tar.gz -C FlexRAN-18.09/ - #. Run the SDK extractor script and accept the license: .. code-block:: console - cd <path-to-workspace>/FlexRAN-18.09/ - ./SDK-18.09.sh + cd <path-to-workspace> + ./FlexRAN-FEC-SDK-19-04.sh #. Generate makefiles based on system configuration: .. code-block:: console - cd <path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ + cd <path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/ ./create-makefiles-linux.sh #. A build folder is generated in this form ``build-<ISA>-<CC>``, enter that @@ -129,12 +135,14 @@ Example: .. code-block:: console - export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/build-avx2-icc/install - export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ - + export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/install + export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/ -* Set ``CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y`` in DPDK common configuration - file ``config/common_base``. +* Set ``CONFIG_RTE_BBDEV_SDK_AVX2=y`` + in DPDK common configuration file ``config/common_base`` to be able to use + the SDK libraries as mentioned above. + If no flag are set the PMD driver will still build but its capabilities + will be limited accordingly. To use the PMD in an application, user must: -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [dpdk-dev] [PATCH v2 2/3] docs/guides: updating turbo_sw building steps 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru @ 2019-06-06 10:34 ` Ferruh Yigit 2019-06-06 17:03 ` Chautru, Nicolas 0 siblings, 1 reply; 62+ messages in thread From: Ferruh Yigit @ 2019-06-06 10:34 UTC (permalink / raw) To: Nicolas Chautru, thomas, akhil.goyal, dev; +Cc: amr.mokhtar On 6/5/2019 9:10 PM, Nicolas Chautru wrote: > The documentation is clarified to point to steps on building the > SDK libraries which are now publicly available. > I can see new SDK link is in the documentation, but can you please put that information into the commit log too, to highlight it? > Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> > --- > doc/guides/bbdevs/turbo_sw.rst | 84 +++++++++++++++++++++++------------------- > 1 file changed, 46 insertions(+), 38 deletions(-) > > diff --git a/doc/guides/bbdevs/turbo_sw.rst b/doc/guides/bbdevs/turbo_sw.rst > index 29f7ec9..196b3d6 100644 > --- a/doc/guides/bbdevs/turbo_sw.rst > +++ b/doc/guides/bbdevs/turbo_sw.rst > @@ -1,26 +1,43 @@ > .. SPDX-License-Identifier: BSD-3-Clause > Copyright(c) 2017 Intel Corporation > > -SW Turbo Poll Mode Driver > +SW FEC Poll Mode Driver There seems two groups of changes in this patch, - new SDK related changes, - and renaming 's/Turbo/FEC' For second one can you please put a brief description into commit log? Why Turbo needs to be FEC. (This even can be a separate patch) > ========================= > > -The SW Turbo PMD (**baseband_turbo_sw**) provides a poll mode bbdev driver that utilizes > -Intel optimized libraries for LTE Layer 1 workloads acceleration. This PMD > -supports the functions: Turbo FEC, Rate Matching and CRC functions. > +The SW FEC PMD (**baseband_turbo_sw**) provides a software only poll mode bbdev > +driver that can optionally utilize Intel optimized libraries for LTE and 5GNR > +Layer 1 workloads acceleration. Is 5GNR Layer 1 workloads acceleration available in this stage of the PMD? > + > +Note that the driver can also be built without any dependency with reduced > +functionality for maintenance purpose. > + > +To enable linking to the SDK libraries see detailed installation section below. > +Two flags can be enabled depending on whether the target machine can support > +AVX2 and AVX512 instructions sets and the related SDK libraries for vectorized > +signal processing functions are installed : > +- CONFIG_RTE_BBDEV_SDK_AVX2 > +- CONFIG_RTE_BBDEV_SDK_AVX512 This flag is not available (yet), can you please remove from this doc and add with the patch that adds this flag? <...> ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [dpdk-dev] [PATCH v2 2/3] docs/guides: updating turbo_sw building steps 2019-06-06 10:34 ` Ferruh Yigit @ 2019-06-06 17:03 ` Chautru, Nicolas 0 siblings, 0 replies; 62+ messages in thread From: Chautru, Nicolas @ 2019-06-06 17:03 UTC (permalink / raw) To: Yigit, Ferruh, thomas, akhil.goyal, dev; +Cc: Mokhtar, Amr >-----Original Message----- >From: Yigit, Ferruh >Sent: Thursday, June 6, 2019 3:34 AM > >On 6/5/2019 9:10 PM, Nicolas Chautru wrote: >> The documentation is clarified to point to steps on building the SDK >> libraries which are now publicly available. >> > >I can see new SDK link is in the documentation, but can you please put that information into the commit log too, to highlight it? Fine with me. > >> Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> >> --- >> doc/guides/bbdevs/turbo_sw.rst | 84 >> +++++++++++++++++++++++------------------- >> 1 file changed, 46 insertions(+), 38 deletions(-) >> >> diff --git a/doc/guides/bbdevs/turbo_sw.rst >> b/doc/guides/bbdevs/turbo_sw.rst index 29f7ec9..196b3d6 100644 >> --- a/doc/guides/bbdevs/turbo_sw.rst >> +++ b/doc/guides/bbdevs/turbo_sw.rst >> @@ -1,26 +1,43 @@ >> .. SPDX-License-Identifier: BSD-3-Clause >> Copyright(c) 2017 Intel Corporation >> >> -SW Turbo Poll Mode Driver >> +SW FEC Poll Mode Driver > >There seems two groups of changes in this patch, >- new SDK related changes, >- and renaming 's/Turbo/FEC' >For second one can you please put a brief description into commit log? Why Turbo needs to be FEC. (This even can be a separate patch) > Fair enough. Name could be more accurate as Turbo means so different things but this is fairly cosmetic and we can change this in a future patchset. This would also require changes in the code which is not required now. So will put it back as before now. >> ========================= >> >> -The SW Turbo PMD (**baseband_turbo_sw**) provides a poll mode bbdev >> driver that utilizes -Intel optimized libraries for LTE Layer 1 >> workloads acceleration. This PMD -supports the functions: Turbo FEC, Rate Matching and CRC functions. >> +The SW FEC PMD (**baseband_turbo_sw**) provides a software only poll >> +mode bbdev driver that can optionally utilize Intel optimized >> +libraries for LTE and 5GNR Layer 1 workloads acceleration. > >Is 5GNR Layer 1 workloads acceleration available in this stage of the PMD? > The SDK provided on the link supports both standards and set of libraries and these how they are referred on the webpage : "FlexRAN LTE and 5G NR FEC Software Development Kit Modules". Still if you reckon this is confusing given this current PMD only links to the 4G ones then I can remove any mention of 5GNR. The related turbo_sw in patchwork v1 is extending this and hence documentation will eventually mention back 5GNR anyway. Will change now. >> + >> +Note that the driver can also be built without any dependency with >> +reduced functionality for maintenance purpose. >> + >> +To enable linking to the SDK libraries see detailed installation section below. >> +Two flags can be enabled depending on whether the target machine can >> +support >> +AVX2 and AVX512 instructions sets and the related SDK libraries for >> +vectorized signal processing functions are installed : >> +- CONFIG_RTE_BBDEV_SDK_AVX2 >> +- CONFIG_RTE_BBDEV_SDK_AVX512 > >This flag is not available (yet), can you please remove from this doc and add with the patch that adds this flag? No problem. The intention was to highlight where this is going with 2 sets of functionality depending on HW dependency as it is not necessarily obvious now that these are split into 2 patchsets. But you are right that this can be rephrased to be limited to AVX2 for now. Thanks > ><...> > ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v2 3/3] baseband/turbo_sw: meson build support for PMD driver 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru @ 2019-06-05 20:10 ` Nicolas Chautru 2 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-05 20:10 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- drivers/baseband/meson.build | 2 +- drivers/baseband/turbo_sw/meson.build | 30 ++++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletion(-) create mode 100644 drivers/baseband/turbo_sw/meson.build diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build index 52489df..40a87d2 100644 --- a/drivers/baseband/meson.build +++ b/drivers/baseband/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <bluca@debian.org> -drivers = ['null'] +drivers = ['null', 'turbo_sw'] config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' driver_name_fmt = 'rte_pmd_@0@' diff --git a/drivers/baseband/turbo_sw/meson.build b/drivers/baseband/turbo_sw/meson.build new file mode 100644 index 0000000..9b4fe34 --- /dev/null +++ b/drivers/baseband/turbo_sw/meson.build @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +path = get_option('flexran_sdk') + +if dpdk_conf.has('RTE_BBDEV_SDK_AVX2') + lib = cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: false) + if not lib.found() + build = false + else + ext_deps += cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: true) + ext_deps += cc.find_library('libcrc', dirs: [path + '/lib_crc'], required: true) + ext_deps += cc.find_library('librate_matching', dirs: [path + '/lib_rate_matching'], required: true) + ext_deps += cc.find_library('libcommon', dirs: [path + '/lib_common'], required: true) + ext_deps += cc.find_library('libstdc++', required: true) + ext_deps += cc.find_library('libirc', required: true) + ext_deps += cc.find_library('libimf', required: true) + ext_deps += cc.find_library('libipps', required: true) + ext_deps += cc.find_library('libsvml', required: true) + includes += include_directories(path + '/lib_turbo') + includes += include_directories(path + '/lib_crc') + includes += include_directories(path + '/lib_rate_matching') + includes += include_directories(path + '/lib_common') + endif +endif + +deps += ['bbdev', 'bus_vdev', 'ring'] +name = 'bbdev_turbo_sw' +allow_experimental_apis = true +sources = files('bbdev_turbo_software.c') -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v2 0/3] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-05 17:38 ` [dpdk-dev] [PATCH] baseband/turbo_sw: Option to build turbosw PMD without SDK Nicolas Chautru ` (2 preceding siblings ...) 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru @ 2019-06-05 21:20 ` Nicolas Chautru 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 1/3] " Nicolas Chautru ` (2 more replies) 3 siblings, 3 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-05 21:20 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru This is adding a new PMD driver for BBDEV device based on FPGA implementation on PAC N3000 HW to provide FEC 4G acceleration. v1 was shared earlier on this patchwork : https://patches.dpdk.org/patch/53409/ The existing BBDEV test framework is still supported. Main updates based on feedback during the last v1 were: - correct the documentation which was arguably misleading in hinting that ICC was required. There is no dependency whatsoever. - add meson build support Note that a number of other BBDEV patches are in v1 here and will be resubmitted for 19.08 rc1 https://patches.dpdk.org/project/dpdk/list/?series=4657 Nicolas Chautru (3): baseband/fpga_lte_fec: adding driver for FEC on FPGA doc/guides: documentation for the FPGA BBDEV PMD baseband/fpga_lte_fec: meson support config/common_base | 6 + doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++ doc/guides/bbdevs/index.rst | 1 + drivers/baseband/Makefile | 2 + drivers/baseband/fpga_lte_fec/Makefile | 29 + drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 ++++++++++++++++++++ drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + drivers/baseband/fpga_lte_fec/meson.build | 7 + .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + drivers/baseband/meson.build | 2 +- mk/rte.app.mk | 1 + 11 files changed, 3115 insertions(+), 1 deletion(-) create mode 100644 doc/guides/bbdevs/fpga_lte_fec.rst create mode 100644 drivers/baseband/fpga_lte_fec/Makefile create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h create mode 100644 drivers/baseband/fpga_lte_fec/meson.build create mode 100644 drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v2 1/3] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 0/3] baseband/fpga_lte_fec: adding driver for FEC on FPGA Nicolas Chautru @ 2019-06-05 21:20 ` Nicolas Chautru 2019-06-06 9:04 ` [dpdk-dev] [PATCH v3] " Nicolas Chautru 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 2/3] doc/guides: documentation for the FPGA BBDEV PMD Nicolas Chautru 2019-06-05 21:21 ` [dpdk-dev] [PATCH v2 3/3] baseband/fpga_lte_fec: meson support Nicolas Chautru 2 siblings, 2 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-05 21:20 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Supports for FEC 4G PMD Driver on FPGA card PAC N3000 Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 6 + drivers/baseband/Makefile | 2 + drivers/baseband/fpga_lte_fec/Makefile | 29 + drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 ++++++++++++++++++++ drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + mk/rte.app.mk | 1 + 7 files changed, 2788 insertions(+) create mode 100644 drivers/baseband/fpga_lte_fec/Makefile create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h create mode 100644 drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map diff --git a/config/common_base b/config/common_base index 6b96e0e..ae52028 100644 --- a/config/common_base +++ b/config/common_base @@ -521,6 +521,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y # EXPERIMENTAL: API may change without prior notice # CONFIG_RTE_LIBRTE_BBDEV=y +CONFIG_RTE_LIBRTE_BBDEV_DEBUG=n CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y @@ -535,6 +536,11 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n # +# Compile PMD for Intel FPGA LTE FEC bbdev device +# +CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC=y + +# # Compile generic crypto device library # CONFIG_RTE_LIBRTE_CRYPTODEV=y diff --git a/drivers/baseband/Makefile b/drivers/baseband/Makefile index 4ec83b0..ceffc7d 100644 --- a/drivers/baseband/Makefile +++ b/drivers/baseband/Makefile @@ -10,5 +10,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += null DEPDIRS-null = $(core-libs) DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += turbo_sw DEPDIRS-turbo_sw = $(core-libs) +DIRS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += fpga_lte_fec +DEPDIRS-fpga_lte_fec = $(core-libs) include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/drivers/baseband/fpga_lte_fec/Makefile b/drivers/baseband/fpga_lte_fec/Makefile new file mode 100644 index 0000000..e478d84 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/Makefile @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2018 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_pmd_fpga_lte_fec.a + +# build flags +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring +LDLIBS += -lrte_bbdev +LDLIBS += -lrte_pci -lrte_bus_pci + +# versioning export map +EXPORT_MAP := rte_pmd_bbdev_fpga_lte_fec_version.map + +# library version +LIBABIVER := 1 + +# library source files +SRCS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += fpga_lte_fec.c + +# export include files +SYMLINK-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC)-include += fpga_lte_fec.h + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c new file mode 100644 index 0000000..856b68e --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c @@ -0,0 +1,2674 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2018 Intel Corporation + */ + +#include <unistd.h> + +#include <rte_common.h> +#include <rte_log.h> +#include <rte_dev.h> +#include <rte_malloc.h> +#include <rte_mempool.h> +#include <rte_errno.h> +#include <rte_pci.h> +#include <rte_bus_pci.h> +#include <rte_byteorder.h> +#ifdef RTE_BBDEV_OFFLOAD_COST +#include <rte_cycles.h> +#endif + +#include <rte_bbdev.h> +#include <rte_bbdev_pmd.h> + +#include "fpga_lte_fec.h" + +/* Turbo SW PMD logging ID */ +static int fpga_lte_fec_logtype; + +/* Helper macro for logging */ +#define rte_bbdev_log(level, fmt, ...) \ + rte_log(RTE_LOG_ ## level, fpga_lte_fec_logtype, fmt "\n", \ + ##__VA_ARGS__) + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +#define rte_bbdev_log_debug(fmt, ...) \ + rte_bbdev_log(DEBUG, "fpga_lte_fec: " fmt, \ + ##__VA_ARGS__) +#else +#define rte_bbdev_log_debug(fmt, ...) +#endif + +/* FPGA LTE FEC driver names */ +#define FPGA_LTE_FEC_PF_DRIVER_NAME intel_fpga_lte_fec_pf +#define FPGA_LTE_FEC_VF_DRIVER_NAME intel_fpga_lte_fec_vf + +/* FPGA LTE FEC PCI vendor & device IDs */ +#define FPGA_LTE_FEC_VENDOR_ID (0x1172) +#define FPGA_LTE_FEC_PF_DEVICE_ID (0x5052) +#define FPGA_LTE_FEC_VF_DEVICE_ID (0x5050) + +/* Align DMA descriptors to 256 bytes - cache-aligned */ +#define FPGA_RING_DESC_ENTRY_LENGTH (8) +/* Ring size is in 256 bits (32 bytes) units */ +#define FPGA_RING_DESC_LEN_UNIT_BYTES (32) +/* Maximum size of queue */ +#define FPGA_RING_MAX_SIZE (1024) +#define FPGA_FLR_TIMEOUT_UNIT (16.384) + +#define FPGA_NUM_UL_QUEUES (32) +#define FPGA_NUM_DL_QUEUES (32) +#define FPGA_TOTAL_NUM_QUEUES (FPGA_NUM_UL_QUEUES + FPGA_NUM_DL_QUEUES) +#define FPGA_NUM_INTR_VEC (FPGA_TOTAL_NUM_QUEUES - RTE_INTR_VEC_RXTX_OFFSET) + +#define FPGA_INVALID_HW_QUEUE_ID (0xFFFFFFFF) + +#define FPGA_QUEUE_FLUSH_TIMEOUT_US (1000) +#define FPGA_TIMEOUT_CHECK_INTERVAL (5) + +/* FPGA LTE FEC Register mapping on BAR0 */ +enum { + FPGA_LTE_FEC_VERSION_ID = 0x00000000, /* len: 4B */ + FPGA_LTE_FEC_CONFIGURATION = 0x00000004, /* len: 2B */ + FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE = 0x00000008, /* len: 1B */ + FPGA_LTE_FEC_LOAD_BALANCE_FACTOR = 0x0000000a, /* len: 2B */ + FPGA_LTE_FEC_RING_DESC_LEN = 0x0000000c, /* len: 2B */ + FPGA_LTE_FEC_FLR_TIME_OUT = 0x0000000e, /* len: 2B */ + FPGA_LTE_FEC_VFQ_FLUSH_STATUS_LW = 0x00000018, /* len: 4B */ + FPGA_LTE_FEC_VFQ_FLUSH_STATUS_HI = 0x0000001c, /* len: 4B */ + FPGA_LTE_FEC_VF0_DEBUG = 0x00000020, /* len: 4B */ + FPGA_LTE_FEC_VF1_DEBUG = 0x00000024, /* len: 4B */ + FPGA_LTE_FEC_VF2_DEBUG = 0x00000028, /* len: 4B */ + FPGA_LTE_FEC_VF3_DEBUG = 0x0000002c, /* len: 4B */ + FPGA_LTE_FEC_VF4_DEBUG = 0x00000030, /* len: 4B */ + FPGA_LTE_FEC_VF5_DEBUG = 0x00000034, /* len: 4B */ + FPGA_LTE_FEC_VF6_DEBUG = 0x00000038, /* len: 4B */ + FPGA_LTE_FEC_VF7_DEBUG = 0x0000003c, /* len: 4B */ + FPGA_LTE_FEC_QUEUE_MAP = 0x00000040, /* len: 256B */ + FPGA_LTE_FEC_RING_CTRL_REGS = 0x00000200 /* len: 2048B */ +}; + +/* FPGA LTE FEC Ring Control Registers */ +enum { + FPGA_LTE_FEC_RING_HEAD_ADDR = 0x00000008, + FPGA_LTE_FEC_RING_SIZE = 0x00000010, + FPGA_LTE_FEC_RING_MISC = 0x00000014, + FPGA_LTE_FEC_RING_ENABLE = 0x00000015, + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN = 0x00000016, + FPGA_LTE_FEC_RING_SHADOW_TAIL = 0x00000018, + FPGA_LTE_FEC_RING_HEAD_POINT = 0x0000001C +}; + +/* FPGA LTE FEC DESCRIPTOR ERROR */ +enum { + DESC_ERR_NO_ERR = 0x0, + DESC_ERR_K_OUT_OF_RANGE = 0x1, + DESC_ERR_K_NOT_NORMAL = 0x2, + DESC_ERR_KPAI_NOT_NORMAL = 0x3, + DESC_ERR_DESC_OFFSET_ERR = 0x4, + DESC_ERR_DESC_READ_FAIL = 0x8, + DESC_ERR_DESC_READ_TIMEOUT = 0x9, + DESC_ERR_DESC_READ_TLP_POISONED = 0xA, + DESC_ERR_CB_READ_FAIL = 0xC, + DESC_ERR_CB_READ_TIMEOUT = 0xD, + DESC_ERR_CB_READ_TLP_POISONED = 0xE +}; + +/* FPGA LTE FEC DMA Encoding Request Descriptor */ +struct __attribute__((__packed__)) fpga_dma_enc_desc { + uint32_t done:1, + rsrvd0:11, + error:4, + rsrvd1:16; + uint32_t ncb:16, + rsrvd2:14, + rv:2; + uint32_t bypass_rm:1, + irq_en:1, + crc_en:1, + rsrvd3:13, + offset:10, + rsrvd4:6; + uint16_t e; + uint16_t k; + uint32_t out_addr_lw; + uint32_t out_addr_hi; + uint32_t in_addr_lw; + uint32_t in_addr_hi; + + union { + struct { + /* Virtual addresses used to retrieve SW context info */ + void *op_addr; + /* Stores information about total number of Code Blocks + * in currently processed Transport Block + */ + uint64_t cbs_in_op; + }; + + uint8_t sw_ctxt[FPGA_RING_DESC_LEN_UNIT_BYTES * + (FPGA_RING_DESC_ENTRY_LENGTH - 1)]; + }; +}; + +/* FPGA LTE FEC DMA Decoding Request Descriptor */ +struct __attribute__((__packed__)) fpga_dma_dec_desc { + uint32_t done:1, + iter:5, + rsrvd0:2, + crc_pass:1, + rsrvd1:3, + error:4, + crc_type:1, + rsrvd2:7, + max_iter:5, + rsrvd3:3; + uint32_t rsrvd4; + uint32_t bypass_rm:1, + irq_en:1, + drop_crc:1, + rsrvd5:13, + offset:10, + rsrvd6:6; + uint16_t k; + uint16_t in_len; + uint32_t out_addr_lw; + uint32_t out_addr_hi; + uint32_t in_addr_lw; + uint32_t in_addr_hi; + + union { + struct { + /* Virtual addresses used to retrieve SW context info */ + void *op_addr; + /* Stores information about total number of Code Blocks + * in currently processed Transport Block + */ + uint8_t cbs_in_op; + }; + + uint32_t sw_ctxt[8 * (FPGA_RING_DESC_ENTRY_LENGTH - 1)]; + }; +}; + +/* FPGA LTE DMA Descriptor */ +union fpga_dma_desc { + struct fpga_dma_enc_desc enc_req; + struct fpga_dma_dec_desc dec_req; +}; + +/* FPGA LTE FEC Ring Control Register */ +struct __attribute__((__packed__)) fpga_ring_ctrl_reg { + uint64_t ring_base_addr; + uint64_t ring_head_addr; + uint16_t ring_size:11; + uint16_t rsrvd0; + union { /* Miscellaneous register */ + uint8_t misc; + uint8_t max_ul_dec:5, + max_ul_dec_en:1, + rsrvd1:2; + }; + uint8_t enable; + uint8_t flush_queue_en; + uint8_t rsrvd2; + uint16_t shadow_tail; + uint16_t rsrvd3; + uint16_t head_point; + uint16_t rsrvd4; + +}; + +/* Private data structure for each FPGA FEC device */ +struct fpga_lte_fec_device { + /** Base address of MMIO registers (BAR0) */ + void *mmio_base; + /** Base address of memory for sw rings */ + void *sw_rings; + /** Physical address of sw_rings */ + rte_iova_t sw_rings_phys; + /** Number of bytes available for each queue in device. */ + uint32_t sw_ring_size; + /** Max number of entries available for each queue in device */ + uint32_t sw_ring_max_depth; + /** Base address of response tail pointer buffer */ + uint32_t *tail_ptrs; + /** Physical address of tail pointers */ + rte_iova_t tail_ptr_phys; + /** Queues flush completion flag */ + uint64_t *flush_queue_status; + /* Bitmap capturing which Queues are bound to the PF/VF */ + uint64_t q_bound_bit_map; + /* Bitmap capturing which Queues have already been assigned */ + uint64_t q_assigned_bit_map; + /** True if this is a PF FPGA FEC device */ + bool pf_device; +}; + +/* Structure associated with each queue. */ +struct __rte_cache_aligned fpga_queue { + struct fpga_ring_ctrl_reg ring_ctrl_reg; /* Ring Control Register */ + union fpga_dma_desc *ring_addr; /* Virtual address of software ring */ + uint64_t *ring_head_addr; /* Virtual address of completion_head */ + uint64_t shadow_completion_head; /* Shadow completion head value */ + uint16_t head_free_desc; /* Ring head */ + uint16_t tail; /* Ring tail */ + /* Mask used to wrap enqueued descriptors on the sw ring */ + uint32_t sw_ring_wrap_mask; + uint32_t irq_enable; /* Enable ops dequeue interrupts if set to 1 */ + uint8_t q_idx; /* Queue index */ + struct fpga_lte_fec_device *d; + /* MMIO register of shadow_tail used to enqueue descriptors */ + void *shadow_tail_addr; +}; + +/* Write to 16 bit MMIO register address */ +static inline void +mmio_write_16(void *addr, uint16_t value) +{ + *((volatile uint16_t *)(addr)) = rte_cpu_to_le_16(value); +} + +/* Write to 32 bit MMIO register address */ +static inline void +mmio_write_32(void *addr, uint32_t value) +{ + *((volatile uint32_t *)(addr)) = rte_cpu_to_le_32(value); +} + +/* Write to 64 bit MMIO register address */ +static inline void +mmio_write_64(void *addr, uint64_t value) +{ + *((volatile uint64_t *)(addr)) = rte_cpu_to_le_64(value); +} + +/* Write a 8 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_8(void *mmio_base, uint32_t offset, uint8_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + *((volatile uint8_t *)(reg_addr)) = payload; +} + +/* Write a 16 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_16(void *mmio_base, uint32_t offset, uint16_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_16(reg_addr, payload); +} + +/* Write a 32 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_32(void *mmio_base, uint32_t offset, uint32_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_32(reg_addr, payload); +} + +/* Write a 64 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_64(void *mmio_base, uint32_t offset, uint64_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_64(reg_addr, payload); +} + +/* Write a ring control register of a FPGA LTE FEC device */ +static inline void +fpga_ring_reg_write(void *mmio_base, uint32_t offset, + struct fpga_ring_ctrl_reg payload) +{ + fpga_reg_write_64(mmio_base, offset, payload.ring_base_addr); + fpga_reg_write_64(mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_ADDR, + payload.ring_head_addr); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_SIZE, + payload.ring_size); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_POINT, + payload.head_point); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN, + payload.flush_queue_en); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_SHADOW_TAIL, + payload.shadow_tail); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_MISC, + payload.misc); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + payload.enable); +} + +/* Read a register of FPGA LTE FEC device */ +static uint32_t +fpga_reg_read_32(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint32_t ret = *((volatile uint32_t *)(reg_addr)); + return rte_le_to_cpu_32(ret); +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Read a register of FPGA LTE FEC device */ +static uint8_t +fpga_reg_read_8(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + return *((volatile uint8_t *)(reg_addr)); +} + +/* Read a register of FPGA LTE FEC device */ +static uint16_t +fpga_reg_read_16(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint16_t ret = *((volatile uint16_t *)(reg_addr)); + return rte_le_to_cpu_16(ret); +} + +/* Read a register of FPGA LTE FEC device */ +static uint64_t +fpga_reg_read_64(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint64_t ret = *((volatile uint64_t *)(reg_addr)); + return rte_le_to_cpu_64(ret); +} + +/* Read Ring Control Register of FPGA LTE FEC device */ +static inline void +print_ring_reg_debug_info(void *mmio_base, uint32_t offset) +{ + rte_bbdev_log_debug( + "FPGA MMIO base address @ %p | Ring Control Register @ offset = 0x%08" + PRIx32, mmio_base, offset); + rte_bbdev_log_debug( + "RING_BASE_ADDR = 0x%016"PRIx64, + fpga_reg_read_64(mmio_base, offset)); + rte_bbdev_log_debug( + "RING_HEAD_ADDR = 0x%016"PRIx64, + fpga_reg_read_64(mmio_base, offset + + FPGA_LTE_FEC_RING_HEAD_ADDR)); + rte_bbdev_log_debug( + "RING_SIZE = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_SIZE)); + rte_bbdev_log_debug( + "RING_MISC = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_MISC)); + rte_bbdev_log_debug( + "RING_ENABLE = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_ENABLE)); + rte_bbdev_log_debug( + "RING_FLUSH_QUEUE_EN = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN)); + rte_bbdev_log_debug( + "RING_SHADOW_TAIL = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_SHADOW_TAIL)); + rte_bbdev_log_debug( + "RING_HEAD_POINT = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_HEAD_POINT)); +} + +/* Read Static Register of FPGA LTE FEC device */ +static inline void +print_static_reg_debug_info(void *mmio_base) +{ + uint16_t config = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_CONFIGURATION); + uint8_t qmap_done = fpga_reg_read_8(mmio_base, + FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE); + uint16_t lb_factor = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_LOAD_BALANCE_FACTOR); + uint16_t ring_desc_len = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_RING_DESC_LEN); + uint16_t flr_time_out = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_FLR_TIME_OUT); + + rte_bbdev_log_debug("UL.DL Weights = %u.%u", + ((uint8_t)config), ((uint8_t)(config >> 8))); + rte_bbdev_log_debug("UL.DL Load Balance = %u.%u", + ((uint8_t)lb_factor), ((uint8_t)(lb_factor >> 8))); + rte_bbdev_log_debug("Queue-PF/VF Mapping Table = %s", + (qmap_done > 0) ? "READY" : "NOT-READY"); + rte_bbdev_log_debug("Ring Descriptor Size = %u bytes", + ring_desc_len*FPGA_RING_DESC_LEN_UNIT_BYTES); + rte_bbdev_log_debug("FLR Timeout = %f usec", + (float)flr_time_out*FPGA_FLR_TIMEOUT_UNIT); +} + +/* Print decode DMA Descriptor of FPGA LTE FEC device */ +static void +print_dma_dec_desc_debug_info(union fpga_dma_desc *desc) +{ + rte_bbdev_log_debug("DMA response desc %p\n" + "\t-- done(%"PRIu32") | iter(%"PRIu32") | crc_pass(%"PRIu32")" + " | error (%"PRIu32") | crc_type(%"PRIu32")\n" + "\t-- max_iter(%"PRIu32") | bypass_rm(%"PRIu32") | " + "irq_en (%"PRIu32") | drop_crc(%"PRIu32") | offset(%"PRIu32")\n" + "\t-- k(%"PRIu32") | in_len (%"PRIu16") | op_add(%p)\n" + "\t-- cbs_in_op(%"PRIu32") | in_add (0x%08"PRIx32"%08"PRIx32") | " + "out_add (0x%08"PRIx32"%08"PRIx32")", + desc, + (uint32_t)desc->dec_req.done, + (uint32_t)desc->dec_req.iter, + (uint32_t)desc->dec_req.crc_pass, + (uint32_t)desc->dec_req.error, + (uint32_t)desc->dec_req.crc_type, + (uint32_t)desc->dec_req.max_iter, + (uint32_t)desc->dec_req.bypass_rm, + (uint32_t)desc->dec_req.irq_en, + (uint32_t)desc->dec_req.drop_crc, + (uint32_t)desc->dec_req.offset, + (uint32_t)desc->dec_req.k, + (uint16_t)desc->dec_req.in_len, + desc->dec_req.op_addr, + (uint32_t)desc->dec_req.cbs_in_op, + (uint32_t)desc->dec_req.in_addr_hi, + (uint32_t)desc->dec_req.in_addr_lw, + (uint32_t)desc->dec_req.out_addr_hi, + (uint32_t)desc->dec_req.out_addr_lw); +} +#endif + +static int +fpga_setup_queues(struct rte_bbdev *dev, uint16_t num_queues, int socket_id) +{ + /* Number of queues bound to a PF/VF */ + uint32_t hw_q_num = 0; + uint32_t ring_size, payload, address, q_id, offset; + rte_iova_t phys_addr; + struct fpga_ring_ctrl_reg ring_reg; + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + + address = FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE; + if (!(fpga_reg_read_32(fpga_dev->mmio_base, address) & 0x1)) { + rte_bbdev_log(ERR, + "Queue-PF/VF mapping is not set! Was PF configured for device (%s) ?", + dev->data->name); + return -EPERM; + } + + /* Clear queue registers structure */ + memset(&ring_reg, 0, sizeof(struct fpga_ring_ctrl_reg)); + + /* Scan queue map. + * If a queue is valid and mapped to a calling PF/VF the read value is + * replaced with a queue ID and if it's not then + * FPGA_INVALID_HW_QUEUE_ID is returned. + */ + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + uint32_t hw_q_id = fpga_reg_read_32(fpga_dev->mmio_base, + FPGA_LTE_FEC_QUEUE_MAP + (q_id << 2)); + + rte_bbdev_log_debug("%s: queue ID: %u, registry queue ID: %u", + dev->device->name, q_id, hw_q_id); + + if (hw_q_id != FPGA_INVALID_HW_QUEUE_ID) { + fpga_dev->q_bound_bit_map |= (1ULL << q_id); + /* Clear queue register of found queue */ + offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q_id); + fpga_ring_reg_write(fpga_dev->mmio_base, + offset, ring_reg); + ++hw_q_num; + } + } + if (hw_q_num == 0) { + rte_bbdev_log(ERR, + "No HW queues assigned to this device. Probably this is a VF configured for PF mode. Check device configuration!"); + return -ENODEV; + } + + if (num_queues > hw_q_num) { + rte_bbdev_log(ERR, + "Not enough queues for device %s! Requested: %u, available: %u", + dev->device->name, num_queues, hw_q_num); + return -EINVAL; + } + + ring_size = FPGA_RING_MAX_SIZE * sizeof(struct fpga_dma_dec_desc); + + /* Enforce 32 byte alignment */ + RTE_BUILD_BUG_ON((RTE_CACHE_LINE_SIZE % 32) != 0); + + /* Allocate memory for SW descriptor rings */ + fpga_dev->sw_rings = rte_zmalloc_socket(dev->device->driver->name, + num_queues * ring_size, RTE_CACHE_LINE_SIZE, + socket_id); + if (fpga_dev->sw_rings == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u sw_rings", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + + fpga_dev->sw_rings_phys = rte_malloc_virt2iova(fpga_dev->sw_rings); + fpga_dev->sw_ring_size = ring_size; + fpga_dev->sw_ring_max_depth = FPGA_RING_MAX_SIZE; + + /* Allocate memory for ring flush status */ + fpga_dev->flush_queue_status = rte_zmalloc_socket(NULL, + sizeof(uint64_t), RTE_CACHE_LINE_SIZE, socket_id); + if (fpga_dev->flush_queue_status == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u flush_queue_status", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + + /* Set the flush status address registers */ + phys_addr = rte_malloc_virt2iova(fpga_dev->flush_queue_status); + + address = FPGA_LTE_FEC_VFQ_FLUSH_STATUS_LW; + payload = (uint32_t)(phys_addr); + fpga_reg_write_32(fpga_dev->mmio_base, address, payload); + + address = FPGA_LTE_FEC_VFQ_FLUSH_STATUS_HI; + payload = (uint32_t)(phys_addr >> 32); + fpga_reg_write_32(fpga_dev->mmio_base, address, payload); + + return 0; +} + +static int +fpga_dev_close(struct rte_bbdev *dev) +{ + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + + rte_free(fpga_dev->sw_rings); + rte_free(fpga_dev->flush_queue_status); + + return 0; +} + +static void +fpga_dev_info_get(struct rte_bbdev *dev, + struct rte_bbdev_driver_info *dev_info) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + uint32_t q_id = 0; + + /* TODO RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN and numbers of buffers are set + * to temporary values as they are required by test application while + * validation phase. + */ + static const struct rte_bbdev_op_cap bbdev_capabilities[] = { + { + .type = RTE_BBDEV_OP_TURBO_DEC, + .cap.turbo_dec = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_TYPE_24B | + RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE | + RTE_BBDEV_TURBO_DEC_INTERRUPTS | + RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN | + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP, + .max_llr_modulus = INT8_MAX, + .num_buffers_src = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_hard_out = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_soft_out = 0 + } + }, + { + .type = RTE_BBDEV_OP_TURBO_ENC, + .cap.turbo_enc = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_24B_ATTACH | + RTE_BBDEV_TURBO_RATE_MATCH | + RTE_BBDEV_TURBO_ENC_INTERRUPTS, + .num_buffers_src = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_dst = + RTE_BBDEV_MAX_CODE_BLOCKS + } + }, + RTE_BBDEV_END_OF_CAPABILITIES_LIST() + }; + + static struct rte_bbdev_queue_conf default_queue_conf; + default_queue_conf.socket = dev->data->socket_id; + default_queue_conf.queue_size = FPGA_RING_MAX_SIZE; + + + dev_info->driver_name = dev->device->driver->name; + dev_info->queue_size_lim = FPGA_RING_MAX_SIZE; + dev_info->hardware_accelerated = true; + dev_info->min_alignment = 64; + dev_info->default_queue_conf = default_queue_conf; + dev_info->capabilities = bbdev_capabilities; + dev_info->cpu_flag_reqs = NULL; + + /* Calculates number of queues assigned to device */ + dev_info->max_num_queues = 0; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + uint32_t hw_q_id = fpga_reg_read_32(d->mmio_base, + FPGA_LTE_FEC_QUEUE_MAP + (q_id << 2)); + if (hw_q_id != FPGA_INVALID_HW_QUEUE_ID) + dev_info->max_num_queues++; + } +} + +/** + * Find index of queue bound to current PF/VF which is unassigned. Return -1 + * when there is no available queue + */ +static int +fpga_find_free_queue_idx(struct rte_bbdev *dev, + const struct rte_bbdev_queue_conf *conf) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + uint64_t q_idx; + uint8_t i = 0; + uint8_t range = FPGA_TOTAL_NUM_QUEUES >> 1; + + if (conf->op_type == RTE_BBDEV_OP_TURBO_ENC) { + i = FPGA_NUM_DL_QUEUES; + range = FPGA_TOTAL_NUM_QUEUES; + } + + for (; i < range; ++i) { + q_idx = 1ULL << i; + /* Check if index of queue is bound to current PF/VF */ + if (d->q_bound_bit_map & q_idx) + /* Check if found queue was not already assigned */ + if (!(d->q_assigned_bit_map & q_idx)) { + d->q_assigned_bit_map |= q_idx; + return i; + } + } + + rte_bbdev_log(INFO, "Failed to find free queue on %s", dev->data->name); + + return -1; +} + +static int +fpga_queue_setup(struct rte_bbdev *dev, uint16_t queue_id, + const struct rte_bbdev_queue_conf *conf) +{ + uint32_t address, ring_offset; + struct fpga_lte_fec_device *d = dev->data->dev_private; + struct fpga_queue *q; + int8_t q_idx; + + /* Check if there is a free queue to assign */ + q_idx = fpga_find_free_queue_idx(dev, conf); + if (q_idx == -1) + return -1; + + /* Allocate the queue data structure. */ + q = rte_zmalloc_socket(dev->device->driver->name, sizeof(*q), + RTE_CACHE_LINE_SIZE, conf->socket); + if (q == NULL) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_bbdev_log(ERR, "Failed to allocate queue memory"); + return -ENOMEM; + } + + q->d = d; + q->q_idx = q_idx; + + /* Set ring_base_addr */ + q->ring_addr = RTE_PTR_ADD(d->sw_rings, (d->sw_ring_size * queue_id)); + q->ring_ctrl_reg.ring_base_addr = d->sw_rings_phys + + (d->sw_ring_size * queue_id); + + /* Allocate memory for Completion Head variable*/ + q->ring_head_addr = rte_zmalloc_socket(dev->device->driver->name, + sizeof(uint64_t), RTE_CACHE_LINE_SIZE, conf->socket); + if (q->ring_head_addr == NULL) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_free(q); + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u completion_head", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + /* Set ring_head_addr */ + q->ring_ctrl_reg.ring_head_addr = + rte_malloc_virt2iova(q->ring_head_addr); + + /* Clear shadow_completion_head */ + q->shadow_completion_head = 0; + + /* Set ring_size */ + if (conf->queue_size > FPGA_RING_MAX_SIZE) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_free(q->ring_head_addr); + rte_free(q); + rte_bbdev_log(ERR, + "Size of queue is too big %d (MAX: %d ) for %s:%u", + conf->queue_size, FPGA_RING_MAX_SIZE, + dev->device->driver->name, dev->data->dev_id); + return -EINVAL; + } + q->ring_ctrl_reg.ring_size = conf->queue_size; + + /* Set Miscellaneous FPGA register*/ + /* Max iteration number for TTI mitigation - todo */ + q->ring_ctrl_reg.max_ul_dec = 0; + /* Enable max iteration number for TTI - todo */ + q->ring_ctrl_reg.max_ul_dec_en = 0; + + /* Enable the ring */ + q->ring_ctrl_reg.enable = 1; + + /* Set FPGA head_point and tail registers */ + q->ring_ctrl_reg.head_point = q->tail = 0; + + /* Set FPGA shadow_tail register */ + q->ring_ctrl_reg.shadow_tail = q->tail; + + /* Calculates the ring offset for found queue */ + ring_offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q_idx); + + /* Set FPGA Ring Control Registers */ + fpga_ring_reg_write(d->mmio_base, ring_offset, q->ring_ctrl_reg); + + /* Store MMIO register of shadow_tail */ + address = ring_offset + FPGA_LTE_FEC_RING_SHADOW_TAIL; + q->shadow_tail_addr = RTE_PTR_ADD(d->mmio_base, address); + + q->head_free_desc = q->tail; + + /* Set wrap mask */ + q->sw_ring_wrap_mask = conf->queue_size - 1; + + rte_bbdev_log_debug("Setup dev%u q%u: queue_idx=%u", + dev->data->dev_id, queue_id, q->q_idx); + + dev->data->queues[queue_id].queue_private = q; + + rte_bbdev_log_debug("BBDEV queue[%d] set up for FPGA queue[%d]", + queue_id, q_idx); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Read FPGA Ring Control Registers after configuration*/ + print_ring_reg_debug_info(d->mmio_base, ring_offset); +#endif + return 0; +} + +static int +fpga_queue_release(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + struct fpga_ring_ctrl_reg ring_reg; + uint32_t offset; + + rte_bbdev_log_debug("FPGA Queue[%d] released", queue_id); + + if (q != NULL) { + memset(&ring_reg, 0, sizeof(struct fpga_ring_ctrl_reg)); + offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + /* Disable queue */ + fpga_reg_write_8(d->mmio_base, + offset + FPGA_LTE_FEC_RING_ENABLE, 0x00); + /* Clear queue registers */ + fpga_ring_reg_write(d->mmio_base, offset, ring_reg); + + /* Mark the Queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q->q_idx)); + rte_free(q->ring_head_addr); + rte_free(q); + dev->data->queues[queue_id].queue_private = NULL; + } + + return 0; +} + +/* Function starts a device queue. */ +static int +fpga_queue_start(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (d == NULL) { + rte_bbdev_log(ERR, "Invalid device pointer"); + return -1; + } +#endif + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + uint32_t offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + uint8_t enable = 0x01; + uint16_t zero = 0x0000; + + /* Clear queue head and tail variables */ + q->tail = q->head_free_desc = 0; + + /* Clear FPGA head_point and tail registers */ + fpga_reg_write_16(d->mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_POINT, + zero); + fpga_reg_write_16(d->mmio_base, offset + FPGA_LTE_FEC_RING_SHADOW_TAIL, + zero); + + /* Enable queue */ + fpga_reg_write_8(d->mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + enable); + + rte_bbdev_log_debug("FPGA Queue[%d] started", queue_id); + return 0; +} + +/* Function stops a device queue. */ +static int +fpga_queue_stop(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (d == NULL) { + rte_bbdev_log(ERR, "Invalid device pointer"); + return -1; + } +#endif + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + uint32_t offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + uint8_t payload = 0x01; + uint8_t counter = 0; + uint8_t timeout = FPGA_QUEUE_FLUSH_TIMEOUT_US / + FPGA_TIMEOUT_CHECK_INTERVAL; + + /* Set flush_queue_en bit to trigger queue flushing */ + fpga_reg_write_8(d->mmio_base, + offset + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN, payload); + + /** Check if queue flush is completed. + * FPGA will update the completion flag after queue flushing is + * completed. If completion flag is not updated within 1ms it is + * considered as a failure. + */ + while (!(*((uint8_t *)d->flush_queue_status + q->q_idx) & payload)) { + if (counter > timeout) { + rte_bbdev_log(ERR, "FPGA Queue Flush failed for queue %d", + queue_id); + return -1; + } + usleep(FPGA_TIMEOUT_CHECK_INTERVAL); + counter++; + } + + /* Disable queue */ + payload = 0x00; + fpga_reg_write_8(d->mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + payload); + + rte_bbdev_log_debug("FPGA Queue[%d] stopped", queue_id); + return 0; +} + +static inline uint16_t +get_queue_id(struct rte_bbdev_data *data, uint8_t q_idx) +{ + uint16_t queue_id; + + for (queue_id = 0; queue_id < data->num_queues; ++queue_id) { + struct fpga_queue *q = data->queues[queue_id].queue_private; + if (q != NULL && q->q_idx == q_idx) + return queue_id; + } + + return -1; +} + +/* Interrupt handler triggered by FPGA dev for handling specific interrupt */ +static void +fpga_dev_interrupt_handler(void *cb_arg) +{ + struct rte_bbdev *dev = cb_arg; + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + struct fpga_queue *q; + uint64_t ring_head; + uint64_t q_idx; + uint16_t queue_id; + uint8_t i; + + /* Scan queue assigned to this device */ + for (i = 0; i < FPGA_TOTAL_NUM_QUEUES; ++i) { + q_idx = 1ULL << i; + if (fpga_dev->q_bound_bit_map & q_idx) { + queue_id = get_queue_id(dev->data, i); + if (queue_id == (uint16_t) -1) + continue; + + /* Check if completion head was changed */ + q = dev->data->queues[queue_id].queue_private; + ring_head = *q->ring_head_addr; + if (q->shadow_completion_head != ring_head && + q->irq_enable == 1) { + q->shadow_completion_head = ring_head; + rte_bbdev_pmd_callback_process( + dev, + RTE_BBDEV_EVENT_DEQUEUE, + &queue_id); + } + } + } +} + +static int +fpga_queue_intr_enable(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + + if (!rte_intr_cap_multiple(dev->intr_handle)) + return -ENOTSUP; + + q->irq_enable = 1; + + return 0; +} + +static int +fpga_queue_intr_disable(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + q->irq_enable = 0; + + return 0; +} + +static int +fpga_intr_enable(struct rte_bbdev *dev) +{ + int ret; + uint8_t i; + + if (!rte_intr_cap_multiple(dev->intr_handle)) { + rte_bbdev_log(ERR, "Multiple intr vector is not supported by FPGA (%s)", + dev->data->name); + return -ENOTSUP; + } + + /* Create event file descriptors for each of 64 queue. Event fds will be + * mapped to FPGA IRQs in rte_intr_enable(). This is a 1:1 mapping where + * the IRQ number is a direct translation to the queue number. + * + * 63 (FPGA_NUM_INTR_VEC) event fds are created as rte_intr_enable() + * mapped the first IRQ to already created interrupt event file + * descriptor (intr_handle->fd). + */ + if (rte_intr_efd_enable(dev->intr_handle, FPGA_NUM_INTR_VEC)) { + rte_bbdev_log(ERR, "Failed to create fds for %u queues", + dev->data->num_queues); + return -1; + } + + /* TODO Each event file descriptor is overwritten by interrupt event + * file descriptor. That descriptor is added to epoll observed list. + * It ensures that callback function assigned to that descriptor will + * invoked when any FPGA queue issues interrupt. + */ + for (i = 0; i < FPGA_NUM_INTR_VEC; ++i) + dev->intr_handle->efds[i] = dev->intr_handle->fd; + + if (!dev->intr_handle->intr_vec) { + dev->intr_handle->intr_vec = rte_zmalloc("intr_vec", + dev->data->num_queues * sizeof(int), 0); + if (!dev->intr_handle->intr_vec) { + rte_bbdev_log(ERR, "Failed to allocate %u vectors", + dev->data->num_queues); + return -ENOMEM; + } + } + + ret = rte_intr_enable(dev->intr_handle); + if (ret < 0) { + rte_bbdev_log(ERR, + "Couldn't enable interrupts for device: %s", + dev->data->name); + return ret; + } + + ret = rte_intr_callback_register(dev->intr_handle, + fpga_dev_interrupt_handler, dev); + if (ret < 0) { + rte_bbdev_log(ERR, + "Couldn't register interrupt callback for device: %s", + dev->data->name); + return ret; + } + + return 0; +} + +static const struct rte_bbdev_ops fpga_ops = { + .setup_queues = fpga_setup_queues, + .intr_enable = fpga_intr_enable, + .close = fpga_dev_close, + .info_get = fpga_dev_info_get, + .queue_setup = fpga_queue_setup, + .queue_stop = fpga_queue_stop, + .queue_start = fpga_queue_start, + .queue_release = fpga_queue_release, + .queue_intr_enable = fpga_queue_intr_enable, + .queue_intr_disable = fpga_queue_intr_disable +}; + +static inline void +fpga_dma_enqueue(struct fpga_queue *q, uint16_t num_desc, + struct rte_bbdev_stats *queue_stats) +{ +#ifdef RTE_BBDEV_OFFLOAD_COST + uint64_t start_time = 0; + queue_stats->acc_offload_cycles = 0; +#else + RTE_SET_USED(queue_stats); +#endif + + /* Update tail and shadow_tail register */ + q->tail = (q->tail + num_desc) & q->sw_ring_wrap_mask; + + rte_wmb(); + +#ifdef RTE_BBDEV_OFFLOAD_COST + /* Start time measurement for enqueue function offload. */ + start_time = rte_rdtsc_precise(); +#endif + mmio_write_16(q->shadow_tail_addr, q->tail); + +#ifdef RTE_BBDEV_OFFLOAD_COST + rte_wmb(); + queue_stats->acc_offload_cycles += rte_rdtsc_precise() - start_time; +#endif +} + +/* Calculates number of CBs in processed encoder TB based on 'r' and input + * length. + */ +static inline uint8_t +get_num_cbs_in_op_enc(struct rte_bbdev_op_turbo_enc *turbo_enc) +{ + uint8_t c, c_neg, r, crc24_bits = 0; + uint16_t k, k_neg, k_pos; + uint8_t cbs_in_op = 0; + int32_t length; + + length = turbo_enc->input.length; + r = turbo_enc->tb_params.r; + c = turbo_enc->tb_params.c; + c_neg = turbo_enc->tb_params.c_neg; + k_neg = turbo_enc->tb_params.k_neg; + k_pos = turbo_enc->tb_params.k_pos; + crc24_bits = 24; + while (length > 0 && r < c) { + k = (r < c_neg) ? k_neg : k_pos; + length -= (k - crc24_bits) >> 3; + r++; + cbs_in_op++; + } + + return cbs_in_op; +} + +/* Calculates number of CBs in processed decoder TB based on 'r' and input + * length. + */ +static inline uint16_t +get_num_cbs_in_op_dec(struct rte_bbdev_op_turbo_dec *turbo_dec) +{ + uint8_t c, c_neg, r = 0; + uint16_t kw, k, k_neg, k_pos, cbs_in_op = 0; + int32_t length; + + length = turbo_dec->input.length; + r = turbo_dec->tb_params.r; + c = turbo_dec->tb_params.c; + c_neg = turbo_dec->tb_params.c_neg; + k_neg = turbo_dec->tb_params.k_neg; + k_pos = turbo_dec->tb_params.k_pos; + while (length > 0 && r < c) { + k = (r < c_neg) ? k_neg : k_pos; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + length -= kw; + r++; + cbs_in_op++; + } + + return cbs_in_op; +} + +/* Read flag value 0/1/ from bitmap */ +static inline bool +check_bit(uint32_t bitmap, uint32_t bitmask) +{ + return bitmap & bitmask; +} + +/* Print an error if a descriptor error has occurred. + * Return 0 on success, 1 on failure + */ +static inline int +check_desc_error(uint32_t error_code) { + switch (error_code) { + case DESC_ERR_NO_ERR: + return 0; + case DESC_ERR_K_OUT_OF_RANGE: + rte_bbdev_log(ERR, "Block_size_k is out of range (k<40 or k>6144)"); + break; + case DESC_ERR_K_NOT_NORMAL: + rte_bbdev_log(ERR, "Block_size_k is not a normal value within normal range"); + break; + case DESC_ERR_KPAI_NOT_NORMAL: + rte_bbdev_log(ERR, "Three_kpai is not a normal value for UL only"); + break; + case DESC_ERR_DESC_OFFSET_ERR: + rte_bbdev_log(ERR, "Queue offset does not meet the expectation in the FPGA"); + break; + case (DESC_ERR_K_OUT_OF_RANGE | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Block_size_k is out of range (k<40 or k>6144) and queue offset error"); + break; + case (DESC_ERR_K_NOT_NORMAL | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Block_size_k is not a normal value within normal range and queue offset error"); + break; + case (DESC_ERR_KPAI_NOT_NORMAL | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Three_kpai is not a normal value for UL only and queue offset error"); + break; + case DESC_ERR_DESC_READ_FAIL: + rte_bbdev_log(ERR, "Unsuccessful completion for descriptor read"); + break; + case DESC_ERR_DESC_READ_TIMEOUT: + rte_bbdev_log(ERR, "Descriptor read time-out"); + break; + case DESC_ERR_DESC_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Descriptor read TLP poisoned"); + break; + case DESC_ERR_CB_READ_FAIL: + rte_bbdev_log(ERR, "Unsuccessful completion for code block"); + break; + case DESC_ERR_CB_READ_TIMEOUT: + rte_bbdev_log(ERR, "Code block read time-out"); + break; + case DESC_ERR_CB_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Code block read TLP poisoned"); + break; + default: + rte_bbdev_log(ERR, "Descriptor error unknown error code %u", + error_code); + break; + } + return 1; +} + +/** + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param k + * K value (length of input in bits). + * @param e + * E value (length of output in bits). + * @param ncb + * Ncb value (size of the soft buffer). + * @param out_length + * Length of output buffer + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +fpga_dma_desc_te_fill(struct rte_bbdev_enc_op *op, + struct fpga_dma_enc_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t k, uint16_t e, uint16_t ncb, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) + +{ + /* reset */ + desc->done = 0; + desc->crc_en = check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_CRC_24B_ATTACH); + desc->bypass_rm = !check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_RATE_MATCH); + desc->k = k; + desc->e = e; + desc->ncb = ncb; + desc->rv = op->turbo_enc.rv_index; + desc->offset = desc_offset; + /* Set inbound data buffer address */ + desc->in_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset) >> 32); + desc->in_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset)); + + desc->out_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset) >> 32); + desc->out_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset)); + + /* Save software context needed for dequeue */ + desc->op_addr = op; + + /* Set total number of CBs in an op */ + desc->cbs_in_op = cbs_in_op; + + return 0; +} + +/** + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param in_length + * Length of an input. + * @param k + * K value (length of an output in bits). + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +fpga_dma_desc_td_fill(struct rte_bbdev_dec_op *op, + struct fpga_dma_dec_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t in_length, uint16_t k, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) +{ + /* reset */ + desc->done = 0; + /* Set inbound data buffer address */ + desc->in_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset) >> 32); + desc->in_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset)); + desc->in_len = in_length; + desc->k = k; + desc->crc_type = !check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_CRC_TYPE_24B); + if ((op->turbo_dec.code_block_mode == 0) + && !check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP)) + desc->drop_crc = 1; + desc->max_iter = op->turbo_dec.iter_max * 2; + desc->offset = desc_offset; + desc->out_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset) >> 32); + desc->out_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset)); + + /* Save software context needed for dequeue */ + desc->op_addr = op; + + /* Set total number of CBs in an op */ + desc->cbs_in_op = cbs_in_op; + + return 0; +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Validates turbo encoder parameters */ +static int +validate_enc_op(struct rte_bbdev_enc_op *op) +{ + struct rte_bbdev_op_turbo_enc *turbo_enc = &op->turbo_enc; + struct rte_bbdev_op_enc_turbo_cb_params *cb = NULL; + struct rte_bbdev_op_enc_turbo_tb_params *tb = NULL; + uint16_t kw, kw_neg, kw_pos; + + if (turbo_enc->input.length > + RTE_BBDEV_MAX_TB_SIZE >> 3) { + rte_bbdev_log(ERR, "TB size (%u) is too big, max: %d", + turbo_enc->input.length, RTE_BBDEV_MAX_TB_SIZE); + op->status = 1 << RTE_BBDEV_DATA_ERROR; + return -1; + } + + if (op->mempool == NULL) { + rte_bbdev_log(ERR, "Invalid mempool pointer"); + return -1; + } + if (turbo_enc->input.data == NULL) { + rte_bbdev_log(ERR, "Invalid input pointer"); + return -1; + } + if (turbo_enc->output.data == NULL) { + rte_bbdev_log(ERR, "Invalid output pointer"); + return -1; + } + if (turbo_enc->rv_index > 3) { + rte_bbdev_log(ERR, + "rv_index (%u) is out of range 0 <= value <= 3", + turbo_enc->rv_index); + return -1; + } + if (turbo_enc->code_block_mode != 0 && + turbo_enc->code_block_mode != 1) { + rte_bbdev_log(ERR, + "code_block_mode (%u) is out of range 0 <= value <= 1", + turbo_enc->code_block_mode); + return -1; + } + + if (turbo_enc->code_block_mode == 0) { + tb = &turbo_enc->tb_params; + if ((tb->k_neg < RTE_BBDEV_MIN_CB_SIZE + || tb->k_neg > RTE_BBDEV_MAX_CB_SIZE) + && tb->c_neg > 0) { + rte_bbdev_log(ERR, + "k_neg (%u) is out of range %u <= value <= %u", + tb->k_neg, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->k_pos < RTE_BBDEV_MIN_CB_SIZE + || tb->k_pos > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k_pos (%u) is out of range %u <= value <= %u", + tb->k_pos, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->c_neg > (RTE_BBDEV_MAX_CODE_BLOCKS - 1)) + rte_bbdev_log(ERR, + "c_neg (%u) is out of range 0 <= value <= %u", + tb->c_neg, + RTE_BBDEV_MAX_CODE_BLOCKS - 1); + if (tb->c < 1 || tb->c > RTE_BBDEV_MAX_CODE_BLOCKS) { + rte_bbdev_log(ERR, + "c (%u) is out of range 1 <= value <= %u", + tb->c, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + if (tb->cab > tb->c) { + rte_bbdev_log(ERR, + "cab (%u) is greater than c (%u)", + tb->cab, tb->c); + return -1; + } + if ((tb->ea < RTE_BBDEV_MIN_CB_SIZE || (tb->ea % 2)) + && tb->r < tb->cab) { + rte_bbdev_log(ERR, + "ea (%u) is less than %u or it is not even", + tb->ea, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + if ((tb->eb < RTE_BBDEV_MIN_CB_SIZE || (tb->eb % 2)) + && tb->c > tb->cab) { + rte_bbdev_log(ERR, + "eb (%u) is less than %u or it is not even", + tb->eb, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + + kw_neg = 3 * RTE_ALIGN_CEIL(tb->k_neg + 4, + RTE_BBDEV_TURBO_C_SUBBLOCK); + if (tb->ncb_neg < tb->k_neg || tb->ncb_neg > kw_neg) { + rte_bbdev_log(ERR, + "ncb_neg (%u) is out of range (%u) k_neg <= value <= (%u) kw_neg", + tb->ncb_neg, tb->k_neg, kw_neg); + return -1; + } + + kw_pos = 3 * RTE_ALIGN_CEIL(tb->k_pos + 4, + RTE_BBDEV_TURBO_C_SUBBLOCK); + if (tb->ncb_pos < tb->k_pos || tb->ncb_pos > kw_pos) { + rte_bbdev_log(ERR, + "ncb_pos (%u) is out of range (%u) k_pos <= value <= (%u) kw_pos", + tb->ncb_pos, tb->k_pos, kw_pos); + return -1; + } + if (tb->r > (tb->c - 1)) { + rte_bbdev_log(ERR, + "r (%u) is greater than c - 1 (%u)", + tb->r, tb->c - 1); + return -1; + } + } else { + cb = &turbo_enc->cb_params; + if (cb->k < RTE_BBDEV_MIN_CB_SIZE + || cb->k > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k (%u) is out of range %u <= value <= %u", + cb->k, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + + if (cb->e < RTE_BBDEV_MIN_CB_SIZE || (cb->e % 2)) { + rte_bbdev_log(ERR, + "e (%u) is less than %u or it is not even", + cb->e, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + + kw = RTE_ALIGN_CEIL(cb->k + 4, RTE_BBDEV_TURBO_C_SUBBLOCK) * 3; + if (cb->ncb < cb->k || cb->ncb > kw) { + rte_bbdev_log(ERR, + "ncb (%u) is out of range (%u) k <= value <= (%u) kw", + cb->ncb, cb->k, kw); + return -1; + } + } + + return 0; +} +#endif + +static inline char * +mbuf_append(struct rte_mbuf *m_head, struct rte_mbuf *m, uint16_t len) +{ + if (unlikely(len > rte_pktmbuf_tailroom(m))) + return NULL; + + char *tail = (char *)m->buf_addr + m->data_off + m->data_len; + m->data_len = (uint16_t)(m->data_len + len); + m_head->pkt_len = (m_head->pkt_len + len); + return tail; +} + +static inline int +enqueue_enc_one_op_cb(struct fpga_queue *q, struct rte_bbdev_enc_op *op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input; + struct rte_mbuf *output; + int ret; + uint16_t k, e, ncb, ring_offset; + uint32_t total_left, in_length, out_length, in_offset, out_offset; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_enc_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo encoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_enc.input.data; + output = op->turbo_enc.output.data; + in_offset = op->turbo_enc.input.offset; + out_offset = op->turbo_enc.output.offset; + total_left = op->turbo_enc.input.length; + k = op->turbo_enc.cb_params.k; + e = op->turbo_enc.cb_params.e; + ncb = op->turbo_enc.cb_params.ncb; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_CRC_24B_ATTACH)) + in_length = ((k - 24) >> 3); + else + in_length = k >> 3; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_RATE_MATCH)) + out_length = (e + 7) >> 3; + else + out_length = (k >> 3) * 3 + 2; + + mbuf_append(output, output, out_length); + + /* Offset into the ring */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + /* Setup DMA Descriptor */ + desc = q->ring_addr + ring_offset; + + ret = fpga_dma_desc_te_fill(op, &desc->enc_req, input, output, k, e, + ncb, in_offset, out_offset, ring_offset, 1); + if (unlikely(ret < 0)) + return ret; + + /* Update lengths */ + total_left -= in_length; + op->turbo_enc.output.length += out_length; + + if (total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + total_left, in_length); + return -1; + } + + return 1; +} + +static inline int +enqueue_enc_one_op_tb(struct fpga_queue *q, struct rte_bbdev_enc_op *op, + uint16_t desc_offset, uint8_t cbs_in_op) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input, *output_head, *output; + int ret; + uint8_t r, c, crc24_bits = 0; + uint16_t k, e, ncb, ring_offset; + uint32_t mbuf_total_left, in_length, out_length, in_offset, out_offset; + uint32_t seg_total_left; + uint16_t current_enqueued_cbs = 0; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_enc_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo encoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_enc.input.data; + output_head = output = op->turbo_enc.output.data; + in_offset = op->turbo_enc.input.offset; + out_offset = op->turbo_enc.output.offset; + mbuf_total_left = op->turbo_enc.input.length; + + c = op->turbo_enc.tb_params.c; + r = op->turbo_enc.tb_params.r; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_CRC_24B_ATTACH)) + crc24_bits = 24; + + while (mbuf_total_left > 0 && r < c && input != NULL) { + seg_total_left = rte_pktmbuf_data_len(input) - in_offset; + + e = (r < op->turbo_enc.tb_params.cab) ? + op->turbo_enc.tb_params.ea : + op->turbo_enc.tb_params.eb; + k = (r < op->turbo_enc.tb_params.c_neg) ? + op->turbo_enc.tb_params.k_neg : + op->turbo_enc.tb_params.k_pos; + ncb = (r < op->turbo_enc.tb_params.c_neg) ? + op->turbo_enc.tb_params.ncb_neg : + op->turbo_enc.tb_params.ncb_pos; + + in_length = ((k - crc24_bits) >> 3); + + if (check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_RATE_MATCH)) + out_length = (e + 7) >> 3; + else + out_length = (k >> 3) * 3 + 2; + + mbuf_append(output_head, output, out_length); + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_te_fill(op, &desc->enc_req, input, output, + k, e, ncb, in_offset, out_offset, ring_offset, + cbs_in_op); + if (unlikely(ret < 0)) + return ret; + + rte_bbdev_log_debug("DMA request desc %p", desc); + + /* Update lengths */ + op->turbo_enc.output.length += out_length; + mbuf_total_left -= in_length; + + /* Update offsets */ + if (seg_total_left == in_length) { + /* Go to the next mbuf */ + input = input->next; + output = output->next; + in_offset = 0; + out_offset = 0; + } else { + in_offset += in_length; + out_offset += out_length; + } + + r++; + desc_offset++; + current_enqueued_cbs++; + } + + if (mbuf_total_left > 0) { + rte_bbdev_log(ERR, + "Some date still left for processing: mbuf_total_left = %u", + mbuf_total_left); + return -1; + } + + return current_enqueued_cbs; +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Validates turbo decoder parameters */ +static int +validate_dec_op(struct rte_bbdev_dec_op *op) +{ + struct rte_bbdev_op_turbo_dec *turbo_dec = &op->turbo_dec; + struct rte_bbdev_op_dec_turbo_cb_params *cb = NULL; + struct rte_bbdev_op_dec_turbo_tb_params *tb = NULL; + + if (op->mempool == NULL) { + rte_bbdev_log(ERR, "Invalid mempool pointer"); + return -1; + } + if (turbo_dec->input.data == NULL) { + rte_bbdev_log(ERR, "Invalid input pointer"); + return -1; + } + if (turbo_dec->hard_output.data == NULL) { + rte_bbdev_log(ERR, "Invalid hard_output pointer"); + return -1; + } + if (turbo_dec->rv_index > 3) { + rte_bbdev_log(ERR, + "rv_index (%u) is out of range 0 <= value <= 3", + turbo_dec->rv_index); + return -1; + } + if (turbo_dec->iter_min < 1) { + rte_bbdev_log(ERR, + "iter_min (%u) is less than 1", + turbo_dec->iter_min); + return -1; + } + if (turbo_dec->iter_max <= 2) { + rte_bbdev_log(ERR, + "iter_max (%u) is less than or equal to 2", + turbo_dec->iter_max); + return -1; + } + if (turbo_dec->iter_min > turbo_dec->iter_max) { + rte_bbdev_log(ERR, + "iter_min (%u) is greater than iter_max (%u)", + turbo_dec->iter_min, turbo_dec->iter_max); + return -1; + } + if (turbo_dec->code_block_mode != 0 && + turbo_dec->code_block_mode != 1) { + rte_bbdev_log(ERR, + "code_block_mode (%u) is out of range 0 <= value <= 1", + turbo_dec->code_block_mode); + return -1; + } + + if (turbo_dec->code_block_mode == 0) { + + if ((turbo_dec->op_flags & + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP) && + !(turbo_dec->op_flags & RTE_BBDEV_TURBO_CRC_TYPE_24B)) { + rte_bbdev_log(ERR, + "RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP should accompany RTE_BBDEV_TURBO_CRC_TYPE_24B"); + return -1; + } + + tb = &turbo_dec->tb_params; + if ((tb->k_neg < RTE_BBDEV_MIN_CB_SIZE + || tb->k_neg > RTE_BBDEV_MAX_CB_SIZE) + && tb->c_neg > 0) { + rte_bbdev_log(ERR, + "k_neg (%u) is out of range %u <= value <= %u", + tb->k_neg, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if ((tb->k_pos < RTE_BBDEV_MIN_CB_SIZE + || tb->k_pos > RTE_BBDEV_MAX_CB_SIZE) + && tb->c > tb->c_neg) { + rte_bbdev_log(ERR, + "k_pos (%u) is out of range %u <= value <= %u", + tb->k_pos, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->c_neg > (RTE_BBDEV_MAX_CODE_BLOCKS - 1)) + rte_bbdev_log(ERR, + "c_neg (%u) is out of range 0 <= value <= %u", + tb->c_neg, + RTE_BBDEV_MAX_CODE_BLOCKS - 1); + if (tb->c < 1 || tb->c > RTE_BBDEV_MAX_CODE_BLOCKS) { + rte_bbdev_log(ERR, + "c (%u) is out of range 1 <= value <= %u", + tb->c, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + if (tb->cab > tb->c) { + rte_bbdev_log(ERR, + "cab (%u) is greater than c (%u)", + tb->cab, tb->c); + return -1; + } + } else { + + if (turbo_dec->op_flags & RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP) { + rte_bbdev_log(ERR, + "RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP is invalid in CB-mode"); + return -1; + } + + cb = &turbo_dec->cb_params; + if (cb->k < RTE_BBDEV_MIN_CB_SIZE + || cb->k > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k (%u) is out of range %u <= value <= %u", + cb->k, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + } + + return 0; +} +#endif + +static inline int +enqueue_dec_one_op_cb(struct fpga_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input; + struct rte_mbuf *output; + int ret; + uint16_t k, kw, ring_offset; + uint32_t total_left, in_length, out_length, in_offset, out_offset; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_dec_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo decoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_dec.input.data; + output = op->turbo_dec.hard_output.data; + total_left = op->turbo_dec.input.length; + in_offset = op->turbo_dec.input.offset; + out_offset = op->turbo_dec.hard_output.offset; + + k = op->turbo_dec.cb_params.k; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + in_length = kw; + out_length = k >> 3; + + mbuf_append(output, output, out_length); + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_td_fill(op, &desc->dec_req, input, output, + in_length, k, in_offset, out_offset, ring_offset, 1); + if (unlikely(ret < 0)) + return ret; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_dma_dec_desc_debug_info(desc); +#endif + + /* Update lengths */ + total_left -= in_length; + op->turbo_dec.hard_output.length += out_length; + + if (total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + total_left, in_length); + return -1; + } + + return 1; +} + + +static inline int +enqueue_dec_one_op_tb(struct fpga_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset, uint8_t cbs_in_op) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input, *output_head, *output; + int ret; + uint8_t r, c; + uint16_t k, kw, in_length, out_length, ring_offset; + uint32_t mbuf_total_left, seg_total_left, in_offset, out_offset; + uint16_t current_enqueued_cbs = 0; + uint16_t crc24_overlap = 0; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_dec_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo decoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_dec.input.data; + output_head = output = op->turbo_dec.hard_output.data; + mbuf_total_left = op->turbo_dec.input.length; + in_offset = op->turbo_dec.input.offset; + out_offset = op->turbo_dec.hard_output.offset; + + if (!check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP)) + crc24_overlap = 24; + + c = op->turbo_dec.tb_params.c; + r = op->turbo_dec.tb_params.r; + + while (mbuf_total_left > 0 && r < c && input != NULL) { + seg_total_left = rte_pktmbuf_data_len(input) - in_offset; + k = (r < op->turbo_dec.tb_params.c_neg) ? + op->turbo_dec.tb_params.k_neg : + op->turbo_dec.tb_params.k_pos; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + + in_length = kw; + out_length = (k - crc24_overlap) >> 3; + + mbuf_append(output_head, output, out_length); + + if (seg_total_left < in_length) { + rte_bbdev_log(ERR, + "Partial CB found in a TB. FPGA Driver doesn't support scatter-gather operations!"); + return -1; + } + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_td_fill(op, &desc->dec_req, input, output, + in_length, k, in_offset, out_offset, + ring_offset, cbs_in_op); + if (unlikely(ret < 0)) + return ret; + + /* Update lengths */ + ret = rte_pktmbuf_trim(op->turbo_dec.hard_output.data, + (crc24_overlap >> 3)); +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (ret < 0) { + rte_bbdev_log(ERR, + "The length to remove is greater than the length of the last segment"); + return -EINVAL; + } +#endif + op->turbo_dec.hard_output.length += out_length; + mbuf_total_left -= in_length; + + /* Update offsets */ + if (seg_total_left == in_length) { + /* Go to the next mbuf */ + input = input->next; + output = output->next; + in_offset = 0; + out_offset = 0; + } else { + in_offset += in_length; + out_offset += out_length; + } + + r++; + desc_offset++; + current_enqueued_cbs++; + } + + if (mbuf_total_left > 0) { + rte_bbdev_log(ERR, + "Some date still left for processing: mbuf_total_left = %u", + mbuf_total_left); + return -1; + } + + return current_enqueued_cbs; +} + +static uint16_t +fpga_enqueue_enc(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num) +{ + uint8_t cbs_in_op; + uint16_t i, total_enqueued_cbs = 0; + int32_t avail; + int enqueued_cbs; + struct fpga_queue *q = q_data->queue_private; + union fpga_dma_desc *desc; + + /* Check if queue is not full */ + if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == + q->head_free_desc)) + return 0; + + /* Calculates available space */ + avail = (q->head_free_desc > q->tail) ? + q->head_free_desc - q->tail - 1 : + q->ring_ctrl_reg.ring_size + q->head_free_desc - q->tail - 1; + + for (i = 0; i < num; ++i) { + if (ops[i]->turbo_enc.code_block_mode == 0) { + cbs_in_op = get_num_cbs_in_op_enc(&ops[i]->turbo_enc); + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - cbs_in_op < 0)) + break; + avail -= cbs_in_op; + enqueued_cbs = enqueue_enc_one_op_tb(q, ops[i], + total_enqueued_cbs, cbs_in_op); + } else { + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - 1 < 0)) + break; + avail -= 1; + enqueued_cbs = enqueue_enc_one_op_cb(q, ops[i], + total_enqueued_cbs); + } + + if (enqueued_cbs < 0) + break; + + total_enqueued_cbs += enqueued_cbs; + + rte_bbdev_log_debug("enqueuing enc ops [%d/%d] | head %d | tail %d", + total_enqueued_cbs, num, + q->head_free_desc, q->tail); + } + + /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt + * only when all previous CBs were already processed. + */ + desc = q->ring_addr + ((q->tail + total_enqueued_cbs - 1) + & q->sw_ring_wrap_mask); + desc->enc_req.irq_en = q->irq_enable; + + fpga_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); + + /* Update stats */ + q_data->queue_stats.enqueued_count += i; + q_data->queue_stats.enqueue_err_count += num - i; + + return i; +} + +static uint16_t +fpga_enqueue_dec(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num) +{ + uint8_t cbs_in_op; + uint16_t i, total_enqueued_cbs = 0; + int32_t avail; + int enqueued_cbs; + struct fpga_queue *q = q_data->queue_private; + union fpga_dma_desc *desc; + + /* Check if queue is not full */ + if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == + q->head_free_desc)) + return 0; + + /* Calculates available space */ + avail = (q->head_free_desc > q->tail) ? + q->head_free_desc - q->tail - 1 : + q->ring_ctrl_reg.ring_size + q->head_free_desc - q->tail - 1; + + for (i = 0; i < num; ++i) { + if (ops[i]->turbo_dec.code_block_mode == 0) { + cbs_in_op = get_num_cbs_in_op_dec(&ops[i]->turbo_dec); + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - cbs_in_op < 0)) + break; + avail -= cbs_in_op; + enqueued_cbs = enqueue_dec_one_op_tb(q, ops[i], + total_enqueued_cbs, cbs_in_op); + } else { + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - 1 < 0)) + break; + avail -= 1; + enqueued_cbs = enqueue_dec_one_op_cb(q, ops[i], + total_enqueued_cbs); + } + + if (enqueued_cbs < 0) + break; + + total_enqueued_cbs += enqueued_cbs; + + rte_bbdev_log_debug("enqueuing dec ops [%d/%d] | head %d | tail %d", + total_enqueued_cbs, num, + q->head_free_desc, q->tail); + } + + /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt + * only when all previous CBs were already processed. + */ + desc = q->ring_addr + ((q->tail + total_enqueued_cbs - 1) + & q->sw_ring_wrap_mask); + desc->dec_req.irq_en = q->irq_enable; + + fpga_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); + + /* Update stats */ + q_data->queue_stats.enqueued_count += i; + q_data->queue_stats.enqueue_err_count += num - i; + + return i; +} + +static inline int +dequeue_enc_one_op_cb(struct fpga_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + int desc_error = 0; + + /* Set current desc */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /*check if done */ + if (desc->enc_req.done == 0) + return -1; + + /* make sure the response is read atomically */ + rte_smp_rmb(); + + rte_bbdev_log_debug("DMA response desc %p", desc); + + *op = desc->enc_req.op_addr; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + (*op)->status = desc_error << RTE_BBDEV_DATA_ERROR; + + return 1; +} + +static inline int +dequeue_enc_one_op_tb(struct fpga_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + uint8_t cbs_in_op, cb_idx; + int desc_error = 0; + int status = 0; + + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify if done bit is set */ + if (desc->enc_req.done == 0) + return -1; + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + /* Verify if done bit in all CBs is set */ + cbs_in_op = desc->enc_req.cbs_in_op; + for (cb_idx = 1; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + if (desc->enc_req.done == 0) + return -1; + } + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + for (cb_idx = 0; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + status |= desc_error << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log_debug("DMA response desc %p", desc); + } + + *op = desc->enc_req.op_addr; + (*op)->status = status; + return cbs_in_op; +} + +static inline int +dequeue_dec_one_op_cb(struct fpga_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + int desc_error = 0; + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify done bit is set */ + if (desc->dec_req.done == 0) + return -1; + + /* make sure the response is read atomically */ + rte_smp_rmb(); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_dma_dec_desc_debug_info(desc); + +#endif + + *op = desc->dec_req.op_addr; + /* FPGA reports in half-iterations, from 0 to 31. get ceiling */ + (*op)->turbo_dec.iter_count = (desc->dec_req.iter + 2) >> 1; + /* crc_pass = 0 when decoder fails */ + (*op)->status = !(desc->dec_req.crc_pass) << RTE_BBDEV_CRC_ERROR; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + (*op)->status |= desc_error << RTE_BBDEV_DATA_ERROR; + return 1; +} + +static inline int +dequeue_dec_one_op_tb(struct fpga_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + uint8_t cbs_in_op, cb_idx, iter_count = 0; + int status = 0; + int desc_error = 0; + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify if done bit is set */ + if (desc->dec_req.done == 0) + return -1; + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + /* Verify if done bit in all CBs is set */ + cbs_in_op = desc->dec_req.cbs_in_op; + for (cb_idx = 1; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + if (desc->dec_req.done == 0) + return -1; + } + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + for (cb_idx = 0; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + /* get max iter_count for all CBs in op */ + iter_count = RTE_MAX(iter_count, (uint8_t) desc->dec_req.iter); + /* crc_pass = 0 when decoder fails, one fails all */ + status |= !(desc->dec_req.crc_pass) << RTE_BBDEV_CRC_ERROR; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + status |= desc_error << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log_debug("DMA response desc %p", desc); + } + + *op = desc->dec_req.op_addr; + + /* FPGA reports in half-iterations, get ceiling */ + (*op)->turbo_dec.iter_count = (iter_count + 2) >> 1; + (*op)->status = status; + return cbs_in_op; +} + +static uint16_t +fpga_dequeue_enc(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num) +{ + struct fpga_queue *q = q_data->queue_private; + uint32_t avail = (q->tail - q->head_free_desc) & q->sw_ring_wrap_mask; + uint16_t i; + uint16_t dequeued_cbs = 0; + struct rte_bbdev_enc_op *op; + int ret; + + for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { + op = (q->ring_addr + ((q->head_free_desc + dequeued_cbs) + & q->sw_ring_wrap_mask))->enc_req.op_addr; + if (op->turbo_enc.code_block_mode == 0) + ret = dequeue_enc_one_op_tb(q, &ops[i], dequeued_cbs); + else + ret = dequeue_enc_one_op_cb(q, &ops[i], dequeued_cbs); + + if (ret < 0) + break; + + dequeued_cbs += ret; + + rte_bbdev_log_debug("dequeuing enc ops [%d/%d] | head %d | tail %d", + dequeued_cbs, num, q->head_free_desc, q->tail); + } + + /* Update head */ + q->head_free_desc = (q->head_free_desc + dequeued_cbs) & + q->sw_ring_wrap_mask; + + /* Update stats */ + q_data->queue_stats.dequeued_count += i; + + return i; +} + +static uint16_t +fpga_dequeue_dec(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num) +{ + struct fpga_queue *q = q_data->queue_private; + uint32_t avail = (q->tail - q->head_free_desc) & q->sw_ring_wrap_mask; + uint16_t i; + uint16_t dequeued_cbs = 0; + struct rte_bbdev_dec_op *op; + int ret; + + for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { + op = (q->ring_addr + ((q->head_free_desc + dequeued_cbs) + & q->sw_ring_wrap_mask))->dec_req.op_addr; + if (op->turbo_dec.code_block_mode == 0) + ret = dequeue_dec_one_op_tb(q, &ops[i], dequeued_cbs); + else + ret = dequeue_dec_one_op_cb(q, &ops[i], dequeued_cbs); + + if (ret < 0) + break; + + dequeued_cbs += ret; + + rte_bbdev_log_debug("dequeuing dec ops [%d/%d] | head %d | tail %d", + dequeued_cbs, num, q->head_free_desc, q->tail); + } + + /* Update head */ + q->head_free_desc = (q->head_free_desc + dequeued_cbs) & + q->sw_ring_wrap_mask; + + /* Update stats */ + q_data->queue_stats.dequeued_count += i; + + return i; +} + +/* Initialization Function */ +static void +fpga_lte_fec_init(struct rte_bbdev *dev) +{ + struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev->device); + + dev->dev_ops = &fpga_ops; + dev->enqueue_enc_ops = fpga_enqueue_enc; + dev->enqueue_dec_ops = fpga_enqueue_dec; + dev->dequeue_enc_ops = fpga_dequeue_enc; + dev->dequeue_dec_ops = fpga_dequeue_dec; + + ((struct fpga_lte_fec_device *) dev->data->dev_private)->pf_device = + !strcmp(dev->device->driver->name, + RTE_STR(FPGA_LTE_FEC_PF_DRIVER_NAME)); + ((struct fpga_lte_fec_device *) dev->data->dev_private)->mmio_base = + pci_dev->mem_resource[0].addr; + + rte_bbdev_log_debug( + "Init device %s [%s] @ virtaddr %p phyaddr %#"PRIx64, + dev->device->driver->name, dev->data->name, + (void *)pci_dev->mem_resource[0].addr, + pci_dev->mem_resource[0].phys_addr); +} + +static int +fpga_lte_fec_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct rte_bbdev *bbdev = NULL; + char dev_name[RTE_BBDEV_NAME_MAX_LEN]; + + if (pci_dev == NULL) { + rte_bbdev_log(ERR, "NULL PCI device"); + return -EINVAL; + } + + rte_pci_device_name(&pci_dev->addr, dev_name, sizeof(dev_name)); + + /* Allocate memory to be used privately by drivers */ + bbdev = rte_bbdev_allocate(pci_dev->device.name); + if (bbdev == NULL) + return -ENODEV; + + /* allocate device private memory */ + bbdev->data->dev_private = rte_zmalloc_socket(dev_name, + sizeof(struct fpga_lte_fec_device), RTE_CACHE_LINE_SIZE, + pci_dev->device.numa_node); + + if (bbdev->data->dev_private == NULL) { + rte_bbdev_log(CRIT, + "Allocate of %zu bytes for device \"%s\" failed", + sizeof(struct fpga_lte_fec_device), dev_name); + rte_bbdev_release(bbdev); + return -ENOMEM; + } + + /* Fill HW specific part of device structure */ + bbdev->device = &pci_dev->device; + bbdev->intr_handle = &pci_dev->intr_handle; + bbdev->data->socket_id = pci_dev->device.numa_node; + + /* Invoke FEC FPGA device initialization function */ + fpga_lte_fec_init(bbdev); + + rte_bbdev_log_debug("bbdev id = %u [%s]", + bbdev->data->dev_id, dev_name); + + struct fpga_lte_fec_device *d = bbdev->data->dev_private; + uint32_t version_id = fpga_reg_read_32(d->mmio_base, + FPGA_LTE_FEC_VERSION_ID); + rte_bbdev_log(INFO, "FEC FPGA RTL v%u.%u", + ((uint16_t)(version_id >> 16)), ((uint16_t)version_id)); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (!strcmp(bbdev->device->driver->name, + RTE_STR(FPGA_LTE_FEC_PF_DRIVER_NAME))) + print_static_reg_debug_info(d->mmio_base); +#endif + return 0; +} + +static int +fpga_lte_fec_remove(struct rte_pci_device *pci_dev) +{ + struct rte_bbdev *bbdev; + int ret; + uint8_t dev_id; + + if (pci_dev == NULL) + return -EINVAL; + + /* Find device */ + bbdev = rte_bbdev_get_named_dev(pci_dev->device.name); + if (bbdev == NULL) { + rte_bbdev_log(CRIT, + "Couldn't find HW dev \"%s\" to uninitialise it", + pci_dev->device.name); + return -ENODEV; + } + dev_id = bbdev->data->dev_id; + + /* free device private memory before close */ + rte_free(bbdev->data->dev_private); + + /* Close device */ + ret = rte_bbdev_close(dev_id); + if (ret < 0) + rte_bbdev_log(ERR, + "Device %i failed to close during uninit: %i", + dev_id, ret); + + /* release bbdev from library */ + ret = rte_bbdev_release(bbdev); + if (ret) + rte_bbdev_log(ERR, "Device %i failed to uninit: %i", dev_id, + ret); + + rte_bbdev_log_debug("Destroyed bbdev = %u", dev_id); + + return 0; +} + +static inline void +set_default_fpga_conf(struct fpga_lte_fec_conf *def_conf) +{ + /* clear default configuration before initialization */ + memset(def_conf, 0, sizeof(struct fpga_lte_fec_conf)); + /* Set pf mode to true */ + def_conf->pf_mode_en = true; + + /* Set ratio between UL and DL to 1:1 (unit of weight is 3 CBs) */ + def_conf->ul_bandwidth = 3; + def_conf->dl_bandwidth = 3; + + /* Set Load Balance Factor to 64 */ + def_conf->dl_load_balance = 64; + def_conf->ul_load_balance = 64; +} + +/* Initial configuration of FPGA LTE FEC device */ +int +fpga_lte_fec_configure(const char *dev_name, + const struct fpga_lte_fec_conf *conf) +{ + uint32_t payload_32, address; + uint16_t payload_16; + uint8_t payload_8; + uint16_t q_id, vf_id, total_q_id, total_ul_q_id, total_dl_q_id; + struct rte_bbdev *bbdev = rte_bbdev_get_named_dev(dev_name); + struct fpga_lte_fec_conf def_conf; + + if (bbdev == NULL) { + rte_bbdev_log(ERR, + "Invalid dev_name (%s), or device is not yet initialised", + dev_name); + return -ENODEV; + } + + struct fpga_lte_fec_device *d = bbdev->data->dev_private; + + if (conf == NULL) { + rte_bbdev_log(ERR, + "FPGA Configuration was not provided. Default configuration will be loaded."); + set_default_fpga_conf(&def_conf); + conf = &def_conf; + } + + /* + * Configure UL:DL ratio. + * [7:0]: UL weight + * [15:8]: DL weight + */ + payload_16 = (conf->dl_bandwidth << 8) | conf->ul_bandwidth; + address = FPGA_LTE_FEC_CONFIGURATION; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Clear all queues registers */ + payload_32 = FPGA_INVALID_HW_QUEUE_ID; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + address = (q_id << 2) + FPGA_LTE_FEC_QUEUE_MAP; + fpga_reg_write_32(d->mmio_base, address, payload_32); + } + + /* + * If PF mode is enabled allocate all queues for PF only. + * + * For VF mode each VF can have different number of UL and DL queues. + * Total number of queues to configure cannot exceed FPGA + * capabilities - 64 queues - 32 queues for UL and 32 queues for DL. + * Queues mapping is done according to configuration: + * + * UL queues: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | ... | 0 | + * | conf->vf_dl_queues_number[0] - 1 | 0 | + * | conf->vf_dl_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_dl_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_dl_queues_number[7] - 1 | 7 | + * + * DL queues: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | ... | 0 | + * | conf->vf_ul_queues_number[0] - 1 | 0 | + * | conf->vf_ul_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_ul_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_ul_queues_number[7] - 1 | 7 | + * + * Example of configuration: + * conf->vf_ul_queues_number[0] = 4; -> 4 UL queues for VF0 + * conf->vf_dl_queues_number[0] = 4; -> 4 DL queues for VF0 + * conf->vf_ul_queues_number[1] = 2; -> 2 UL queues for VF1 + * conf->vf_dl_queues_number[1] = 2; -> 2 DL queues for VF1 + * + * UL: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | 1 | 0 | + * | 2 | 0 | + * | 3 | 0 | + * | 4 | 1 | + * | 5 | 1 | + * + * DL: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | 33 | 0 | + * | 34 | 0 | + * | 35 | 0 | + * | 36 | 1 | + * | 37 | 1 | + */ + if (conf->pf_mode_en) { + payload_32 = 0x1; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + address = (q_id << 2) + FPGA_LTE_FEC_QUEUE_MAP; + fpga_reg_write_32(d->mmio_base, address, payload_32); + } + } else { + /* Calculate total number of UL and DL queues to configure */ + total_ul_q_id = total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + total_ul_q_id += conf->vf_ul_queues_number[vf_id]; + total_dl_q_id += conf->vf_dl_queues_number[vf_id]; + } + total_q_id = total_dl_q_id + total_ul_q_id; + /* + * Check if total number of queues to configure does not exceed + * FPGA capabilities (64 queues - 32 UL and 32 DL queues) + */ + if ((total_ul_q_id > FPGA_NUM_UL_QUEUES) || + (total_dl_q_id > FPGA_NUM_DL_QUEUES) || + (total_q_id > FPGA_TOTAL_NUM_QUEUES)) { + rte_bbdev_log(ERR, + "FPGA Configuration failed. Too many queues to configure: UL_Q %u, DL_Q %u, FPGA_Q %u", + total_ul_q_id, total_dl_q_id, + FPGA_TOTAL_NUM_QUEUES); + return -EINVAL; + } + total_ul_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_ul_queues_number[vf_id]; + ++q_id, ++total_ul_q_id) { + address = (total_ul_q_id << 2) + + FPGA_LTE_FEC_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_reg_write_32(d->mmio_base, address, + payload_32); + } + } + total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_dl_queues_number[vf_id]; + ++q_id, ++total_dl_q_id) { + address = ((total_dl_q_id + FPGA_NUM_UL_QUEUES) + << 2) + FPGA_LTE_FEC_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_reg_write_32(d->mmio_base, address, + payload_32); + } + } + } + + /* Setting Load Balance Factor */ + payload_16 = (conf->dl_load_balance << 8) | (conf->ul_load_balance); + address = FPGA_LTE_FEC_LOAD_BALANCE_FACTOR; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting length of ring descriptor entry */ + payload_16 = FPGA_RING_DESC_ENTRY_LENGTH; + address = FPGA_LTE_FEC_RING_DESC_LEN; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting FLR timeout value */ + payload_16 = conf->flr_time_out; + address = FPGA_LTE_FEC_FLR_TIME_OUT; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Queue PF/VF mapping table is ready */ + payload_8 = 0x1; + address = FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE; + fpga_reg_write_8(d->mmio_base, address, payload_8); + + rte_bbdev_log_debug("PF FPGA LTE FEC configuration complete for %s", + dev_name); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_static_reg_debug_info(d->mmio_base); +#endif + return 0; +} + +/* FPGA LTE FEC PCI PF address map */ +static struct rte_pci_id pci_id_fpga_lte_fec_pf_map[] = { + { + RTE_PCI_DEVICE(FPGA_LTE_FEC_VENDOR_ID, + FPGA_LTE_FEC_PF_DEVICE_ID) + }, + {.device_id = 0}, +}; + +static struct rte_pci_driver fpga_lte_fec_pci_pf_driver = { + .probe = fpga_lte_fec_probe, + .remove = fpga_lte_fec_remove, + .id_table = pci_id_fpga_lte_fec_pf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING +}; + +/* FPGA LTE FEC PCI VF address map */ +static struct rte_pci_id pci_id_fpga_lte_fec_vf_map[] = { + { + RTE_PCI_DEVICE(FPGA_LTE_FEC_VENDOR_ID, + FPGA_LTE_FEC_VF_DEVICE_ID) + }, + {.device_id = 0}, +}; + +static struct rte_pci_driver fpga_lte_fec_pci_vf_driver = { + .probe = fpga_lte_fec_probe, + .remove = fpga_lte_fec_remove, + .id_table = pci_id_fpga_lte_fec_vf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING +}; + + +RTE_PMD_REGISTER_PCI(FPGA_LTE_FEC_PF_DRIVER_NAME, fpga_lte_fec_pci_pf_driver); +RTE_PMD_REGISTER_PCI_TABLE(FPGA_LTE_FEC_PF_DRIVER_NAME, + pci_id_fpga_lte_fec_pf_map); +RTE_PMD_REGISTER_PCI(FPGA_LTE_FEC_VF_DRIVER_NAME, fpga_lte_fec_pci_vf_driver); +RTE_PMD_REGISTER_PCI_TABLE(FPGA_LTE_FEC_VF_DRIVER_NAME, + pci_id_fpga_lte_fec_vf_map); + +RTE_INIT(fpga_lte_fec_init_log) +{ + fpga_lte_fec_logtype = rte_log_register("pmd.bb.fpga_lte_fec"); + if (fpga_lte_fec_logtype >= 0) +#ifdef RTE_LIBRTE_BBDEV_DEBUG + rte_log_set_level(fpga_lte_fec_logtype, RTE_LOG_DEBUG); +#else + rte_log_set_level(fpga_lte_fec_logtype, RTE_LOG_NOTICE); +#endif +} diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h new file mode 100644 index 0000000..2c6952e --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2018 Intel Corporation + */ + +#ifndef _FPGA_LTE_FEC_H_ +#define _FPGA_LTE_FEC_H_ + +#include <stdint.h> +#include <stdbool.h> + +/** + * @file fpga_lte_fec.h + * + * Interface for Intel(R) FGPA LTE FEC device configuration at the host level, + * directly accessible by the application. + * Configuration related to LTE Turbo coding functionality is done through + * librte_bbdev library. + * + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + */ + +#ifdef __cplusplus +extern "C" { +#endif + +/**< Number of Virtual Functions FGPA 4G FEC supports */ +#define FPGA_LTE_FEC_NUM_VFS 8 + +/** + * Structure to pass FPGA 4G FEC configuration. + */ +struct fpga_lte_fec_conf { + /**< 1 if PF is used for dataplane, 0 for VFs */ + bool pf_mode_en; + /**< Number of UL queues per VF */ + uint8_t vf_ul_queues_number[FPGA_LTE_FEC_NUM_VFS]; + /**< Number of DL queues per VF */ + uint8_t vf_dl_queues_number[FPGA_LTE_FEC_NUM_VFS]; + /**< UL bandwidth. Needed for schedule algorithm */ + uint8_t ul_bandwidth; + /**< DL bandwidth. Needed for schedule algorithm */ + uint8_t dl_bandwidth; + /**< UL Load Balance */ + uint8_t ul_load_balance; + /**< DL Load Balance */ + uint8_t dl_load_balance; + /**< FLR timeout value */ + uint16_t flr_time_out; +}; + +/** + * Configure Intel(R) FPGA LTE FEC device + * + * @param dev_name + * The name of the device. This is the short form of PCI BDF, e.g. 00:01.0. + * It can also be retrieved for a bbdev device from the dev_name field in the + * rte_bbdev_info structure returned by rte_bbdev_info_get(). + * @param conf + * Configuration to apply to FPGA 4G FEC. + * + * @return + * Zero on success, negative value on failure. + */ +int +fpga_lte_fec_configure(const char *dev_name, + const struct fpga_lte_fec_conf *conf); + +#ifdef __cplusplus +} +#endif + +#endif /* _FPGA_LTE_FEC_H_ */ diff --git a/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map new file mode 100644 index 0000000..1a99f8d --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map @@ -0,0 +1,3 @@ +DPDK_18.08 { + local: *; +}; diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 7c9b4b5..21d32a2 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -218,6 +218,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_NETVSC_PMD) += -lrte_pmd_netvsc ifeq ($(CONFIG_RTE_LIBRTE_BBDEV),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += -lrte_pmd_fpga_lte_fec # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v3] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 1/3] " Nicolas Chautru @ 2019-06-06 9:04 ` Nicolas Chautru 2019-06-06 9:04 ` Nicolas Chautru 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 1 sibling, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-06 9:04 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Update since v2 : Squashing 3 previous patches into one as recommended. This is adding a new PMD driver for BBDEV device based on FPGA implementation on PAC N3000 HW to provide FEC 4G acceleration. v1 was shared earlier on this patchwork : https://patches.dpdk.org/patch/53409/ The existing BBDEV test framework is still supported. Main updates based on feedback during the last v1 were: - correct the documentation which was arguably misleading in hinting that ICC was required. There is no dependency whatsoever. - add meson build support Note that a number of other BBDEV patches are in v1 here and will be resubmitted for 19.08 rc1 https://patches.dpdk.org/project/dpdk/list/?series=4657 Nicolas Chautru (1): baseband/fpga_lte_fec: adding driver for FEC on FPGA config/common_base | 6 + doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++ doc/guides/bbdevs/index.rst | 1 + drivers/baseband/Makefile | 2 + drivers/baseband/fpga_lte_fec/Makefile | 29 + drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 ++++++++++++++++++++ drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + drivers/baseband/fpga_lte_fec/meson.build | 7 + .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + drivers/baseband/meson.build | 2 +- mk/rte.app.mk | 1 + 11 files changed, 3115 insertions(+), 1 deletion(-) create mode 100644 doc/guides/bbdevs/fpga_lte_fec.rst create mode 100644 drivers/baseband/fpga_lte_fec/Makefile create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h create mode 100644 drivers/baseband/fpga_lte_fec/meson.build create mode 100644 drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v3] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-06 9:04 ` [dpdk-dev] [PATCH v3] " Nicolas Chautru @ 2019-06-06 9:04 ` Nicolas Chautru 2019-06-08 0:17 ` [dpdk-dev] [PATCH v4] " Nicolas Chautru 2019-06-10 17:01 ` Nicolas Chautru 0 siblings, 2 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-06 9:04 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Supports for FEC 4G PMD Driver on FPGA card PAC N3000 Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 6 + doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++ doc/guides/bbdevs/index.rst | 1 + drivers/baseband/Makefile | 2 + drivers/baseband/fpga_lte_fec/Makefile | 29 + drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 ++++++++++++++++++++ drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + drivers/baseband/fpga_lte_fec/meson.build | 7 + .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + drivers/baseband/meson.build | 2 +- mk/rte.app.mk | 1 + 11 files changed, 3115 insertions(+), 1 deletion(-) create mode 100644 doc/guides/bbdevs/fpga_lte_fec.rst create mode 100644 drivers/baseband/fpga_lte_fec/Makefile create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h create mode 100644 drivers/baseband/fpga_lte_fec/meson.build create mode 100644 drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map diff --git a/config/common_base b/config/common_base index 6b96e0e..ae52028 100644 --- a/config/common_base +++ b/config/common_base @@ -521,6 +521,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y # EXPERIMENTAL: API may change without prior notice # CONFIG_RTE_LIBRTE_BBDEV=y +CONFIG_RTE_LIBRTE_BBDEV_DEBUG=n CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y @@ -535,6 +536,11 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n # +# Compile PMD for Intel FPGA LTE FEC bbdev device +# +CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC=y + +# # Compile generic crypto device library # CONFIG_RTE_LIBRTE_CRYPTODEV=y diff --git a/doc/guides/bbdevs/fpga_lte_fec.rst b/doc/guides/bbdevs/fpga_lte_fec.rst new file mode 100644 index 0000000..71b058c --- /dev/null +++ b/doc/guides/bbdevs/fpga_lte_fec.rst @@ -0,0 +1,318 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation + +Intel(R) FPGA LTE FEC Poll Mode Driver +====================================== + +The BBDEV FPGA LTE FEC poll mode driver (PMD) supports an FPGA implementation of a VRAN +Turbo Encode / Decode LTE wireless acceleration function, using Intel's PCI-e and FPGA +based Vista Creek device. + +Features +-------- + +FPGA LTE FEC PMD supports the following features: + +- Turbo Encode in the DL with total throughput of 4.5 Gbits/s +- Turbo Decode in the UL with total throughput of 1.5 Gbits/s assuming 8 decoder iterations +- 8 VFs per PF (physical device) +- Maximum of 32 UL queues per VF +- Maximum of 32 DL queues per VF +- PCIe Gen-3 x8 Interface +- MSI-X +- SR-IOV + + +FPGA LTE FEC PMD supports the following BBDEV capabilities: + +* For the turbo encode operation: + - ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` : set to attach CRC24B to CB(s) + - ``RTE_BBDEV_TURBO_RATE_MATCH`` : if set then do not do Rate Match bypass + - ``RTE_BBDEV_TURBO_ENC_INTERRUPTS`` : set for encoder dequeue interrupts + + +* For the turbo decode operation: + - ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` : check CRC24B from CB(s) + - ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` : perform subblock de-interleave + - ``RTE_BBDEV_TURBO_DEC_INTERRUPTS`` : set for decoder dequeue interrupts + - ``RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN`` : set if negative LLR encoder i/p is supported + - ``RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP`` : keep CRC24B bits appended while decoding + + +Limitations +----------- + +FPGA LTE FEC does not support the following: + +- Scatter-Gather function + + +Installation +-------------- + +Section 3 of the DPDK manual provides instuctions on installing and compiling DPDK. The +default set of bbdev compile flags may be found in config/common_base, where for example +the flag to build the FPGA LTE FEC device, ``CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC``, is already +set. It is assumed DPDK has been compiled using for instance: + +.. code-block:: console + + make install T=x86_64-native-linuxapp-gcc + + +DPDK requires hugepages to be configured as detailed in section 2 of the DPDK manual. +The bbdev test application has been tested with a configuration 40 x 1GB hugepages. The +hugepage configuration of a server may be examined using: + +.. code-block:: console + + grep Huge* /proc/meminfo + + +Initialization +-------------- + +When the device first powers up, its PCI Physical Functions (PF) can be listed through this command: + +.. code-block:: console + + sudo lspci -vd1172:5052 + +The physical and virtual functions are compatible with Linux UIO drivers: +``vfio`` and ``igb_uio``. However, in order to work the FPGA LTE FEC device firstly needs +to be bound to one of these linux drivers through DPDK. + + +Bind PF UIO driver(s) +~~~~~~~~~~~~~~~~~~~~~ + +Install the DPDK igb_uio driver, bind it with the PF PCI device ID and use +``lspci`` to confirm the PF device is under use by ``igb_uio`` DPDK UIO driver. + +The igb_uio driver may be bound to the PF PCI device using one of three methods: + + +1. PCI functions (physical or virtual, depending on the use case) can be bound to +the UIO driver by repeating this command for every function. + +.. code-block:: console + + cd <dpdk-top-level-directory> + insmod ./build/kmod/igb_uio.ko + echo "1172 5052" > /sys/bus/pci/drivers/igb_uio/new_id + lspci -vd1172: + + +2. Another way to bind PF with DPDK UIO driver is by using the ``dpdk-devbind.py`` tool + +.. code-block:: console + + cd <dpdk-top-level-directory> + ./usertools/dpdk-devbind.py -b igb_uio 0000:06:00.0 + +where the PCI device ID (example: 0000:06:00.0) is obtained using lspci -vd1172: + + +3. A third way to bind is to use ``dpdk-setup.sh`` tool + +.. code-block:: console + + cd <dpdk-top-level-directory> + ./usertools/dpdk-setup.sh + + select 'Bind Ethernet/Crypto/Baseband device to IGB UIO module' + or + select 'Bind Ethernet/Crypto/Baseband device to VFIO module' depending on driver required + enter PCI device ID + select 'Display current Ethernet/Crypto/Baseband device settings' to confirm binding + + +In the same way the FPGA LTE FEC PF can be bound with vfio, but vfio driver does not +support SR-IOV configuration right out of the box, so it will need to be patched. + + +Enable Virtual Functions +~~~~~~~~~~~~~~~~~~~~~~~~ + +Now, it should be visible in the printouts that PCI PF is under igb_uio control +"``Kernel driver in use: igb_uio``" + +To show the number of available VFs on the device, read ``sriov_totalvfs`` file.. + +.. code-block:: console + + cat /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_totalvfs + + where 0000\:<b>\:<d>.<f> is the PCI device ID + + +To enable VFs via igb_uio, echo the number of virtual functions intended to +enable to ``max_vfs`` file.. + +.. code-block:: console + + echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/max_vfs + + +Afterwards, all VFs must be bound to appropriate UIO drivers as required, same +way it was done with the physical function previously. + +Enabling SR-IOV via vfio driver is pretty much the same, except that the file +name is different: + +.. code-block:: console + + echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_numvfs + + +Configure the VFs through PF +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The PCI virtual functions must be configured before working or getting assigned +to VMs/Containers. The configuration involves allocating the number of hardware +queues, priorities, load balance, bandwidth and other settings necessary for the +device to perform FEC functions. + +This configuration needs to be executed at least once after reboot or PCI FLR and can +be achieved by using the function ``fpga_lte_fec_configure()``, which sets up the +parameters defined in ``fpga_lte_fec_conf`` structure: + +.. code-block:: c + + struct fpga_lte_fec_conf { + bool pf_mode_en; + uint8_t vf_ul_queues_number[FPGA_LTE_FEC_NUM_VFS]; + uint8_t vf_dl_queues_number[FPGA_LTE_FEC_NUM_VFS]; + uint8_t ul_bandwidth; + uint8_t dl_bandwidth; + uint8_t ul_load_balance; + uint8_t dl_load_balance; + uint16_t flr_time_out; + }; + +- ``pf_mode_en``: identifies whether only PF is to be used, or the VFs. PF and + VFs are mutually exclusive and cannot run simultaneously. + Set to 1 for PF mode enabled. + If PF mode is enabled all queues available in the device are assigned + exclusively to PF and 0 queues given to VFs. + +- ``vf_*l_queues_number``: defines the hardware queue mapping for every VF. + +- ``*l_bandwidth``: in case of congestion on PCIe interface. The device + allocates different bandwidth to UL and DL. The weight is configured by this + setting. The unit of weight is 3 code blocks. For example, if the code block + cbps (code block per second) ratio between UL and DL is 12:1, then the + configuration value should be set to 36:3. The schedule algorithm is based + on code block regardless the length of each block. + +- ``*l_load_balance``: hardware queues are load-balanced in a round-robin + fashion. Queues get filled first-in first-out until they reach a pre-defined + watermark level, if exceeded, they won't get assigned new code blocks.. + This watermark is defined by this setting. + + If all hardware queues exceeds the watermark, no code blocks will be + streamed in from UL/DL code block FIFO. + +- ``flr_time_out``: specifies how many 16.384us to be FLR time out. The + time_out = flr_time_out x 16.384us. For instance, if you want to set 10ms for + the FLR time out then set this setting to 0x262=610. + + +An example configuration code calling the function ``fpga_lte_fec_configure()`` is shown +below: + +.. code-block:: c + + struct fpga_lte_fec_conf conf; + unsigned int i; + + memset(&conf, 0, sizeof(struct fpga_lte_fec_conf)); + conf.pf_mode_en = 1; + + for (i = 0; i < FPGA_LTE_FEC_NUM_VFS; ++i) { + conf.vf_ul_queues_number[i] = 4; + conf.vf_dl_queues_number[i] = 4; + } + conf.ul_bandwidth = 12; + conf.dl_bandwidth = 5; + conf.dl_load_balance = 64; + conf.ul_load_balance = 64; + + /* setup FPGA PF */ + ret = fpga_lte_fec_configure(info->dev_name, &conf); + TEST_ASSERT_SUCCESS(ret, + "Failed to configure 4G FPGA PF for bbdev %s", + info->dev_name); + + +Test Application +---------------- + +BBDEV provides a test application, ``test-bbdev.py`` and range of test data for testing +the functionality of FPGA LTE FEC turbo encode and turbo decode, depending on the device's +capabilities. The test application is located under app->test-bbdev folder and has the +following options: + +.. code-block:: console + + "-p", "--testapp-path": specifies path to the bbdev test app. + "-e", "--eal-params" : EAL arguments which are passed to the test app. + "-t", "--timeout" : Timeout in seconds (default=300). + "-c", "--test-cases" : Defines test cases to run. Run all if not specified. + "-v", "--test-vector" : Test vector path (default=dpdk_path+/app/test-bbdev/test_vectors/bbdev_null.data). + "-n", "--num-ops" : Number of operations to process on device (default=32). + "-b", "--burst-size" : Operations enqueue/dequeue burst size (default=32). + "-l", "--num-lcores" : Number of lcores to run (default=16). + "-i", "--init-device" : Initialise PF device with default values. + + +To execute the test application tool using simple turbo decode or turbo encode data, +type one of the following: + +.. code-block:: console + + ./test-bbdev.py -c validation -n 64 -b 8 -v ./turbo_dec_default.data + ./test-bbdev.py -c validation -n 64 -b 8 -v ./turbo_enc_default.data + + +The test application ``test-bbdev.py``, supports the ability to configure the PF device with +a default set of values, if the "-i" or "- -init-device" option is included. The default values +are defined in test_bbdev_perf.c as: + +- VF_UL_QUEUE_VALUE 4 +- VF_DL_QUEUE_VALUE 4 +- UL_BANDWIDTH 3 +- DL_BANDWIDTH 3 +- UL_LOAD_BALANCE 128 +- DL_LOAD_BALANCE 128 +- FLR_TIMEOUT 610 + + +Test Vectors +~~~~~~~~~~~~ + +In addition to the simple turbo decoder and turbo encoder tests, bbdev also provides +a range of additional tests under the test_vectors folder, which may be useful. The results +of these tests will depend on the FPGA LTE FEC capabilities: + +* turbo decoder tests: + - ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_high_snr.data`` + - ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_low_snr.data`` + - ``turbo_dec_c1_k6144_r0_e34560_negllr.data`` + - ``turbo_dec_c1_k6144_r0_e34560_sbd_negllr.data`` + - ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr_crc24b.data`` + - ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr.data`` + + +* turbo encoder tests: + - ``turbo_enc_c1_k40_r0_e1190_rm.data`` + - ``turbo_enc_c1_k40_r0_e1194_rm.data`` + - ``turbo_enc_c1_k40_r0_e1196_rm.data`` + - ``turbo_enc_c1_k40_r0_e272_rm.data`` + - ``turbo_enc_c1_k6144_r0_e18444.data`` + - ``turbo_enc_c1_k6144_r0_e32256_crc24b_rm.data`` + - ``turbo_enc_c2_k5952_r0_e17868_crc24b.data`` + - ``turbo_enc_c3_k4800_r2_e14412_crc24b.data`` + - ``turbo_enc_c4_k4800_r2_e14412_crc24b.data`` + + diff --git a/doc/guides/bbdevs/index.rst b/doc/guides/bbdevs/index.rst index 93276ed..005b95e 100644 --- a/doc/guides/bbdevs/index.rst +++ b/doc/guides/bbdevs/index.rst @@ -10,3 +10,4 @@ Baseband Device Drivers null turbo_sw + fpga_lte_fec diff --git a/drivers/baseband/Makefile b/drivers/baseband/Makefile index 4ec83b0..ceffc7d 100644 --- a/drivers/baseband/Makefile +++ b/drivers/baseband/Makefile @@ -10,5 +10,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += null DEPDIRS-null = $(core-libs) DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += turbo_sw DEPDIRS-turbo_sw = $(core-libs) +DIRS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += fpga_lte_fec +DEPDIRS-fpga_lte_fec = $(core-libs) include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/drivers/baseband/fpga_lte_fec/Makefile b/drivers/baseband/fpga_lte_fec/Makefile new file mode 100644 index 0000000..e478d84 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/Makefile @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2018 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_pmd_fpga_lte_fec.a + +# build flags +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring +LDLIBS += -lrte_bbdev +LDLIBS += -lrte_pci -lrte_bus_pci + +# versioning export map +EXPORT_MAP := rte_pmd_bbdev_fpga_lte_fec_version.map + +# library version +LIBABIVER := 1 + +# library source files +SRCS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += fpga_lte_fec.c + +# export include files +SYMLINK-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC)-include += fpga_lte_fec.h + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c new file mode 100644 index 0000000..856b68e --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c @@ -0,0 +1,2674 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2018 Intel Corporation + */ + +#include <unistd.h> + +#include <rte_common.h> +#include <rte_log.h> +#include <rte_dev.h> +#include <rte_malloc.h> +#include <rte_mempool.h> +#include <rte_errno.h> +#include <rte_pci.h> +#include <rte_bus_pci.h> +#include <rte_byteorder.h> +#ifdef RTE_BBDEV_OFFLOAD_COST +#include <rte_cycles.h> +#endif + +#include <rte_bbdev.h> +#include <rte_bbdev_pmd.h> + +#include "fpga_lte_fec.h" + +/* Turbo SW PMD logging ID */ +static int fpga_lte_fec_logtype; + +/* Helper macro for logging */ +#define rte_bbdev_log(level, fmt, ...) \ + rte_log(RTE_LOG_ ## level, fpga_lte_fec_logtype, fmt "\n", \ + ##__VA_ARGS__) + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +#define rte_bbdev_log_debug(fmt, ...) \ + rte_bbdev_log(DEBUG, "fpga_lte_fec: " fmt, \ + ##__VA_ARGS__) +#else +#define rte_bbdev_log_debug(fmt, ...) +#endif + +/* FPGA LTE FEC driver names */ +#define FPGA_LTE_FEC_PF_DRIVER_NAME intel_fpga_lte_fec_pf +#define FPGA_LTE_FEC_VF_DRIVER_NAME intel_fpga_lte_fec_vf + +/* FPGA LTE FEC PCI vendor & device IDs */ +#define FPGA_LTE_FEC_VENDOR_ID (0x1172) +#define FPGA_LTE_FEC_PF_DEVICE_ID (0x5052) +#define FPGA_LTE_FEC_VF_DEVICE_ID (0x5050) + +/* Align DMA descriptors to 256 bytes - cache-aligned */ +#define FPGA_RING_DESC_ENTRY_LENGTH (8) +/* Ring size is in 256 bits (32 bytes) units */ +#define FPGA_RING_DESC_LEN_UNIT_BYTES (32) +/* Maximum size of queue */ +#define FPGA_RING_MAX_SIZE (1024) +#define FPGA_FLR_TIMEOUT_UNIT (16.384) + +#define FPGA_NUM_UL_QUEUES (32) +#define FPGA_NUM_DL_QUEUES (32) +#define FPGA_TOTAL_NUM_QUEUES (FPGA_NUM_UL_QUEUES + FPGA_NUM_DL_QUEUES) +#define FPGA_NUM_INTR_VEC (FPGA_TOTAL_NUM_QUEUES - RTE_INTR_VEC_RXTX_OFFSET) + +#define FPGA_INVALID_HW_QUEUE_ID (0xFFFFFFFF) + +#define FPGA_QUEUE_FLUSH_TIMEOUT_US (1000) +#define FPGA_TIMEOUT_CHECK_INTERVAL (5) + +/* FPGA LTE FEC Register mapping on BAR0 */ +enum { + FPGA_LTE_FEC_VERSION_ID = 0x00000000, /* len: 4B */ + FPGA_LTE_FEC_CONFIGURATION = 0x00000004, /* len: 2B */ + FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE = 0x00000008, /* len: 1B */ + FPGA_LTE_FEC_LOAD_BALANCE_FACTOR = 0x0000000a, /* len: 2B */ + FPGA_LTE_FEC_RING_DESC_LEN = 0x0000000c, /* len: 2B */ + FPGA_LTE_FEC_FLR_TIME_OUT = 0x0000000e, /* len: 2B */ + FPGA_LTE_FEC_VFQ_FLUSH_STATUS_LW = 0x00000018, /* len: 4B */ + FPGA_LTE_FEC_VFQ_FLUSH_STATUS_HI = 0x0000001c, /* len: 4B */ + FPGA_LTE_FEC_VF0_DEBUG = 0x00000020, /* len: 4B */ + FPGA_LTE_FEC_VF1_DEBUG = 0x00000024, /* len: 4B */ + FPGA_LTE_FEC_VF2_DEBUG = 0x00000028, /* len: 4B */ + FPGA_LTE_FEC_VF3_DEBUG = 0x0000002c, /* len: 4B */ + FPGA_LTE_FEC_VF4_DEBUG = 0x00000030, /* len: 4B */ + FPGA_LTE_FEC_VF5_DEBUG = 0x00000034, /* len: 4B */ + FPGA_LTE_FEC_VF6_DEBUG = 0x00000038, /* len: 4B */ + FPGA_LTE_FEC_VF7_DEBUG = 0x0000003c, /* len: 4B */ + FPGA_LTE_FEC_QUEUE_MAP = 0x00000040, /* len: 256B */ + FPGA_LTE_FEC_RING_CTRL_REGS = 0x00000200 /* len: 2048B */ +}; + +/* FPGA LTE FEC Ring Control Registers */ +enum { + FPGA_LTE_FEC_RING_HEAD_ADDR = 0x00000008, + FPGA_LTE_FEC_RING_SIZE = 0x00000010, + FPGA_LTE_FEC_RING_MISC = 0x00000014, + FPGA_LTE_FEC_RING_ENABLE = 0x00000015, + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN = 0x00000016, + FPGA_LTE_FEC_RING_SHADOW_TAIL = 0x00000018, + FPGA_LTE_FEC_RING_HEAD_POINT = 0x0000001C +}; + +/* FPGA LTE FEC DESCRIPTOR ERROR */ +enum { + DESC_ERR_NO_ERR = 0x0, + DESC_ERR_K_OUT_OF_RANGE = 0x1, + DESC_ERR_K_NOT_NORMAL = 0x2, + DESC_ERR_KPAI_NOT_NORMAL = 0x3, + DESC_ERR_DESC_OFFSET_ERR = 0x4, + DESC_ERR_DESC_READ_FAIL = 0x8, + DESC_ERR_DESC_READ_TIMEOUT = 0x9, + DESC_ERR_DESC_READ_TLP_POISONED = 0xA, + DESC_ERR_CB_READ_FAIL = 0xC, + DESC_ERR_CB_READ_TIMEOUT = 0xD, + DESC_ERR_CB_READ_TLP_POISONED = 0xE +}; + +/* FPGA LTE FEC DMA Encoding Request Descriptor */ +struct __attribute__((__packed__)) fpga_dma_enc_desc { + uint32_t done:1, + rsrvd0:11, + error:4, + rsrvd1:16; + uint32_t ncb:16, + rsrvd2:14, + rv:2; + uint32_t bypass_rm:1, + irq_en:1, + crc_en:1, + rsrvd3:13, + offset:10, + rsrvd4:6; + uint16_t e; + uint16_t k; + uint32_t out_addr_lw; + uint32_t out_addr_hi; + uint32_t in_addr_lw; + uint32_t in_addr_hi; + + union { + struct { + /* Virtual addresses used to retrieve SW context info */ + void *op_addr; + /* Stores information about total number of Code Blocks + * in currently processed Transport Block + */ + uint64_t cbs_in_op; + }; + + uint8_t sw_ctxt[FPGA_RING_DESC_LEN_UNIT_BYTES * + (FPGA_RING_DESC_ENTRY_LENGTH - 1)]; + }; +}; + +/* FPGA LTE FEC DMA Decoding Request Descriptor */ +struct __attribute__((__packed__)) fpga_dma_dec_desc { + uint32_t done:1, + iter:5, + rsrvd0:2, + crc_pass:1, + rsrvd1:3, + error:4, + crc_type:1, + rsrvd2:7, + max_iter:5, + rsrvd3:3; + uint32_t rsrvd4; + uint32_t bypass_rm:1, + irq_en:1, + drop_crc:1, + rsrvd5:13, + offset:10, + rsrvd6:6; + uint16_t k; + uint16_t in_len; + uint32_t out_addr_lw; + uint32_t out_addr_hi; + uint32_t in_addr_lw; + uint32_t in_addr_hi; + + union { + struct { + /* Virtual addresses used to retrieve SW context info */ + void *op_addr; + /* Stores information about total number of Code Blocks + * in currently processed Transport Block + */ + uint8_t cbs_in_op; + }; + + uint32_t sw_ctxt[8 * (FPGA_RING_DESC_ENTRY_LENGTH - 1)]; + }; +}; + +/* FPGA LTE DMA Descriptor */ +union fpga_dma_desc { + struct fpga_dma_enc_desc enc_req; + struct fpga_dma_dec_desc dec_req; +}; + +/* FPGA LTE FEC Ring Control Register */ +struct __attribute__((__packed__)) fpga_ring_ctrl_reg { + uint64_t ring_base_addr; + uint64_t ring_head_addr; + uint16_t ring_size:11; + uint16_t rsrvd0; + union { /* Miscellaneous register */ + uint8_t misc; + uint8_t max_ul_dec:5, + max_ul_dec_en:1, + rsrvd1:2; + }; + uint8_t enable; + uint8_t flush_queue_en; + uint8_t rsrvd2; + uint16_t shadow_tail; + uint16_t rsrvd3; + uint16_t head_point; + uint16_t rsrvd4; + +}; + +/* Private data structure for each FPGA FEC device */ +struct fpga_lte_fec_device { + /** Base address of MMIO registers (BAR0) */ + void *mmio_base; + /** Base address of memory for sw rings */ + void *sw_rings; + /** Physical address of sw_rings */ + rte_iova_t sw_rings_phys; + /** Number of bytes available for each queue in device. */ + uint32_t sw_ring_size; + /** Max number of entries available for each queue in device */ + uint32_t sw_ring_max_depth; + /** Base address of response tail pointer buffer */ + uint32_t *tail_ptrs; + /** Physical address of tail pointers */ + rte_iova_t tail_ptr_phys; + /** Queues flush completion flag */ + uint64_t *flush_queue_status; + /* Bitmap capturing which Queues are bound to the PF/VF */ + uint64_t q_bound_bit_map; + /* Bitmap capturing which Queues have already been assigned */ + uint64_t q_assigned_bit_map; + /** True if this is a PF FPGA FEC device */ + bool pf_device; +}; + +/* Structure associated with each queue. */ +struct __rte_cache_aligned fpga_queue { + struct fpga_ring_ctrl_reg ring_ctrl_reg; /* Ring Control Register */ + union fpga_dma_desc *ring_addr; /* Virtual address of software ring */ + uint64_t *ring_head_addr; /* Virtual address of completion_head */ + uint64_t shadow_completion_head; /* Shadow completion head value */ + uint16_t head_free_desc; /* Ring head */ + uint16_t tail; /* Ring tail */ + /* Mask used to wrap enqueued descriptors on the sw ring */ + uint32_t sw_ring_wrap_mask; + uint32_t irq_enable; /* Enable ops dequeue interrupts if set to 1 */ + uint8_t q_idx; /* Queue index */ + struct fpga_lte_fec_device *d; + /* MMIO register of shadow_tail used to enqueue descriptors */ + void *shadow_tail_addr; +}; + +/* Write to 16 bit MMIO register address */ +static inline void +mmio_write_16(void *addr, uint16_t value) +{ + *((volatile uint16_t *)(addr)) = rte_cpu_to_le_16(value); +} + +/* Write to 32 bit MMIO register address */ +static inline void +mmio_write_32(void *addr, uint32_t value) +{ + *((volatile uint32_t *)(addr)) = rte_cpu_to_le_32(value); +} + +/* Write to 64 bit MMIO register address */ +static inline void +mmio_write_64(void *addr, uint64_t value) +{ + *((volatile uint64_t *)(addr)) = rte_cpu_to_le_64(value); +} + +/* Write a 8 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_8(void *mmio_base, uint32_t offset, uint8_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + *((volatile uint8_t *)(reg_addr)) = payload; +} + +/* Write a 16 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_16(void *mmio_base, uint32_t offset, uint16_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_16(reg_addr, payload); +} + +/* Write a 32 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_32(void *mmio_base, uint32_t offset, uint32_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_32(reg_addr, payload); +} + +/* Write a 64 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_64(void *mmio_base, uint32_t offset, uint64_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_64(reg_addr, payload); +} + +/* Write a ring control register of a FPGA LTE FEC device */ +static inline void +fpga_ring_reg_write(void *mmio_base, uint32_t offset, + struct fpga_ring_ctrl_reg payload) +{ + fpga_reg_write_64(mmio_base, offset, payload.ring_base_addr); + fpga_reg_write_64(mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_ADDR, + payload.ring_head_addr); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_SIZE, + payload.ring_size); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_POINT, + payload.head_point); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN, + payload.flush_queue_en); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_SHADOW_TAIL, + payload.shadow_tail); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_MISC, + payload.misc); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + payload.enable); +} + +/* Read a register of FPGA LTE FEC device */ +static uint32_t +fpga_reg_read_32(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint32_t ret = *((volatile uint32_t *)(reg_addr)); + return rte_le_to_cpu_32(ret); +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Read a register of FPGA LTE FEC device */ +static uint8_t +fpga_reg_read_8(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + return *((volatile uint8_t *)(reg_addr)); +} + +/* Read a register of FPGA LTE FEC device */ +static uint16_t +fpga_reg_read_16(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint16_t ret = *((volatile uint16_t *)(reg_addr)); + return rte_le_to_cpu_16(ret); +} + +/* Read a register of FPGA LTE FEC device */ +static uint64_t +fpga_reg_read_64(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint64_t ret = *((volatile uint64_t *)(reg_addr)); + return rte_le_to_cpu_64(ret); +} + +/* Read Ring Control Register of FPGA LTE FEC device */ +static inline void +print_ring_reg_debug_info(void *mmio_base, uint32_t offset) +{ + rte_bbdev_log_debug( + "FPGA MMIO base address @ %p | Ring Control Register @ offset = 0x%08" + PRIx32, mmio_base, offset); + rte_bbdev_log_debug( + "RING_BASE_ADDR = 0x%016"PRIx64, + fpga_reg_read_64(mmio_base, offset)); + rte_bbdev_log_debug( + "RING_HEAD_ADDR = 0x%016"PRIx64, + fpga_reg_read_64(mmio_base, offset + + FPGA_LTE_FEC_RING_HEAD_ADDR)); + rte_bbdev_log_debug( + "RING_SIZE = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_SIZE)); + rte_bbdev_log_debug( + "RING_MISC = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_MISC)); + rte_bbdev_log_debug( + "RING_ENABLE = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_ENABLE)); + rte_bbdev_log_debug( + "RING_FLUSH_QUEUE_EN = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN)); + rte_bbdev_log_debug( + "RING_SHADOW_TAIL = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_SHADOW_TAIL)); + rte_bbdev_log_debug( + "RING_HEAD_POINT = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_HEAD_POINT)); +} + +/* Read Static Register of FPGA LTE FEC device */ +static inline void +print_static_reg_debug_info(void *mmio_base) +{ + uint16_t config = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_CONFIGURATION); + uint8_t qmap_done = fpga_reg_read_8(mmio_base, + FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE); + uint16_t lb_factor = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_LOAD_BALANCE_FACTOR); + uint16_t ring_desc_len = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_RING_DESC_LEN); + uint16_t flr_time_out = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_FLR_TIME_OUT); + + rte_bbdev_log_debug("UL.DL Weights = %u.%u", + ((uint8_t)config), ((uint8_t)(config >> 8))); + rte_bbdev_log_debug("UL.DL Load Balance = %u.%u", + ((uint8_t)lb_factor), ((uint8_t)(lb_factor >> 8))); + rte_bbdev_log_debug("Queue-PF/VF Mapping Table = %s", + (qmap_done > 0) ? "READY" : "NOT-READY"); + rte_bbdev_log_debug("Ring Descriptor Size = %u bytes", + ring_desc_len*FPGA_RING_DESC_LEN_UNIT_BYTES); + rte_bbdev_log_debug("FLR Timeout = %f usec", + (float)flr_time_out*FPGA_FLR_TIMEOUT_UNIT); +} + +/* Print decode DMA Descriptor of FPGA LTE FEC device */ +static void +print_dma_dec_desc_debug_info(union fpga_dma_desc *desc) +{ + rte_bbdev_log_debug("DMA response desc %p\n" + "\t-- done(%"PRIu32") | iter(%"PRIu32") | crc_pass(%"PRIu32")" + " | error (%"PRIu32") | crc_type(%"PRIu32")\n" + "\t-- max_iter(%"PRIu32") | bypass_rm(%"PRIu32") | " + "irq_en (%"PRIu32") | drop_crc(%"PRIu32") | offset(%"PRIu32")\n" + "\t-- k(%"PRIu32") | in_len (%"PRIu16") | op_add(%p)\n" + "\t-- cbs_in_op(%"PRIu32") | in_add (0x%08"PRIx32"%08"PRIx32") | " + "out_add (0x%08"PRIx32"%08"PRIx32")", + desc, + (uint32_t)desc->dec_req.done, + (uint32_t)desc->dec_req.iter, + (uint32_t)desc->dec_req.crc_pass, + (uint32_t)desc->dec_req.error, + (uint32_t)desc->dec_req.crc_type, + (uint32_t)desc->dec_req.max_iter, + (uint32_t)desc->dec_req.bypass_rm, + (uint32_t)desc->dec_req.irq_en, + (uint32_t)desc->dec_req.drop_crc, + (uint32_t)desc->dec_req.offset, + (uint32_t)desc->dec_req.k, + (uint16_t)desc->dec_req.in_len, + desc->dec_req.op_addr, + (uint32_t)desc->dec_req.cbs_in_op, + (uint32_t)desc->dec_req.in_addr_hi, + (uint32_t)desc->dec_req.in_addr_lw, + (uint32_t)desc->dec_req.out_addr_hi, + (uint32_t)desc->dec_req.out_addr_lw); +} +#endif + +static int +fpga_setup_queues(struct rte_bbdev *dev, uint16_t num_queues, int socket_id) +{ + /* Number of queues bound to a PF/VF */ + uint32_t hw_q_num = 0; + uint32_t ring_size, payload, address, q_id, offset; + rte_iova_t phys_addr; + struct fpga_ring_ctrl_reg ring_reg; + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + + address = FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE; + if (!(fpga_reg_read_32(fpga_dev->mmio_base, address) & 0x1)) { + rte_bbdev_log(ERR, + "Queue-PF/VF mapping is not set! Was PF configured for device (%s) ?", + dev->data->name); + return -EPERM; + } + + /* Clear queue registers structure */ + memset(&ring_reg, 0, sizeof(struct fpga_ring_ctrl_reg)); + + /* Scan queue map. + * If a queue is valid and mapped to a calling PF/VF the read value is + * replaced with a queue ID and if it's not then + * FPGA_INVALID_HW_QUEUE_ID is returned. + */ + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + uint32_t hw_q_id = fpga_reg_read_32(fpga_dev->mmio_base, + FPGA_LTE_FEC_QUEUE_MAP + (q_id << 2)); + + rte_bbdev_log_debug("%s: queue ID: %u, registry queue ID: %u", + dev->device->name, q_id, hw_q_id); + + if (hw_q_id != FPGA_INVALID_HW_QUEUE_ID) { + fpga_dev->q_bound_bit_map |= (1ULL << q_id); + /* Clear queue register of found queue */ + offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q_id); + fpga_ring_reg_write(fpga_dev->mmio_base, + offset, ring_reg); + ++hw_q_num; + } + } + if (hw_q_num == 0) { + rte_bbdev_log(ERR, + "No HW queues assigned to this device. Probably this is a VF configured for PF mode. Check device configuration!"); + return -ENODEV; + } + + if (num_queues > hw_q_num) { + rte_bbdev_log(ERR, + "Not enough queues for device %s! Requested: %u, available: %u", + dev->device->name, num_queues, hw_q_num); + return -EINVAL; + } + + ring_size = FPGA_RING_MAX_SIZE * sizeof(struct fpga_dma_dec_desc); + + /* Enforce 32 byte alignment */ + RTE_BUILD_BUG_ON((RTE_CACHE_LINE_SIZE % 32) != 0); + + /* Allocate memory for SW descriptor rings */ + fpga_dev->sw_rings = rte_zmalloc_socket(dev->device->driver->name, + num_queues * ring_size, RTE_CACHE_LINE_SIZE, + socket_id); + if (fpga_dev->sw_rings == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u sw_rings", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + + fpga_dev->sw_rings_phys = rte_malloc_virt2iova(fpga_dev->sw_rings); + fpga_dev->sw_ring_size = ring_size; + fpga_dev->sw_ring_max_depth = FPGA_RING_MAX_SIZE; + + /* Allocate memory for ring flush status */ + fpga_dev->flush_queue_status = rte_zmalloc_socket(NULL, + sizeof(uint64_t), RTE_CACHE_LINE_SIZE, socket_id); + if (fpga_dev->flush_queue_status == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u flush_queue_status", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + + /* Set the flush status address registers */ + phys_addr = rte_malloc_virt2iova(fpga_dev->flush_queue_status); + + address = FPGA_LTE_FEC_VFQ_FLUSH_STATUS_LW; + payload = (uint32_t)(phys_addr); + fpga_reg_write_32(fpga_dev->mmio_base, address, payload); + + address = FPGA_LTE_FEC_VFQ_FLUSH_STATUS_HI; + payload = (uint32_t)(phys_addr >> 32); + fpga_reg_write_32(fpga_dev->mmio_base, address, payload); + + return 0; +} + +static int +fpga_dev_close(struct rte_bbdev *dev) +{ + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + + rte_free(fpga_dev->sw_rings); + rte_free(fpga_dev->flush_queue_status); + + return 0; +} + +static void +fpga_dev_info_get(struct rte_bbdev *dev, + struct rte_bbdev_driver_info *dev_info) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + uint32_t q_id = 0; + + /* TODO RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN and numbers of buffers are set + * to temporary values as they are required by test application while + * validation phase. + */ + static const struct rte_bbdev_op_cap bbdev_capabilities[] = { + { + .type = RTE_BBDEV_OP_TURBO_DEC, + .cap.turbo_dec = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_TYPE_24B | + RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE | + RTE_BBDEV_TURBO_DEC_INTERRUPTS | + RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN | + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP, + .max_llr_modulus = INT8_MAX, + .num_buffers_src = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_hard_out = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_soft_out = 0 + } + }, + { + .type = RTE_BBDEV_OP_TURBO_ENC, + .cap.turbo_enc = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_24B_ATTACH | + RTE_BBDEV_TURBO_RATE_MATCH | + RTE_BBDEV_TURBO_ENC_INTERRUPTS, + .num_buffers_src = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_dst = + RTE_BBDEV_MAX_CODE_BLOCKS + } + }, + RTE_BBDEV_END_OF_CAPABILITIES_LIST() + }; + + static struct rte_bbdev_queue_conf default_queue_conf; + default_queue_conf.socket = dev->data->socket_id; + default_queue_conf.queue_size = FPGA_RING_MAX_SIZE; + + + dev_info->driver_name = dev->device->driver->name; + dev_info->queue_size_lim = FPGA_RING_MAX_SIZE; + dev_info->hardware_accelerated = true; + dev_info->min_alignment = 64; + dev_info->default_queue_conf = default_queue_conf; + dev_info->capabilities = bbdev_capabilities; + dev_info->cpu_flag_reqs = NULL; + + /* Calculates number of queues assigned to device */ + dev_info->max_num_queues = 0; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + uint32_t hw_q_id = fpga_reg_read_32(d->mmio_base, + FPGA_LTE_FEC_QUEUE_MAP + (q_id << 2)); + if (hw_q_id != FPGA_INVALID_HW_QUEUE_ID) + dev_info->max_num_queues++; + } +} + +/** + * Find index of queue bound to current PF/VF which is unassigned. Return -1 + * when there is no available queue + */ +static int +fpga_find_free_queue_idx(struct rte_bbdev *dev, + const struct rte_bbdev_queue_conf *conf) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + uint64_t q_idx; + uint8_t i = 0; + uint8_t range = FPGA_TOTAL_NUM_QUEUES >> 1; + + if (conf->op_type == RTE_BBDEV_OP_TURBO_ENC) { + i = FPGA_NUM_DL_QUEUES; + range = FPGA_TOTAL_NUM_QUEUES; + } + + for (; i < range; ++i) { + q_idx = 1ULL << i; + /* Check if index of queue is bound to current PF/VF */ + if (d->q_bound_bit_map & q_idx) + /* Check if found queue was not already assigned */ + if (!(d->q_assigned_bit_map & q_idx)) { + d->q_assigned_bit_map |= q_idx; + return i; + } + } + + rte_bbdev_log(INFO, "Failed to find free queue on %s", dev->data->name); + + return -1; +} + +static int +fpga_queue_setup(struct rte_bbdev *dev, uint16_t queue_id, + const struct rte_bbdev_queue_conf *conf) +{ + uint32_t address, ring_offset; + struct fpga_lte_fec_device *d = dev->data->dev_private; + struct fpga_queue *q; + int8_t q_idx; + + /* Check if there is a free queue to assign */ + q_idx = fpga_find_free_queue_idx(dev, conf); + if (q_idx == -1) + return -1; + + /* Allocate the queue data structure. */ + q = rte_zmalloc_socket(dev->device->driver->name, sizeof(*q), + RTE_CACHE_LINE_SIZE, conf->socket); + if (q == NULL) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_bbdev_log(ERR, "Failed to allocate queue memory"); + return -ENOMEM; + } + + q->d = d; + q->q_idx = q_idx; + + /* Set ring_base_addr */ + q->ring_addr = RTE_PTR_ADD(d->sw_rings, (d->sw_ring_size * queue_id)); + q->ring_ctrl_reg.ring_base_addr = d->sw_rings_phys + + (d->sw_ring_size * queue_id); + + /* Allocate memory for Completion Head variable*/ + q->ring_head_addr = rte_zmalloc_socket(dev->device->driver->name, + sizeof(uint64_t), RTE_CACHE_LINE_SIZE, conf->socket); + if (q->ring_head_addr == NULL) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_free(q); + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u completion_head", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + /* Set ring_head_addr */ + q->ring_ctrl_reg.ring_head_addr = + rte_malloc_virt2iova(q->ring_head_addr); + + /* Clear shadow_completion_head */ + q->shadow_completion_head = 0; + + /* Set ring_size */ + if (conf->queue_size > FPGA_RING_MAX_SIZE) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_free(q->ring_head_addr); + rte_free(q); + rte_bbdev_log(ERR, + "Size of queue is too big %d (MAX: %d ) for %s:%u", + conf->queue_size, FPGA_RING_MAX_SIZE, + dev->device->driver->name, dev->data->dev_id); + return -EINVAL; + } + q->ring_ctrl_reg.ring_size = conf->queue_size; + + /* Set Miscellaneous FPGA register*/ + /* Max iteration number for TTI mitigation - todo */ + q->ring_ctrl_reg.max_ul_dec = 0; + /* Enable max iteration number for TTI - todo */ + q->ring_ctrl_reg.max_ul_dec_en = 0; + + /* Enable the ring */ + q->ring_ctrl_reg.enable = 1; + + /* Set FPGA head_point and tail registers */ + q->ring_ctrl_reg.head_point = q->tail = 0; + + /* Set FPGA shadow_tail register */ + q->ring_ctrl_reg.shadow_tail = q->tail; + + /* Calculates the ring offset for found queue */ + ring_offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q_idx); + + /* Set FPGA Ring Control Registers */ + fpga_ring_reg_write(d->mmio_base, ring_offset, q->ring_ctrl_reg); + + /* Store MMIO register of shadow_tail */ + address = ring_offset + FPGA_LTE_FEC_RING_SHADOW_TAIL; + q->shadow_tail_addr = RTE_PTR_ADD(d->mmio_base, address); + + q->head_free_desc = q->tail; + + /* Set wrap mask */ + q->sw_ring_wrap_mask = conf->queue_size - 1; + + rte_bbdev_log_debug("Setup dev%u q%u: queue_idx=%u", + dev->data->dev_id, queue_id, q->q_idx); + + dev->data->queues[queue_id].queue_private = q; + + rte_bbdev_log_debug("BBDEV queue[%d] set up for FPGA queue[%d]", + queue_id, q_idx); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Read FPGA Ring Control Registers after configuration*/ + print_ring_reg_debug_info(d->mmio_base, ring_offset); +#endif + return 0; +} + +static int +fpga_queue_release(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + struct fpga_ring_ctrl_reg ring_reg; + uint32_t offset; + + rte_bbdev_log_debug("FPGA Queue[%d] released", queue_id); + + if (q != NULL) { + memset(&ring_reg, 0, sizeof(struct fpga_ring_ctrl_reg)); + offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + /* Disable queue */ + fpga_reg_write_8(d->mmio_base, + offset + FPGA_LTE_FEC_RING_ENABLE, 0x00); + /* Clear queue registers */ + fpga_ring_reg_write(d->mmio_base, offset, ring_reg); + + /* Mark the Queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q->q_idx)); + rte_free(q->ring_head_addr); + rte_free(q); + dev->data->queues[queue_id].queue_private = NULL; + } + + return 0; +} + +/* Function starts a device queue. */ +static int +fpga_queue_start(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (d == NULL) { + rte_bbdev_log(ERR, "Invalid device pointer"); + return -1; + } +#endif + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + uint32_t offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + uint8_t enable = 0x01; + uint16_t zero = 0x0000; + + /* Clear queue head and tail variables */ + q->tail = q->head_free_desc = 0; + + /* Clear FPGA head_point and tail registers */ + fpga_reg_write_16(d->mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_POINT, + zero); + fpga_reg_write_16(d->mmio_base, offset + FPGA_LTE_FEC_RING_SHADOW_TAIL, + zero); + + /* Enable queue */ + fpga_reg_write_8(d->mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + enable); + + rte_bbdev_log_debug("FPGA Queue[%d] started", queue_id); + return 0; +} + +/* Function stops a device queue. */ +static int +fpga_queue_stop(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (d == NULL) { + rte_bbdev_log(ERR, "Invalid device pointer"); + return -1; + } +#endif + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + uint32_t offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + uint8_t payload = 0x01; + uint8_t counter = 0; + uint8_t timeout = FPGA_QUEUE_FLUSH_TIMEOUT_US / + FPGA_TIMEOUT_CHECK_INTERVAL; + + /* Set flush_queue_en bit to trigger queue flushing */ + fpga_reg_write_8(d->mmio_base, + offset + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN, payload); + + /** Check if queue flush is completed. + * FPGA will update the completion flag after queue flushing is + * completed. If completion flag is not updated within 1ms it is + * considered as a failure. + */ + while (!(*((uint8_t *)d->flush_queue_status + q->q_idx) & payload)) { + if (counter > timeout) { + rte_bbdev_log(ERR, "FPGA Queue Flush failed for queue %d", + queue_id); + return -1; + } + usleep(FPGA_TIMEOUT_CHECK_INTERVAL); + counter++; + } + + /* Disable queue */ + payload = 0x00; + fpga_reg_write_8(d->mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + payload); + + rte_bbdev_log_debug("FPGA Queue[%d] stopped", queue_id); + return 0; +} + +static inline uint16_t +get_queue_id(struct rte_bbdev_data *data, uint8_t q_idx) +{ + uint16_t queue_id; + + for (queue_id = 0; queue_id < data->num_queues; ++queue_id) { + struct fpga_queue *q = data->queues[queue_id].queue_private; + if (q != NULL && q->q_idx == q_idx) + return queue_id; + } + + return -1; +} + +/* Interrupt handler triggered by FPGA dev for handling specific interrupt */ +static void +fpga_dev_interrupt_handler(void *cb_arg) +{ + struct rte_bbdev *dev = cb_arg; + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + struct fpga_queue *q; + uint64_t ring_head; + uint64_t q_idx; + uint16_t queue_id; + uint8_t i; + + /* Scan queue assigned to this device */ + for (i = 0; i < FPGA_TOTAL_NUM_QUEUES; ++i) { + q_idx = 1ULL << i; + if (fpga_dev->q_bound_bit_map & q_idx) { + queue_id = get_queue_id(dev->data, i); + if (queue_id == (uint16_t) -1) + continue; + + /* Check if completion head was changed */ + q = dev->data->queues[queue_id].queue_private; + ring_head = *q->ring_head_addr; + if (q->shadow_completion_head != ring_head && + q->irq_enable == 1) { + q->shadow_completion_head = ring_head; + rte_bbdev_pmd_callback_process( + dev, + RTE_BBDEV_EVENT_DEQUEUE, + &queue_id); + } + } + } +} + +static int +fpga_queue_intr_enable(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + + if (!rte_intr_cap_multiple(dev->intr_handle)) + return -ENOTSUP; + + q->irq_enable = 1; + + return 0; +} + +static int +fpga_queue_intr_disable(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + q->irq_enable = 0; + + return 0; +} + +static int +fpga_intr_enable(struct rte_bbdev *dev) +{ + int ret; + uint8_t i; + + if (!rte_intr_cap_multiple(dev->intr_handle)) { + rte_bbdev_log(ERR, "Multiple intr vector is not supported by FPGA (%s)", + dev->data->name); + return -ENOTSUP; + } + + /* Create event file descriptors for each of 64 queue. Event fds will be + * mapped to FPGA IRQs in rte_intr_enable(). This is a 1:1 mapping where + * the IRQ number is a direct translation to the queue number. + * + * 63 (FPGA_NUM_INTR_VEC) event fds are created as rte_intr_enable() + * mapped the first IRQ to already created interrupt event file + * descriptor (intr_handle->fd). + */ + if (rte_intr_efd_enable(dev->intr_handle, FPGA_NUM_INTR_VEC)) { + rte_bbdev_log(ERR, "Failed to create fds for %u queues", + dev->data->num_queues); + return -1; + } + + /* TODO Each event file descriptor is overwritten by interrupt event + * file descriptor. That descriptor is added to epoll observed list. + * It ensures that callback function assigned to that descriptor will + * invoked when any FPGA queue issues interrupt. + */ + for (i = 0; i < FPGA_NUM_INTR_VEC; ++i) + dev->intr_handle->efds[i] = dev->intr_handle->fd; + + if (!dev->intr_handle->intr_vec) { + dev->intr_handle->intr_vec = rte_zmalloc("intr_vec", + dev->data->num_queues * sizeof(int), 0); + if (!dev->intr_handle->intr_vec) { + rte_bbdev_log(ERR, "Failed to allocate %u vectors", + dev->data->num_queues); + return -ENOMEM; + } + } + + ret = rte_intr_enable(dev->intr_handle); + if (ret < 0) { + rte_bbdev_log(ERR, + "Couldn't enable interrupts for device: %s", + dev->data->name); + return ret; + } + + ret = rte_intr_callback_register(dev->intr_handle, + fpga_dev_interrupt_handler, dev); + if (ret < 0) { + rte_bbdev_log(ERR, + "Couldn't register interrupt callback for device: %s", + dev->data->name); + return ret; + } + + return 0; +} + +static const struct rte_bbdev_ops fpga_ops = { + .setup_queues = fpga_setup_queues, + .intr_enable = fpga_intr_enable, + .close = fpga_dev_close, + .info_get = fpga_dev_info_get, + .queue_setup = fpga_queue_setup, + .queue_stop = fpga_queue_stop, + .queue_start = fpga_queue_start, + .queue_release = fpga_queue_release, + .queue_intr_enable = fpga_queue_intr_enable, + .queue_intr_disable = fpga_queue_intr_disable +}; + +static inline void +fpga_dma_enqueue(struct fpga_queue *q, uint16_t num_desc, + struct rte_bbdev_stats *queue_stats) +{ +#ifdef RTE_BBDEV_OFFLOAD_COST + uint64_t start_time = 0; + queue_stats->acc_offload_cycles = 0; +#else + RTE_SET_USED(queue_stats); +#endif + + /* Update tail and shadow_tail register */ + q->tail = (q->tail + num_desc) & q->sw_ring_wrap_mask; + + rte_wmb(); + +#ifdef RTE_BBDEV_OFFLOAD_COST + /* Start time measurement for enqueue function offload. */ + start_time = rte_rdtsc_precise(); +#endif + mmio_write_16(q->shadow_tail_addr, q->tail); + +#ifdef RTE_BBDEV_OFFLOAD_COST + rte_wmb(); + queue_stats->acc_offload_cycles += rte_rdtsc_precise() - start_time; +#endif +} + +/* Calculates number of CBs in processed encoder TB based on 'r' and input + * length. + */ +static inline uint8_t +get_num_cbs_in_op_enc(struct rte_bbdev_op_turbo_enc *turbo_enc) +{ + uint8_t c, c_neg, r, crc24_bits = 0; + uint16_t k, k_neg, k_pos; + uint8_t cbs_in_op = 0; + int32_t length; + + length = turbo_enc->input.length; + r = turbo_enc->tb_params.r; + c = turbo_enc->tb_params.c; + c_neg = turbo_enc->tb_params.c_neg; + k_neg = turbo_enc->tb_params.k_neg; + k_pos = turbo_enc->tb_params.k_pos; + crc24_bits = 24; + while (length > 0 && r < c) { + k = (r < c_neg) ? k_neg : k_pos; + length -= (k - crc24_bits) >> 3; + r++; + cbs_in_op++; + } + + return cbs_in_op; +} + +/* Calculates number of CBs in processed decoder TB based on 'r' and input + * length. + */ +static inline uint16_t +get_num_cbs_in_op_dec(struct rte_bbdev_op_turbo_dec *turbo_dec) +{ + uint8_t c, c_neg, r = 0; + uint16_t kw, k, k_neg, k_pos, cbs_in_op = 0; + int32_t length; + + length = turbo_dec->input.length; + r = turbo_dec->tb_params.r; + c = turbo_dec->tb_params.c; + c_neg = turbo_dec->tb_params.c_neg; + k_neg = turbo_dec->tb_params.k_neg; + k_pos = turbo_dec->tb_params.k_pos; + while (length > 0 && r < c) { + k = (r < c_neg) ? k_neg : k_pos; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + length -= kw; + r++; + cbs_in_op++; + } + + return cbs_in_op; +} + +/* Read flag value 0/1/ from bitmap */ +static inline bool +check_bit(uint32_t bitmap, uint32_t bitmask) +{ + return bitmap & bitmask; +} + +/* Print an error if a descriptor error has occurred. + * Return 0 on success, 1 on failure + */ +static inline int +check_desc_error(uint32_t error_code) { + switch (error_code) { + case DESC_ERR_NO_ERR: + return 0; + case DESC_ERR_K_OUT_OF_RANGE: + rte_bbdev_log(ERR, "Block_size_k is out of range (k<40 or k>6144)"); + break; + case DESC_ERR_K_NOT_NORMAL: + rte_bbdev_log(ERR, "Block_size_k is not a normal value within normal range"); + break; + case DESC_ERR_KPAI_NOT_NORMAL: + rte_bbdev_log(ERR, "Three_kpai is not a normal value for UL only"); + break; + case DESC_ERR_DESC_OFFSET_ERR: + rte_bbdev_log(ERR, "Queue offset does not meet the expectation in the FPGA"); + break; + case (DESC_ERR_K_OUT_OF_RANGE | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Block_size_k is out of range (k<40 or k>6144) and queue offset error"); + break; + case (DESC_ERR_K_NOT_NORMAL | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Block_size_k is not a normal value within normal range and queue offset error"); + break; + case (DESC_ERR_KPAI_NOT_NORMAL | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Three_kpai is not a normal value for UL only and queue offset error"); + break; + case DESC_ERR_DESC_READ_FAIL: + rte_bbdev_log(ERR, "Unsuccessful completion for descriptor read"); + break; + case DESC_ERR_DESC_READ_TIMEOUT: + rte_bbdev_log(ERR, "Descriptor read time-out"); + break; + case DESC_ERR_DESC_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Descriptor read TLP poisoned"); + break; + case DESC_ERR_CB_READ_FAIL: + rte_bbdev_log(ERR, "Unsuccessful completion for code block"); + break; + case DESC_ERR_CB_READ_TIMEOUT: + rte_bbdev_log(ERR, "Code block read time-out"); + break; + case DESC_ERR_CB_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Code block read TLP poisoned"); + break; + default: + rte_bbdev_log(ERR, "Descriptor error unknown error code %u", + error_code); + break; + } + return 1; +} + +/** + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param k + * K value (length of input in bits). + * @param e + * E value (length of output in bits). + * @param ncb + * Ncb value (size of the soft buffer). + * @param out_length + * Length of output buffer + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +fpga_dma_desc_te_fill(struct rte_bbdev_enc_op *op, + struct fpga_dma_enc_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t k, uint16_t e, uint16_t ncb, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) + +{ + /* reset */ + desc->done = 0; + desc->crc_en = check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_CRC_24B_ATTACH); + desc->bypass_rm = !check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_RATE_MATCH); + desc->k = k; + desc->e = e; + desc->ncb = ncb; + desc->rv = op->turbo_enc.rv_index; + desc->offset = desc_offset; + /* Set inbound data buffer address */ + desc->in_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset) >> 32); + desc->in_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset)); + + desc->out_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset) >> 32); + desc->out_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset)); + + /* Save software context needed for dequeue */ + desc->op_addr = op; + + /* Set total number of CBs in an op */ + desc->cbs_in_op = cbs_in_op; + + return 0; +} + +/** + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param in_length + * Length of an input. + * @param k + * K value (length of an output in bits). + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +fpga_dma_desc_td_fill(struct rte_bbdev_dec_op *op, + struct fpga_dma_dec_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t in_length, uint16_t k, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) +{ + /* reset */ + desc->done = 0; + /* Set inbound data buffer address */ + desc->in_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset) >> 32); + desc->in_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset)); + desc->in_len = in_length; + desc->k = k; + desc->crc_type = !check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_CRC_TYPE_24B); + if ((op->turbo_dec.code_block_mode == 0) + && !check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP)) + desc->drop_crc = 1; + desc->max_iter = op->turbo_dec.iter_max * 2; + desc->offset = desc_offset; + desc->out_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset) >> 32); + desc->out_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset)); + + /* Save software context needed for dequeue */ + desc->op_addr = op; + + /* Set total number of CBs in an op */ + desc->cbs_in_op = cbs_in_op; + + return 0; +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Validates turbo encoder parameters */ +static int +validate_enc_op(struct rte_bbdev_enc_op *op) +{ + struct rte_bbdev_op_turbo_enc *turbo_enc = &op->turbo_enc; + struct rte_bbdev_op_enc_turbo_cb_params *cb = NULL; + struct rte_bbdev_op_enc_turbo_tb_params *tb = NULL; + uint16_t kw, kw_neg, kw_pos; + + if (turbo_enc->input.length > + RTE_BBDEV_MAX_TB_SIZE >> 3) { + rte_bbdev_log(ERR, "TB size (%u) is too big, max: %d", + turbo_enc->input.length, RTE_BBDEV_MAX_TB_SIZE); + op->status = 1 << RTE_BBDEV_DATA_ERROR; + return -1; + } + + if (op->mempool == NULL) { + rte_bbdev_log(ERR, "Invalid mempool pointer"); + return -1; + } + if (turbo_enc->input.data == NULL) { + rte_bbdev_log(ERR, "Invalid input pointer"); + return -1; + } + if (turbo_enc->output.data == NULL) { + rte_bbdev_log(ERR, "Invalid output pointer"); + return -1; + } + if (turbo_enc->rv_index > 3) { + rte_bbdev_log(ERR, + "rv_index (%u) is out of range 0 <= value <= 3", + turbo_enc->rv_index); + return -1; + } + if (turbo_enc->code_block_mode != 0 && + turbo_enc->code_block_mode != 1) { + rte_bbdev_log(ERR, + "code_block_mode (%u) is out of range 0 <= value <= 1", + turbo_enc->code_block_mode); + return -1; + } + + if (turbo_enc->code_block_mode == 0) { + tb = &turbo_enc->tb_params; + if ((tb->k_neg < RTE_BBDEV_MIN_CB_SIZE + || tb->k_neg > RTE_BBDEV_MAX_CB_SIZE) + && tb->c_neg > 0) { + rte_bbdev_log(ERR, + "k_neg (%u) is out of range %u <= value <= %u", + tb->k_neg, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->k_pos < RTE_BBDEV_MIN_CB_SIZE + || tb->k_pos > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k_pos (%u) is out of range %u <= value <= %u", + tb->k_pos, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->c_neg > (RTE_BBDEV_MAX_CODE_BLOCKS - 1)) + rte_bbdev_log(ERR, + "c_neg (%u) is out of range 0 <= value <= %u", + tb->c_neg, + RTE_BBDEV_MAX_CODE_BLOCKS - 1); + if (tb->c < 1 || tb->c > RTE_BBDEV_MAX_CODE_BLOCKS) { + rte_bbdev_log(ERR, + "c (%u) is out of range 1 <= value <= %u", + tb->c, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + if (tb->cab > tb->c) { + rte_bbdev_log(ERR, + "cab (%u) is greater than c (%u)", + tb->cab, tb->c); + return -1; + } + if ((tb->ea < RTE_BBDEV_MIN_CB_SIZE || (tb->ea % 2)) + && tb->r < tb->cab) { + rte_bbdev_log(ERR, + "ea (%u) is less than %u or it is not even", + tb->ea, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + if ((tb->eb < RTE_BBDEV_MIN_CB_SIZE || (tb->eb % 2)) + && tb->c > tb->cab) { + rte_bbdev_log(ERR, + "eb (%u) is less than %u or it is not even", + tb->eb, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + + kw_neg = 3 * RTE_ALIGN_CEIL(tb->k_neg + 4, + RTE_BBDEV_TURBO_C_SUBBLOCK); + if (tb->ncb_neg < tb->k_neg || tb->ncb_neg > kw_neg) { + rte_bbdev_log(ERR, + "ncb_neg (%u) is out of range (%u) k_neg <= value <= (%u) kw_neg", + tb->ncb_neg, tb->k_neg, kw_neg); + return -1; + } + + kw_pos = 3 * RTE_ALIGN_CEIL(tb->k_pos + 4, + RTE_BBDEV_TURBO_C_SUBBLOCK); + if (tb->ncb_pos < tb->k_pos || tb->ncb_pos > kw_pos) { + rte_bbdev_log(ERR, + "ncb_pos (%u) is out of range (%u) k_pos <= value <= (%u) kw_pos", + tb->ncb_pos, tb->k_pos, kw_pos); + return -1; + } + if (tb->r > (tb->c - 1)) { + rte_bbdev_log(ERR, + "r (%u) is greater than c - 1 (%u)", + tb->r, tb->c - 1); + return -1; + } + } else { + cb = &turbo_enc->cb_params; + if (cb->k < RTE_BBDEV_MIN_CB_SIZE + || cb->k > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k (%u) is out of range %u <= value <= %u", + cb->k, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + + if (cb->e < RTE_BBDEV_MIN_CB_SIZE || (cb->e % 2)) { + rte_bbdev_log(ERR, + "e (%u) is less than %u or it is not even", + cb->e, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + + kw = RTE_ALIGN_CEIL(cb->k + 4, RTE_BBDEV_TURBO_C_SUBBLOCK) * 3; + if (cb->ncb < cb->k || cb->ncb > kw) { + rte_bbdev_log(ERR, + "ncb (%u) is out of range (%u) k <= value <= (%u) kw", + cb->ncb, cb->k, kw); + return -1; + } + } + + return 0; +} +#endif + +static inline char * +mbuf_append(struct rte_mbuf *m_head, struct rte_mbuf *m, uint16_t len) +{ + if (unlikely(len > rte_pktmbuf_tailroom(m))) + return NULL; + + char *tail = (char *)m->buf_addr + m->data_off + m->data_len; + m->data_len = (uint16_t)(m->data_len + len); + m_head->pkt_len = (m_head->pkt_len + len); + return tail; +} + +static inline int +enqueue_enc_one_op_cb(struct fpga_queue *q, struct rte_bbdev_enc_op *op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input; + struct rte_mbuf *output; + int ret; + uint16_t k, e, ncb, ring_offset; + uint32_t total_left, in_length, out_length, in_offset, out_offset; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_enc_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo encoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_enc.input.data; + output = op->turbo_enc.output.data; + in_offset = op->turbo_enc.input.offset; + out_offset = op->turbo_enc.output.offset; + total_left = op->turbo_enc.input.length; + k = op->turbo_enc.cb_params.k; + e = op->turbo_enc.cb_params.e; + ncb = op->turbo_enc.cb_params.ncb; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_CRC_24B_ATTACH)) + in_length = ((k - 24) >> 3); + else + in_length = k >> 3; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_RATE_MATCH)) + out_length = (e + 7) >> 3; + else + out_length = (k >> 3) * 3 + 2; + + mbuf_append(output, output, out_length); + + /* Offset into the ring */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + /* Setup DMA Descriptor */ + desc = q->ring_addr + ring_offset; + + ret = fpga_dma_desc_te_fill(op, &desc->enc_req, input, output, k, e, + ncb, in_offset, out_offset, ring_offset, 1); + if (unlikely(ret < 0)) + return ret; + + /* Update lengths */ + total_left -= in_length; + op->turbo_enc.output.length += out_length; + + if (total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + total_left, in_length); + return -1; + } + + return 1; +} + +static inline int +enqueue_enc_one_op_tb(struct fpga_queue *q, struct rte_bbdev_enc_op *op, + uint16_t desc_offset, uint8_t cbs_in_op) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input, *output_head, *output; + int ret; + uint8_t r, c, crc24_bits = 0; + uint16_t k, e, ncb, ring_offset; + uint32_t mbuf_total_left, in_length, out_length, in_offset, out_offset; + uint32_t seg_total_left; + uint16_t current_enqueued_cbs = 0; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_enc_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo encoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_enc.input.data; + output_head = output = op->turbo_enc.output.data; + in_offset = op->turbo_enc.input.offset; + out_offset = op->turbo_enc.output.offset; + mbuf_total_left = op->turbo_enc.input.length; + + c = op->turbo_enc.tb_params.c; + r = op->turbo_enc.tb_params.r; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_CRC_24B_ATTACH)) + crc24_bits = 24; + + while (mbuf_total_left > 0 && r < c && input != NULL) { + seg_total_left = rte_pktmbuf_data_len(input) - in_offset; + + e = (r < op->turbo_enc.tb_params.cab) ? + op->turbo_enc.tb_params.ea : + op->turbo_enc.tb_params.eb; + k = (r < op->turbo_enc.tb_params.c_neg) ? + op->turbo_enc.tb_params.k_neg : + op->turbo_enc.tb_params.k_pos; + ncb = (r < op->turbo_enc.tb_params.c_neg) ? + op->turbo_enc.tb_params.ncb_neg : + op->turbo_enc.tb_params.ncb_pos; + + in_length = ((k - crc24_bits) >> 3); + + if (check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_RATE_MATCH)) + out_length = (e + 7) >> 3; + else + out_length = (k >> 3) * 3 + 2; + + mbuf_append(output_head, output, out_length); + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_te_fill(op, &desc->enc_req, input, output, + k, e, ncb, in_offset, out_offset, ring_offset, + cbs_in_op); + if (unlikely(ret < 0)) + return ret; + + rte_bbdev_log_debug("DMA request desc %p", desc); + + /* Update lengths */ + op->turbo_enc.output.length += out_length; + mbuf_total_left -= in_length; + + /* Update offsets */ + if (seg_total_left == in_length) { + /* Go to the next mbuf */ + input = input->next; + output = output->next; + in_offset = 0; + out_offset = 0; + } else { + in_offset += in_length; + out_offset += out_length; + } + + r++; + desc_offset++; + current_enqueued_cbs++; + } + + if (mbuf_total_left > 0) { + rte_bbdev_log(ERR, + "Some date still left for processing: mbuf_total_left = %u", + mbuf_total_left); + return -1; + } + + return current_enqueued_cbs; +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Validates turbo decoder parameters */ +static int +validate_dec_op(struct rte_bbdev_dec_op *op) +{ + struct rte_bbdev_op_turbo_dec *turbo_dec = &op->turbo_dec; + struct rte_bbdev_op_dec_turbo_cb_params *cb = NULL; + struct rte_bbdev_op_dec_turbo_tb_params *tb = NULL; + + if (op->mempool == NULL) { + rte_bbdev_log(ERR, "Invalid mempool pointer"); + return -1; + } + if (turbo_dec->input.data == NULL) { + rte_bbdev_log(ERR, "Invalid input pointer"); + return -1; + } + if (turbo_dec->hard_output.data == NULL) { + rte_bbdev_log(ERR, "Invalid hard_output pointer"); + return -1; + } + if (turbo_dec->rv_index > 3) { + rte_bbdev_log(ERR, + "rv_index (%u) is out of range 0 <= value <= 3", + turbo_dec->rv_index); + return -1; + } + if (turbo_dec->iter_min < 1) { + rte_bbdev_log(ERR, + "iter_min (%u) is less than 1", + turbo_dec->iter_min); + return -1; + } + if (turbo_dec->iter_max <= 2) { + rte_bbdev_log(ERR, + "iter_max (%u) is less than or equal to 2", + turbo_dec->iter_max); + return -1; + } + if (turbo_dec->iter_min > turbo_dec->iter_max) { + rte_bbdev_log(ERR, + "iter_min (%u) is greater than iter_max (%u)", + turbo_dec->iter_min, turbo_dec->iter_max); + return -1; + } + if (turbo_dec->code_block_mode != 0 && + turbo_dec->code_block_mode != 1) { + rte_bbdev_log(ERR, + "code_block_mode (%u) is out of range 0 <= value <= 1", + turbo_dec->code_block_mode); + return -1; + } + + if (turbo_dec->code_block_mode == 0) { + + if ((turbo_dec->op_flags & + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP) && + !(turbo_dec->op_flags & RTE_BBDEV_TURBO_CRC_TYPE_24B)) { + rte_bbdev_log(ERR, + "RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP should accompany RTE_BBDEV_TURBO_CRC_TYPE_24B"); + return -1; + } + + tb = &turbo_dec->tb_params; + if ((tb->k_neg < RTE_BBDEV_MIN_CB_SIZE + || tb->k_neg > RTE_BBDEV_MAX_CB_SIZE) + && tb->c_neg > 0) { + rte_bbdev_log(ERR, + "k_neg (%u) is out of range %u <= value <= %u", + tb->k_neg, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if ((tb->k_pos < RTE_BBDEV_MIN_CB_SIZE + || tb->k_pos > RTE_BBDEV_MAX_CB_SIZE) + && tb->c > tb->c_neg) { + rte_bbdev_log(ERR, + "k_pos (%u) is out of range %u <= value <= %u", + tb->k_pos, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->c_neg > (RTE_BBDEV_MAX_CODE_BLOCKS - 1)) + rte_bbdev_log(ERR, + "c_neg (%u) is out of range 0 <= value <= %u", + tb->c_neg, + RTE_BBDEV_MAX_CODE_BLOCKS - 1); + if (tb->c < 1 || tb->c > RTE_BBDEV_MAX_CODE_BLOCKS) { + rte_bbdev_log(ERR, + "c (%u) is out of range 1 <= value <= %u", + tb->c, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + if (tb->cab > tb->c) { + rte_bbdev_log(ERR, + "cab (%u) is greater than c (%u)", + tb->cab, tb->c); + return -1; + } + } else { + + if (turbo_dec->op_flags & RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP) { + rte_bbdev_log(ERR, + "RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP is invalid in CB-mode"); + return -1; + } + + cb = &turbo_dec->cb_params; + if (cb->k < RTE_BBDEV_MIN_CB_SIZE + || cb->k > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k (%u) is out of range %u <= value <= %u", + cb->k, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + } + + return 0; +} +#endif + +static inline int +enqueue_dec_one_op_cb(struct fpga_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input; + struct rte_mbuf *output; + int ret; + uint16_t k, kw, ring_offset; + uint32_t total_left, in_length, out_length, in_offset, out_offset; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_dec_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo decoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_dec.input.data; + output = op->turbo_dec.hard_output.data; + total_left = op->turbo_dec.input.length; + in_offset = op->turbo_dec.input.offset; + out_offset = op->turbo_dec.hard_output.offset; + + k = op->turbo_dec.cb_params.k; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + in_length = kw; + out_length = k >> 3; + + mbuf_append(output, output, out_length); + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_td_fill(op, &desc->dec_req, input, output, + in_length, k, in_offset, out_offset, ring_offset, 1); + if (unlikely(ret < 0)) + return ret; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_dma_dec_desc_debug_info(desc); +#endif + + /* Update lengths */ + total_left -= in_length; + op->turbo_dec.hard_output.length += out_length; + + if (total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + total_left, in_length); + return -1; + } + + return 1; +} + + +static inline int +enqueue_dec_one_op_tb(struct fpga_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset, uint8_t cbs_in_op) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input, *output_head, *output; + int ret; + uint8_t r, c; + uint16_t k, kw, in_length, out_length, ring_offset; + uint32_t mbuf_total_left, seg_total_left, in_offset, out_offset; + uint16_t current_enqueued_cbs = 0; + uint16_t crc24_overlap = 0; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_dec_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo decoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_dec.input.data; + output_head = output = op->turbo_dec.hard_output.data; + mbuf_total_left = op->turbo_dec.input.length; + in_offset = op->turbo_dec.input.offset; + out_offset = op->turbo_dec.hard_output.offset; + + if (!check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP)) + crc24_overlap = 24; + + c = op->turbo_dec.tb_params.c; + r = op->turbo_dec.tb_params.r; + + while (mbuf_total_left > 0 && r < c && input != NULL) { + seg_total_left = rte_pktmbuf_data_len(input) - in_offset; + k = (r < op->turbo_dec.tb_params.c_neg) ? + op->turbo_dec.tb_params.k_neg : + op->turbo_dec.tb_params.k_pos; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + + in_length = kw; + out_length = (k - crc24_overlap) >> 3; + + mbuf_append(output_head, output, out_length); + + if (seg_total_left < in_length) { + rte_bbdev_log(ERR, + "Partial CB found in a TB. FPGA Driver doesn't support scatter-gather operations!"); + return -1; + } + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_td_fill(op, &desc->dec_req, input, output, + in_length, k, in_offset, out_offset, + ring_offset, cbs_in_op); + if (unlikely(ret < 0)) + return ret; + + /* Update lengths */ + ret = rte_pktmbuf_trim(op->turbo_dec.hard_output.data, + (crc24_overlap >> 3)); +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (ret < 0) { + rte_bbdev_log(ERR, + "The length to remove is greater than the length of the last segment"); + return -EINVAL; + } +#endif + op->turbo_dec.hard_output.length += out_length; + mbuf_total_left -= in_length; + + /* Update offsets */ + if (seg_total_left == in_length) { + /* Go to the next mbuf */ + input = input->next; + output = output->next; + in_offset = 0; + out_offset = 0; + } else { + in_offset += in_length; + out_offset += out_length; + } + + r++; + desc_offset++; + current_enqueued_cbs++; + } + + if (mbuf_total_left > 0) { + rte_bbdev_log(ERR, + "Some date still left for processing: mbuf_total_left = %u", + mbuf_total_left); + return -1; + } + + return current_enqueued_cbs; +} + +static uint16_t +fpga_enqueue_enc(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num) +{ + uint8_t cbs_in_op; + uint16_t i, total_enqueued_cbs = 0; + int32_t avail; + int enqueued_cbs; + struct fpga_queue *q = q_data->queue_private; + union fpga_dma_desc *desc; + + /* Check if queue is not full */ + if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == + q->head_free_desc)) + return 0; + + /* Calculates available space */ + avail = (q->head_free_desc > q->tail) ? + q->head_free_desc - q->tail - 1 : + q->ring_ctrl_reg.ring_size + q->head_free_desc - q->tail - 1; + + for (i = 0; i < num; ++i) { + if (ops[i]->turbo_enc.code_block_mode == 0) { + cbs_in_op = get_num_cbs_in_op_enc(&ops[i]->turbo_enc); + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - cbs_in_op < 0)) + break; + avail -= cbs_in_op; + enqueued_cbs = enqueue_enc_one_op_tb(q, ops[i], + total_enqueued_cbs, cbs_in_op); + } else { + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - 1 < 0)) + break; + avail -= 1; + enqueued_cbs = enqueue_enc_one_op_cb(q, ops[i], + total_enqueued_cbs); + } + + if (enqueued_cbs < 0) + break; + + total_enqueued_cbs += enqueued_cbs; + + rte_bbdev_log_debug("enqueuing enc ops [%d/%d] | head %d | tail %d", + total_enqueued_cbs, num, + q->head_free_desc, q->tail); + } + + /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt + * only when all previous CBs were already processed. + */ + desc = q->ring_addr + ((q->tail + total_enqueued_cbs - 1) + & q->sw_ring_wrap_mask); + desc->enc_req.irq_en = q->irq_enable; + + fpga_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); + + /* Update stats */ + q_data->queue_stats.enqueued_count += i; + q_data->queue_stats.enqueue_err_count += num - i; + + return i; +} + +static uint16_t +fpga_enqueue_dec(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num) +{ + uint8_t cbs_in_op; + uint16_t i, total_enqueued_cbs = 0; + int32_t avail; + int enqueued_cbs; + struct fpga_queue *q = q_data->queue_private; + union fpga_dma_desc *desc; + + /* Check if queue is not full */ + if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == + q->head_free_desc)) + return 0; + + /* Calculates available space */ + avail = (q->head_free_desc > q->tail) ? + q->head_free_desc - q->tail - 1 : + q->ring_ctrl_reg.ring_size + q->head_free_desc - q->tail - 1; + + for (i = 0; i < num; ++i) { + if (ops[i]->turbo_dec.code_block_mode == 0) { + cbs_in_op = get_num_cbs_in_op_dec(&ops[i]->turbo_dec); + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - cbs_in_op < 0)) + break; + avail -= cbs_in_op; + enqueued_cbs = enqueue_dec_one_op_tb(q, ops[i], + total_enqueued_cbs, cbs_in_op); + } else { + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - 1 < 0)) + break; + avail -= 1; + enqueued_cbs = enqueue_dec_one_op_cb(q, ops[i], + total_enqueued_cbs); + } + + if (enqueued_cbs < 0) + break; + + total_enqueued_cbs += enqueued_cbs; + + rte_bbdev_log_debug("enqueuing dec ops [%d/%d] | head %d | tail %d", + total_enqueued_cbs, num, + q->head_free_desc, q->tail); + } + + /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt + * only when all previous CBs were already processed. + */ + desc = q->ring_addr + ((q->tail + total_enqueued_cbs - 1) + & q->sw_ring_wrap_mask); + desc->dec_req.irq_en = q->irq_enable; + + fpga_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); + + /* Update stats */ + q_data->queue_stats.enqueued_count += i; + q_data->queue_stats.enqueue_err_count += num - i; + + return i; +} + +static inline int +dequeue_enc_one_op_cb(struct fpga_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + int desc_error = 0; + + /* Set current desc */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /*check if done */ + if (desc->enc_req.done == 0) + return -1; + + /* make sure the response is read atomically */ + rte_smp_rmb(); + + rte_bbdev_log_debug("DMA response desc %p", desc); + + *op = desc->enc_req.op_addr; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + (*op)->status = desc_error << RTE_BBDEV_DATA_ERROR; + + return 1; +} + +static inline int +dequeue_enc_one_op_tb(struct fpga_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + uint8_t cbs_in_op, cb_idx; + int desc_error = 0; + int status = 0; + + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify if done bit is set */ + if (desc->enc_req.done == 0) + return -1; + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + /* Verify if done bit in all CBs is set */ + cbs_in_op = desc->enc_req.cbs_in_op; + for (cb_idx = 1; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + if (desc->enc_req.done == 0) + return -1; + } + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + for (cb_idx = 0; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + status |= desc_error << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log_debug("DMA response desc %p", desc); + } + + *op = desc->enc_req.op_addr; + (*op)->status = status; + return cbs_in_op; +} + +static inline int +dequeue_dec_one_op_cb(struct fpga_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + int desc_error = 0; + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify done bit is set */ + if (desc->dec_req.done == 0) + return -1; + + /* make sure the response is read atomically */ + rte_smp_rmb(); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_dma_dec_desc_debug_info(desc); + +#endif + + *op = desc->dec_req.op_addr; + /* FPGA reports in half-iterations, from 0 to 31. get ceiling */ + (*op)->turbo_dec.iter_count = (desc->dec_req.iter + 2) >> 1; + /* crc_pass = 0 when decoder fails */ + (*op)->status = !(desc->dec_req.crc_pass) << RTE_BBDEV_CRC_ERROR; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + (*op)->status |= desc_error << RTE_BBDEV_DATA_ERROR; + return 1; +} + +static inline int +dequeue_dec_one_op_tb(struct fpga_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + uint8_t cbs_in_op, cb_idx, iter_count = 0; + int status = 0; + int desc_error = 0; + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify if done bit is set */ + if (desc->dec_req.done == 0) + return -1; + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + /* Verify if done bit in all CBs is set */ + cbs_in_op = desc->dec_req.cbs_in_op; + for (cb_idx = 1; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + if (desc->dec_req.done == 0) + return -1; + } + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + for (cb_idx = 0; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + /* get max iter_count for all CBs in op */ + iter_count = RTE_MAX(iter_count, (uint8_t) desc->dec_req.iter); + /* crc_pass = 0 when decoder fails, one fails all */ + status |= !(desc->dec_req.crc_pass) << RTE_BBDEV_CRC_ERROR; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + status |= desc_error << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log_debug("DMA response desc %p", desc); + } + + *op = desc->dec_req.op_addr; + + /* FPGA reports in half-iterations, get ceiling */ + (*op)->turbo_dec.iter_count = (iter_count + 2) >> 1; + (*op)->status = status; + return cbs_in_op; +} + +static uint16_t +fpga_dequeue_enc(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num) +{ + struct fpga_queue *q = q_data->queue_private; + uint32_t avail = (q->tail - q->head_free_desc) & q->sw_ring_wrap_mask; + uint16_t i; + uint16_t dequeued_cbs = 0; + struct rte_bbdev_enc_op *op; + int ret; + + for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { + op = (q->ring_addr + ((q->head_free_desc + dequeued_cbs) + & q->sw_ring_wrap_mask))->enc_req.op_addr; + if (op->turbo_enc.code_block_mode == 0) + ret = dequeue_enc_one_op_tb(q, &ops[i], dequeued_cbs); + else + ret = dequeue_enc_one_op_cb(q, &ops[i], dequeued_cbs); + + if (ret < 0) + break; + + dequeued_cbs += ret; + + rte_bbdev_log_debug("dequeuing enc ops [%d/%d] | head %d | tail %d", + dequeued_cbs, num, q->head_free_desc, q->tail); + } + + /* Update head */ + q->head_free_desc = (q->head_free_desc + dequeued_cbs) & + q->sw_ring_wrap_mask; + + /* Update stats */ + q_data->queue_stats.dequeued_count += i; + + return i; +} + +static uint16_t +fpga_dequeue_dec(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num) +{ + struct fpga_queue *q = q_data->queue_private; + uint32_t avail = (q->tail - q->head_free_desc) & q->sw_ring_wrap_mask; + uint16_t i; + uint16_t dequeued_cbs = 0; + struct rte_bbdev_dec_op *op; + int ret; + + for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { + op = (q->ring_addr + ((q->head_free_desc + dequeued_cbs) + & q->sw_ring_wrap_mask))->dec_req.op_addr; + if (op->turbo_dec.code_block_mode == 0) + ret = dequeue_dec_one_op_tb(q, &ops[i], dequeued_cbs); + else + ret = dequeue_dec_one_op_cb(q, &ops[i], dequeued_cbs); + + if (ret < 0) + break; + + dequeued_cbs += ret; + + rte_bbdev_log_debug("dequeuing dec ops [%d/%d] | head %d | tail %d", + dequeued_cbs, num, q->head_free_desc, q->tail); + } + + /* Update head */ + q->head_free_desc = (q->head_free_desc + dequeued_cbs) & + q->sw_ring_wrap_mask; + + /* Update stats */ + q_data->queue_stats.dequeued_count += i; + + return i; +} + +/* Initialization Function */ +static void +fpga_lte_fec_init(struct rte_bbdev *dev) +{ + struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev->device); + + dev->dev_ops = &fpga_ops; + dev->enqueue_enc_ops = fpga_enqueue_enc; + dev->enqueue_dec_ops = fpga_enqueue_dec; + dev->dequeue_enc_ops = fpga_dequeue_enc; + dev->dequeue_dec_ops = fpga_dequeue_dec; + + ((struct fpga_lte_fec_device *) dev->data->dev_private)->pf_device = + !strcmp(dev->device->driver->name, + RTE_STR(FPGA_LTE_FEC_PF_DRIVER_NAME)); + ((struct fpga_lte_fec_device *) dev->data->dev_private)->mmio_base = + pci_dev->mem_resource[0].addr; + + rte_bbdev_log_debug( + "Init device %s [%s] @ virtaddr %p phyaddr %#"PRIx64, + dev->device->driver->name, dev->data->name, + (void *)pci_dev->mem_resource[0].addr, + pci_dev->mem_resource[0].phys_addr); +} + +static int +fpga_lte_fec_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct rte_bbdev *bbdev = NULL; + char dev_name[RTE_BBDEV_NAME_MAX_LEN]; + + if (pci_dev == NULL) { + rte_bbdev_log(ERR, "NULL PCI device"); + return -EINVAL; + } + + rte_pci_device_name(&pci_dev->addr, dev_name, sizeof(dev_name)); + + /* Allocate memory to be used privately by drivers */ + bbdev = rte_bbdev_allocate(pci_dev->device.name); + if (bbdev == NULL) + return -ENODEV; + + /* allocate device private memory */ + bbdev->data->dev_private = rte_zmalloc_socket(dev_name, + sizeof(struct fpga_lte_fec_device), RTE_CACHE_LINE_SIZE, + pci_dev->device.numa_node); + + if (bbdev->data->dev_private == NULL) { + rte_bbdev_log(CRIT, + "Allocate of %zu bytes for device \"%s\" failed", + sizeof(struct fpga_lte_fec_device), dev_name); + rte_bbdev_release(bbdev); + return -ENOMEM; + } + + /* Fill HW specific part of device structure */ + bbdev->device = &pci_dev->device; + bbdev->intr_handle = &pci_dev->intr_handle; + bbdev->data->socket_id = pci_dev->device.numa_node; + + /* Invoke FEC FPGA device initialization function */ + fpga_lte_fec_init(bbdev); + + rte_bbdev_log_debug("bbdev id = %u [%s]", + bbdev->data->dev_id, dev_name); + + struct fpga_lte_fec_device *d = bbdev->data->dev_private; + uint32_t version_id = fpga_reg_read_32(d->mmio_base, + FPGA_LTE_FEC_VERSION_ID); + rte_bbdev_log(INFO, "FEC FPGA RTL v%u.%u", + ((uint16_t)(version_id >> 16)), ((uint16_t)version_id)); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (!strcmp(bbdev->device->driver->name, + RTE_STR(FPGA_LTE_FEC_PF_DRIVER_NAME))) + print_static_reg_debug_info(d->mmio_base); +#endif + return 0; +} + +static int +fpga_lte_fec_remove(struct rte_pci_device *pci_dev) +{ + struct rte_bbdev *bbdev; + int ret; + uint8_t dev_id; + + if (pci_dev == NULL) + return -EINVAL; + + /* Find device */ + bbdev = rte_bbdev_get_named_dev(pci_dev->device.name); + if (bbdev == NULL) { + rte_bbdev_log(CRIT, + "Couldn't find HW dev \"%s\" to uninitialise it", + pci_dev->device.name); + return -ENODEV; + } + dev_id = bbdev->data->dev_id; + + /* free device private memory before close */ + rte_free(bbdev->data->dev_private); + + /* Close device */ + ret = rte_bbdev_close(dev_id); + if (ret < 0) + rte_bbdev_log(ERR, + "Device %i failed to close during uninit: %i", + dev_id, ret); + + /* release bbdev from library */ + ret = rte_bbdev_release(bbdev); + if (ret) + rte_bbdev_log(ERR, "Device %i failed to uninit: %i", dev_id, + ret); + + rte_bbdev_log_debug("Destroyed bbdev = %u", dev_id); + + return 0; +} + +static inline void +set_default_fpga_conf(struct fpga_lte_fec_conf *def_conf) +{ + /* clear default configuration before initialization */ + memset(def_conf, 0, sizeof(struct fpga_lte_fec_conf)); + /* Set pf mode to true */ + def_conf->pf_mode_en = true; + + /* Set ratio between UL and DL to 1:1 (unit of weight is 3 CBs) */ + def_conf->ul_bandwidth = 3; + def_conf->dl_bandwidth = 3; + + /* Set Load Balance Factor to 64 */ + def_conf->dl_load_balance = 64; + def_conf->ul_load_balance = 64; +} + +/* Initial configuration of FPGA LTE FEC device */ +int +fpga_lte_fec_configure(const char *dev_name, + const struct fpga_lte_fec_conf *conf) +{ + uint32_t payload_32, address; + uint16_t payload_16; + uint8_t payload_8; + uint16_t q_id, vf_id, total_q_id, total_ul_q_id, total_dl_q_id; + struct rte_bbdev *bbdev = rte_bbdev_get_named_dev(dev_name); + struct fpga_lte_fec_conf def_conf; + + if (bbdev == NULL) { + rte_bbdev_log(ERR, + "Invalid dev_name (%s), or device is not yet initialised", + dev_name); + return -ENODEV; + } + + struct fpga_lte_fec_device *d = bbdev->data->dev_private; + + if (conf == NULL) { + rte_bbdev_log(ERR, + "FPGA Configuration was not provided. Default configuration will be loaded."); + set_default_fpga_conf(&def_conf); + conf = &def_conf; + } + + /* + * Configure UL:DL ratio. + * [7:0]: UL weight + * [15:8]: DL weight + */ + payload_16 = (conf->dl_bandwidth << 8) | conf->ul_bandwidth; + address = FPGA_LTE_FEC_CONFIGURATION; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Clear all queues registers */ + payload_32 = FPGA_INVALID_HW_QUEUE_ID; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + address = (q_id << 2) + FPGA_LTE_FEC_QUEUE_MAP; + fpga_reg_write_32(d->mmio_base, address, payload_32); + } + + /* + * If PF mode is enabled allocate all queues for PF only. + * + * For VF mode each VF can have different number of UL and DL queues. + * Total number of queues to configure cannot exceed FPGA + * capabilities - 64 queues - 32 queues for UL and 32 queues for DL. + * Queues mapping is done according to configuration: + * + * UL queues: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | ... | 0 | + * | conf->vf_dl_queues_number[0] - 1 | 0 | + * | conf->vf_dl_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_dl_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_dl_queues_number[7] - 1 | 7 | + * + * DL queues: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | ... | 0 | + * | conf->vf_ul_queues_number[0] - 1 | 0 | + * | conf->vf_ul_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_ul_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_ul_queues_number[7] - 1 | 7 | + * + * Example of configuration: + * conf->vf_ul_queues_number[0] = 4; -> 4 UL queues for VF0 + * conf->vf_dl_queues_number[0] = 4; -> 4 DL queues for VF0 + * conf->vf_ul_queues_number[1] = 2; -> 2 UL queues for VF1 + * conf->vf_dl_queues_number[1] = 2; -> 2 DL queues for VF1 + * + * UL: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | 1 | 0 | + * | 2 | 0 | + * | 3 | 0 | + * | 4 | 1 | + * | 5 | 1 | + * + * DL: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | 33 | 0 | + * | 34 | 0 | + * | 35 | 0 | + * | 36 | 1 | + * | 37 | 1 | + */ + if (conf->pf_mode_en) { + payload_32 = 0x1; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + address = (q_id << 2) + FPGA_LTE_FEC_QUEUE_MAP; + fpga_reg_write_32(d->mmio_base, address, payload_32); + } + } else { + /* Calculate total number of UL and DL queues to configure */ + total_ul_q_id = total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + total_ul_q_id += conf->vf_ul_queues_number[vf_id]; + total_dl_q_id += conf->vf_dl_queues_number[vf_id]; + } + total_q_id = total_dl_q_id + total_ul_q_id; + /* + * Check if total number of queues to configure does not exceed + * FPGA capabilities (64 queues - 32 UL and 32 DL queues) + */ + if ((total_ul_q_id > FPGA_NUM_UL_QUEUES) || + (total_dl_q_id > FPGA_NUM_DL_QUEUES) || + (total_q_id > FPGA_TOTAL_NUM_QUEUES)) { + rte_bbdev_log(ERR, + "FPGA Configuration failed. Too many queues to configure: UL_Q %u, DL_Q %u, FPGA_Q %u", + total_ul_q_id, total_dl_q_id, + FPGA_TOTAL_NUM_QUEUES); + return -EINVAL; + } + total_ul_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_ul_queues_number[vf_id]; + ++q_id, ++total_ul_q_id) { + address = (total_ul_q_id << 2) + + FPGA_LTE_FEC_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_reg_write_32(d->mmio_base, address, + payload_32); + } + } + total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_dl_queues_number[vf_id]; + ++q_id, ++total_dl_q_id) { + address = ((total_dl_q_id + FPGA_NUM_UL_QUEUES) + << 2) + FPGA_LTE_FEC_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_reg_write_32(d->mmio_base, address, + payload_32); + } + } + } + + /* Setting Load Balance Factor */ + payload_16 = (conf->dl_load_balance << 8) | (conf->ul_load_balance); + address = FPGA_LTE_FEC_LOAD_BALANCE_FACTOR; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting length of ring descriptor entry */ + payload_16 = FPGA_RING_DESC_ENTRY_LENGTH; + address = FPGA_LTE_FEC_RING_DESC_LEN; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting FLR timeout value */ + payload_16 = conf->flr_time_out; + address = FPGA_LTE_FEC_FLR_TIME_OUT; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Queue PF/VF mapping table is ready */ + payload_8 = 0x1; + address = FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE; + fpga_reg_write_8(d->mmio_base, address, payload_8); + + rte_bbdev_log_debug("PF FPGA LTE FEC configuration complete for %s", + dev_name); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_static_reg_debug_info(d->mmio_base); +#endif + return 0; +} + +/* FPGA LTE FEC PCI PF address map */ +static struct rte_pci_id pci_id_fpga_lte_fec_pf_map[] = { + { + RTE_PCI_DEVICE(FPGA_LTE_FEC_VENDOR_ID, + FPGA_LTE_FEC_PF_DEVICE_ID) + }, + {.device_id = 0}, +}; + +static struct rte_pci_driver fpga_lte_fec_pci_pf_driver = { + .probe = fpga_lte_fec_probe, + .remove = fpga_lte_fec_remove, + .id_table = pci_id_fpga_lte_fec_pf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING +}; + +/* FPGA LTE FEC PCI VF address map */ +static struct rte_pci_id pci_id_fpga_lte_fec_vf_map[] = { + { + RTE_PCI_DEVICE(FPGA_LTE_FEC_VENDOR_ID, + FPGA_LTE_FEC_VF_DEVICE_ID) + }, + {.device_id = 0}, +}; + +static struct rte_pci_driver fpga_lte_fec_pci_vf_driver = { + .probe = fpga_lte_fec_probe, + .remove = fpga_lte_fec_remove, + .id_table = pci_id_fpga_lte_fec_vf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING +}; + + +RTE_PMD_REGISTER_PCI(FPGA_LTE_FEC_PF_DRIVER_NAME, fpga_lte_fec_pci_pf_driver); +RTE_PMD_REGISTER_PCI_TABLE(FPGA_LTE_FEC_PF_DRIVER_NAME, + pci_id_fpga_lte_fec_pf_map); +RTE_PMD_REGISTER_PCI(FPGA_LTE_FEC_VF_DRIVER_NAME, fpga_lte_fec_pci_vf_driver); +RTE_PMD_REGISTER_PCI_TABLE(FPGA_LTE_FEC_VF_DRIVER_NAME, + pci_id_fpga_lte_fec_vf_map); + +RTE_INIT(fpga_lte_fec_init_log) +{ + fpga_lte_fec_logtype = rte_log_register("pmd.bb.fpga_lte_fec"); + if (fpga_lte_fec_logtype >= 0) +#ifdef RTE_LIBRTE_BBDEV_DEBUG + rte_log_set_level(fpga_lte_fec_logtype, RTE_LOG_DEBUG); +#else + rte_log_set_level(fpga_lte_fec_logtype, RTE_LOG_NOTICE); +#endif +} diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h new file mode 100644 index 0000000..2c6952e --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2018 Intel Corporation + */ + +#ifndef _FPGA_LTE_FEC_H_ +#define _FPGA_LTE_FEC_H_ + +#include <stdint.h> +#include <stdbool.h> + +/** + * @file fpga_lte_fec.h + * + * Interface for Intel(R) FGPA LTE FEC device configuration at the host level, + * directly accessible by the application. + * Configuration related to LTE Turbo coding functionality is done through + * librte_bbdev library. + * + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + */ + +#ifdef __cplusplus +extern "C" { +#endif + +/**< Number of Virtual Functions FGPA 4G FEC supports */ +#define FPGA_LTE_FEC_NUM_VFS 8 + +/** + * Structure to pass FPGA 4G FEC configuration. + */ +struct fpga_lte_fec_conf { + /**< 1 if PF is used for dataplane, 0 for VFs */ + bool pf_mode_en; + /**< Number of UL queues per VF */ + uint8_t vf_ul_queues_number[FPGA_LTE_FEC_NUM_VFS]; + /**< Number of DL queues per VF */ + uint8_t vf_dl_queues_number[FPGA_LTE_FEC_NUM_VFS]; + /**< UL bandwidth. Needed for schedule algorithm */ + uint8_t ul_bandwidth; + /**< DL bandwidth. Needed for schedule algorithm */ + uint8_t dl_bandwidth; + /**< UL Load Balance */ + uint8_t ul_load_balance; + /**< DL Load Balance */ + uint8_t dl_load_balance; + /**< FLR timeout value */ + uint16_t flr_time_out; +}; + +/** + * Configure Intel(R) FPGA LTE FEC device + * + * @param dev_name + * The name of the device. This is the short form of PCI BDF, e.g. 00:01.0. + * It can also be retrieved for a bbdev device from the dev_name field in the + * rte_bbdev_info structure returned by rte_bbdev_info_get(). + * @param conf + * Configuration to apply to FPGA 4G FEC. + * + * @return + * Zero on success, negative value on failure. + */ +int +fpga_lte_fec_configure(const char *dev_name, + const struct fpga_lte_fec_conf *conf); + +#ifdef __cplusplus +} +#endif + +#endif /* _FPGA_LTE_FEC_H_ */ diff --git a/drivers/baseband/fpga_lte_fec/meson.build b/drivers/baseband/fpga_lte_fec/meson.build new file mode 100644 index 0000000..bf44e6b --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +deps += ['bbdev', 'bus_vdev', 'ring', 'pci', 'bus_pci'] +name = 'bbdev_fpga_lte_fec' +allow_experimental_apis = true +sources = files('fpga_lte_fec.c') diff --git a/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map new file mode 100644 index 0000000..1a99f8d --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map @@ -0,0 +1,3 @@ +DPDK_18.08 { + local: *; +}; diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build index 52489df..a17d549 100644 --- a/drivers/baseband/meson.build +++ b/drivers/baseband/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <bluca@debian.org> -drivers = ['null'] +drivers = ['null', 'fpga_lte_fec'] config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' driver_name_fmt = 'rte_pmd_@0@' diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 7c9b4b5..21d32a2 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -218,6 +218,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_NETVSC_PMD) += -lrte_pmd_netvsc ifeq ($(CONFIG_RTE_LIBRTE_BBDEV),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += -lrte_pmd_fpga_lte_fec # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v4] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-06 9:04 ` Nicolas Chautru @ 2019-06-08 0:17 ` Nicolas Chautru 2019-06-08 0:17 ` Nicolas Chautru 2019-06-10 17:01 ` Nicolas Chautru 1 sibling, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-08 0:17 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Update v4: Fix warning for the DEBUG configuration Update v3: Squashing 3 previous patches into one as recommended. This is adding a new PMD driver for BBDEV device based on FPGA implementation on PAC N3000 HW to provide FEC 4G acceleration. v1 was shared earlier on this patchwork : https://patches.dpdk.org/patch/53409/ The existing BBDEV test framework is still supported. Main updates based on feedback during the last v1 were: - correct the documentation which was arguably misleading in hinting that ICC was required. There is no dependency whatsoever. - add meson build support Note that a number of other BBDEV patches are in v1 here and will be resubmitted for 19.08 rc1 https://patches.dpdk.org/project/dpdk/list/?series=4657 Nicolas Chautru (1): baseband/fpga_lte_fec: adding driver for FEC on FPGA config/common_base | 6 + doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++ doc/guides/bbdevs/index.rst | 1 + drivers/baseband/Makefile | 2 + drivers/baseband/fpga_lte_fec/Makefile | 29 + drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 ++++++++++++++++++++ drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + drivers/baseband/fpga_lte_fec/meson.build | 7 + .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + drivers/baseband/meson.build | 2 +- mk/rte.app.mk | 1 + 11 files changed, 3115 insertions(+), 1 deletion(-) create mode 100644 doc/guides/bbdevs/fpga_lte_fec.rst create mode 100644 drivers/baseband/fpga_lte_fec/Makefile create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h create mode 100644 drivers/baseband/fpga_lte_fec/meson.build create mode 100644 drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v4] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-08 0:17 ` [dpdk-dev] [PATCH v4] " Nicolas Chautru @ 2019-06-08 0:17 ` Nicolas Chautru 0 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-08 0:17 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Supports for FEC 4G PMD Driver on FPGA card PAC N3000 Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 6 + doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++ doc/guides/bbdevs/index.rst | 1 + drivers/baseband/Makefile | 2 + drivers/baseband/fpga_lte_fec/Makefile | 29 + drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 ++++++++++++++++++++ drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + drivers/baseband/fpga_lte_fec/meson.build | 7 + .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + drivers/baseband/meson.build | 2 +- mk/rte.app.mk | 1 + 11 files changed, 3115 insertions(+), 1 deletion(-) create mode 100644 doc/guides/bbdevs/fpga_lte_fec.rst create mode 100644 drivers/baseband/fpga_lte_fec/Makefile create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h create mode 100644 drivers/baseband/fpga_lte_fec/meson.build create mode 100644 drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map diff --git a/config/common_base b/config/common_base index 6b96e0e..ae52028 100644 --- a/config/common_base +++ b/config/common_base @@ -521,6 +521,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y # EXPERIMENTAL: API may change without prior notice # CONFIG_RTE_LIBRTE_BBDEV=y +CONFIG_RTE_LIBRTE_BBDEV_DEBUG=n CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y @@ -535,6 +536,11 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n # +# Compile PMD for Intel FPGA LTE FEC bbdev device +# +CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC=y + +# # Compile generic crypto device library # CONFIG_RTE_LIBRTE_CRYPTODEV=y diff --git a/doc/guides/bbdevs/fpga_lte_fec.rst b/doc/guides/bbdevs/fpga_lte_fec.rst new file mode 100644 index 0000000..71b058c --- /dev/null +++ b/doc/guides/bbdevs/fpga_lte_fec.rst @@ -0,0 +1,318 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation + +Intel(R) FPGA LTE FEC Poll Mode Driver +====================================== + +The BBDEV FPGA LTE FEC poll mode driver (PMD) supports an FPGA implementation of a VRAN +Turbo Encode / Decode LTE wireless acceleration function, using Intel's PCI-e and FPGA +based Vista Creek device. + +Features +-------- + +FPGA LTE FEC PMD supports the following features: + +- Turbo Encode in the DL with total throughput of 4.5 Gbits/s +- Turbo Decode in the UL with total throughput of 1.5 Gbits/s assuming 8 decoder iterations +- 8 VFs per PF (physical device) +- Maximum of 32 UL queues per VF +- Maximum of 32 DL queues per VF +- PCIe Gen-3 x8 Interface +- MSI-X +- SR-IOV + + +FPGA LTE FEC PMD supports the following BBDEV capabilities: + +* For the turbo encode operation: + - ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` : set to attach CRC24B to CB(s) + - ``RTE_BBDEV_TURBO_RATE_MATCH`` : if set then do not do Rate Match bypass + - ``RTE_BBDEV_TURBO_ENC_INTERRUPTS`` : set for encoder dequeue interrupts + + +* For the turbo decode operation: + - ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` : check CRC24B from CB(s) + - ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` : perform subblock de-interleave + - ``RTE_BBDEV_TURBO_DEC_INTERRUPTS`` : set for decoder dequeue interrupts + - ``RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN`` : set if negative LLR encoder i/p is supported + - ``RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP`` : keep CRC24B bits appended while decoding + + +Limitations +----------- + +FPGA LTE FEC does not support the following: + +- Scatter-Gather function + + +Installation +-------------- + +Section 3 of the DPDK manual provides instuctions on installing and compiling DPDK. The +default set of bbdev compile flags may be found in config/common_base, where for example +the flag to build the FPGA LTE FEC device, ``CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC``, is already +set. It is assumed DPDK has been compiled using for instance: + +.. code-block:: console + + make install T=x86_64-native-linuxapp-gcc + + +DPDK requires hugepages to be configured as detailed in section 2 of the DPDK manual. +The bbdev test application has been tested with a configuration 40 x 1GB hugepages. The +hugepage configuration of a server may be examined using: + +.. code-block:: console + + grep Huge* /proc/meminfo + + +Initialization +-------------- + +When the device first powers up, its PCI Physical Functions (PF) can be listed through this command: + +.. code-block:: console + + sudo lspci -vd1172:5052 + +The physical and virtual functions are compatible with Linux UIO drivers: +``vfio`` and ``igb_uio``. However, in order to work the FPGA LTE FEC device firstly needs +to be bound to one of these linux drivers through DPDK. + + +Bind PF UIO driver(s) +~~~~~~~~~~~~~~~~~~~~~ + +Install the DPDK igb_uio driver, bind it with the PF PCI device ID and use +``lspci`` to confirm the PF device is under use by ``igb_uio`` DPDK UIO driver. + +The igb_uio driver may be bound to the PF PCI device using one of three methods: + + +1. PCI functions (physical or virtual, depending on the use case) can be bound to +the UIO driver by repeating this command for every function. + +.. code-block:: console + + cd <dpdk-top-level-directory> + insmod ./build/kmod/igb_uio.ko + echo "1172 5052" > /sys/bus/pci/drivers/igb_uio/new_id + lspci -vd1172: + + +2. Another way to bind PF with DPDK UIO driver is by using the ``dpdk-devbind.py`` tool + +.. code-block:: console + + cd <dpdk-top-level-directory> + ./usertools/dpdk-devbind.py -b igb_uio 0000:06:00.0 + +where the PCI device ID (example: 0000:06:00.0) is obtained using lspci -vd1172: + + +3. A third way to bind is to use ``dpdk-setup.sh`` tool + +.. code-block:: console + + cd <dpdk-top-level-directory> + ./usertools/dpdk-setup.sh + + select 'Bind Ethernet/Crypto/Baseband device to IGB UIO module' + or + select 'Bind Ethernet/Crypto/Baseband device to VFIO module' depending on driver required + enter PCI device ID + select 'Display current Ethernet/Crypto/Baseband device settings' to confirm binding + + +In the same way the FPGA LTE FEC PF can be bound with vfio, but vfio driver does not +support SR-IOV configuration right out of the box, so it will need to be patched. + + +Enable Virtual Functions +~~~~~~~~~~~~~~~~~~~~~~~~ + +Now, it should be visible in the printouts that PCI PF is under igb_uio control +"``Kernel driver in use: igb_uio``" + +To show the number of available VFs on the device, read ``sriov_totalvfs`` file.. + +.. code-block:: console + + cat /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_totalvfs + + where 0000\:<b>\:<d>.<f> is the PCI device ID + + +To enable VFs via igb_uio, echo the number of virtual functions intended to +enable to ``max_vfs`` file.. + +.. code-block:: console + + echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/max_vfs + + +Afterwards, all VFs must be bound to appropriate UIO drivers as required, same +way it was done with the physical function previously. + +Enabling SR-IOV via vfio driver is pretty much the same, except that the file +name is different: + +.. code-block:: console + + echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_numvfs + + +Configure the VFs through PF +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The PCI virtual functions must be configured before working or getting assigned +to VMs/Containers. The configuration involves allocating the number of hardware +queues, priorities, load balance, bandwidth and other settings necessary for the +device to perform FEC functions. + +This configuration needs to be executed at least once after reboot or PCI FLR and can +be achieved by using the function ``fpga_lte_fec_configure()``, which sets up the +parameters defined in ``fpga_lte_fec_conf`` structure: + +.. code-block:: c + + struct fpga_lte_fec_conf { + bool pf_mode_en; + uint8_t vf_ul_queues_number[FPGA_LTE_FEC_NUM_VFS]; + uint8_t vf_dl_queues_number[FPGA_LTE_FEC_NUM_VFS]; + uint8_t ul_bandwidth; + uint8_t dl_bandwidth; + uint8_t ul_load_balance; + uint8_t dl_load_balance; + uint16_t flr_time_out; + }; + +- ``pf_mode_en``: identifies whether only PF is to be used, or the VFs. PF and + VFs are mutually exclusive and cannot run simultaneously. + Set to 1 for PF mode enabled. + If PF mode is enabled all queues available in the device are assigned + exclusively to PF and 0 queues given to VFs. + +- ``vf_*l_queues_number``: defines the hardware queue mapping for every VF. + +- ``*l_bandwidth``: in case of congestion on PCIe interface. The device + allocates different bandwidth to UL and DL. The weight is configured by this + setting. The unit of weight is 3 code blocks. For example, if the code block + cbps (code block per second) ratio between UL and DL is 12:1, then the + configuration value should be set to 36:3. The schedule algorithm is based + on code block regardless the length of each block. + +- ``*l_load_balance``: hardware queues are load-balanced in a round-robin + fashion. Queues get filled first-in first-out until they reach a pre-defined + watermark level, if exceeded, they won't get assigned new code blocks.. + This watermark is defined by this setting. + + If all hardware queues exceeds the watermark, no code blocks will be + streamed in from UL/DL code block FIFO. + +- ``flr_time_out``: specifies how many 16.384us to be FLR time out. The + time_out = flr_time_out x 16.384us. For instance, if you want to set 10ms for + the FLR time out then set this setting to 0x262=610. + + +An example configuration code calling the function ``fpga_lte_fec_configure()`` is shown +below: + +.. code-block:: c + + struct fpga_lte_fec_conf conf; + unsigned int i; + + memset(&conf, 0, sizeof(struct fpga_lte_fec_conf)); + conf.pf_mode_en = 1; + + for (i = 0; i < FPGA_LTE_FEC_NUM_VFS; ++i) { + conf.vf_ul_queues_number[i] = 4; + conf.vf_dl_queues_number[i] = 4; + } + conf.ul_bandwidth = 12; + conf.dl_bandwidth = 5; + conf.dl_load_balance = 64; + conf.ul_load_balance = 64; + + /* setup FPGA PF */ + ret = fpga_lte_fec_configure(info->dev_name, &conf); + TEST_ASSERT_SUCCESS(ret, + "Failed to configure 4G FPGA PF for bbdev %s", + info->dev_name); + + +Test Application +---------------- + +BBDEV provides a test application, ``test-bbdev.py`` and range of test data for testing +the functionality of FPGA LTE FEC turbo encode and turbo decode, depending on the device's +capabilities. The test application is located under app->test-bbdev folder and has the +following options: + +.. code-block:: console + + "-p", "--testapp-path": specifies path to the bbdev test app. + "-e", "--eal-params" : EAL arguments which are passed to the test app. + "-t", "--timeout" : Timeout in seconds (default=300). + "-c", "--test-cases" : Defines test cases to run. Run all if not specified. + "-v", "--test-vector" : Test vector path (default=dpdk_path+/app/test-bbdev/test_vectors/bbdev_null.data). + "-n", "--num-ops" : Number of operations to process on device (default=32). + "-b", "--burst-size" : Operations enqueue/dequeue burst size (default=32). + "-l", "--num-lcores" : Number of lcores to run (default=16). + "-i", "--init-device" : Initialise PF device with default values. + + +To execute the test application tool using simple turbo decode or turbo encode data, +type one of the following: + +.. code-block:: console + + ./test-bbdev.py -c validation -n 64 -b 8 -v ./turbo_dec_default.data + ./test-bbdev.py -c validation -n 64 -b 8 -v ./turbo_enc_default.data + + +The test application ``test-bbdev.py``, supports the ability to configure the PF device with +a default set of values, if the "-i" or "- -init-device" option is included. The default values +are defined in test_bbdev_perf.c as: + +- VF_UL_QUEUE_VALUE 4 +- VF_DL_QUEUE_VALUE 4 +- UL_BANDWIDTH 3 +- DL_BANDWIDTH 3 +- UL_LOAD_BALANCE 128 +- DL_LOAD_BALANCE 128 +- FLR_TIMEOUT 610 + + +Test Vectors +~~~~~~~~~~~~ + +In addition to the simple turbo decoder and turbo encoder tests, bbdev also provides +a range of additional tests under the test_vectors folder, which may be useful. The results +of these tests will depend on the FPGA LTE FEC capabilities: + +* turbo decoder tests: + - ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_high_snr.data`` + - ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_low_snr.data`` + - ``turbo_dec_c1_k6144_r0_e34560_negllr.data`` + - ``turbo_dec_c1_k6144_r0_e34560_sbd_negllr.data`` + - ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr_crc24b.data`` + - ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr.data`` + + +* turbo encoder tests: + - ``turbo_enc_c1_k40_r0_e1190_rm.data`` + - ``turbo_enc_c1_k40_r0_e1194_rm.data`` + - ``turbo_enc_c1_k40_r0_e1196_rm.data`` + - ``turbo_enc_c1_k40_r0_e272_rm.data`` + - ``turbo_enc_c1_k6144_r0_e18444.data`` + - ``turbo_enc_c1_k6144_r0_e32256_crc24b_rm.data`` + - ``turbo_enc_c2_k5952_r0_e17868_crc24b.data`` + - ``turbo_enc_c3_k4800_r2_e14412_crc24b.data`` + - ``turbo_enc_c4_k4800_r2_e14412_crc24b.data`` + + diff --git a/doc/guides/bbdevs/index.rst b/doc/guides/bbdevs/index.rst index 93276ed..005b95e 100644 --- a/doc/guides/bbdevs/index.rst +++ b/doc/guides/bbdevs/index.rst @@ -10,3 +10,4 @@ Baseband Device Drivers null turbo_sw + fpga_lte_fec diff --git a/drivers/baseband/Makefile b/drivers/baseband/Makefile index 4ec83b0..ceffc7d 100644 --- a/drivers/baseband/Makefile +++ b/drivers/baseband/Makefile @@ -10,5 +10,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += null DEPDIRS-null = $(core-libs) DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += turbo_sw DEPDIRS-turbo_sw = $(core-libs) +DIRS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += fpga_lte_fec +DEPDIRS-fpga_lte_fec = $(core-libs) include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/drivers/baseband/fpga_lte_fec/Makefile b/drivers/baseband/fpga_lte_fec/Makefile new file mode 100644 index 0000000..e478d84 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/Makefile @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2018 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_pmd_fpga_lte_fec.a + +# build flags +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring +LDLIBS += -lrte_bbdev +LDLIBS += -lrte_pci -lrte_bus_pci + +# versioning export map +EXPORT_MAP := rte_pmd_bbdev_fpga_lte_fec_version.map + +# library version +LIBABIVER := 1 + +# library source files +SRCS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += fpga_lte_fec.c + +# export include files +SYMLINK-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC)-include += fpga_lte_fec.h + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c new file mode 100644 index 0000000..856b68e --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c @@ -0,0 +1,2674 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2018 Intel Corporation + */ + +#include <unistd.h> + +#include <rte_common.h> +#include <rte_log.h> +#include <rte_dev.h> +#include <rte_malloc.h> +#include <rte_mempool.h> +#include <rte_errno.h> +#include <rte_pci.h> +#include <rte_bus_pci.h> +#include <rte_byteorder.h> +#ifdef RTE_BBDEV_OFFLOAD_COST +#include <rte_cycles.h> +#endif + +#include <rte_bbdev.h> +#include <rte_bbdev_pmd.h> + +#include "fpga_lte_fec.h" + +/* Turbo SW PMD logging ID */ +static int fpga_lte_fec_logtype; + +/* Helper macro for logging */ +#define rte_bbdev_log(level, fmt, ...) \ + rte_log(RTE_LOG_ ## level, fpga_lte_fec_logtype, fmt "\n", \ + ##__VA_ARGS__) + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +#define rte_bbdev_log_debug(fmt, ...) \ + rte_bbdev_log(DEBUG, "fpga_lte_fec: " fmt, \ + ##__VA_ARGS__) +#else +#define rte_bbdev_log_debug(fmt, ...) +#endif + +/* FPGA LTE FEC driver names */ +#define FPGA_LTE_FEC_PF_DRIVER_NAME intel_fpga_lte_fec_pf +#define FPGA_LTE_FEC_VF_DRIVER_NAME intel_fpga_lte_fec_vf + +/* FPGA LTE FEC PCI vendor & device IDs */ +#define FPGA_LTE_FEC_VENDOR_ID (0x1172) +#define FPGA_LTE_FEC_PF_DEVICE_ID (0x5052) +#define FPGA_LTE_FEC_VF_DEVICE_ID (0x5050) + +/* Align DMA descriptors to 256 bytes - cache-aligned */ +#define FPGA_RING_DESC_ENTRY_LENGTH (8) +/* Ring size is in 256 bits (32 bytes) units */ +#define FPGA_RING_DESC_LEN_UNIT_BYTES (32) +/* Maximum size of queue */ +#define FPGA_RING_MAX_SIZE (1024) +#define FPGA_FLR_TIMEOUT_UNIT (16.384) + +#define FPGA_NUM_UL_QUEUES (32) +#define FPGA_NUM_DL_QUEUES (32) +#define FPGA_TOTAL_NUM_QUEUES (FPGA_NUM_UL_QUEUES + FPGA_NUM_DL_QUEUES) +#define FPGA_NUM_INTR_VEC (FPGA_TOTAL_NUM_QUEUES - RTE_INTR_VEC_RXTX_OFFSET) + +#define FPGA_INVALID_HW_QUEUE_ID (0xFFFFFFFF) + +#define FPGA_QUEUE_FLUSH_TIMEOUT_US (1000) +#define FPGA_TIMEOUT_CHECK_INTERVAL (5) + +/* FPGA LTE FEC Register mapping on BAR0 */ +enum { + FPGA_LTE_FEC_VERSION_ID = 0x00000000, /* len: 4B */ + FPGA_LTE_FEC_CONFIGURATION = 0x00000004, /* len: 2B */ + FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE = 0x00000008, /* len: 1B */ + FPGA_LTE_FEC_LOAD_BALANCE_FACTOR = 0x0000000a, /* len: 2B */ + FPGA_LTE_FEC_RING_DESC_LEN = 0x0000000c, /* len: 2B */ + FPGA_LTE_FEC_FLR_TIME_OUT = 0x0000000e, /* len: 2B */ + FPGA_LTE_FEC_VFQ_FLUSH_STATUS_LW = 0x00000018, /* len: 4B */ + FPGA_LTE_FEC_VFQ_FLUSH_STATUS_HI = 0x0000001c, /* len: 4B */ + FPGA_LTE_FEC_VF0_DEBUG = 0x00000020, /* len: 4B */ + FPGA_LTE_FEC_VF1_DEBUG = 0x00000024, /* len: 4B */ + FPGA_LTE_FEC_VF2_DEBUG = 0x00000028, /* len: 4B */ + FPGA_LTE_FEC_VF3_DEBUG = 0x0000002c, /* len: 4B */ + FPGA_LTE_FEC_VF4_DEBUG = 0x00000030, /* len: 4B */ + FPGA_LTE_FEC_VF5_DEBUG = 0x00000034, /* len: 4B */ + FPGA_LTE_FEC_VF6_DEBUG = 0x00000038, /* len: 4B */ + FPGA_LTE_FEC_VF7_DEBUG = 0x0000003c, /* len: 4B */ + FPGA_LTE_FEC_QUEUE_MAP = 0x00000040, /* len: 256B */ + FPGA_LTE_FEC_RING_CTRL_REGS = 0x00000200 /* len: 2048B */ +}; + +/* FPGA LTE FEC Ring Control Registers */ +enum { + FPGA_LTE_FEC_RING_HEAD_ADDR = 0x00000008, + FPGA_LTE_FEC_RING_SIZE = 0x00000010, + FPGA_LTE_FEC_RING_MISC = 0x00000014, + FPGA_LTE_FEC_RING_ENABLE = 0x00000015, + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN = 0x00000016, + FPGA_LTE_FEC_RING_SHADOW_TAIL = 0x00000018, + FPGA_LTE_FEC_RING_HEAD_POINT = 0x0000001C +}; + +/* FPGA LTE FEC DESCRIPTOR ERROR */ +enum { + DESC_ERR_NO_ERR = 0x0, + DESC_ERR_K_OUT_OF_RANGE = 0x1, + DESC_ERR_K_NOT_NORMAL = 0x2, + DESC_ERR_KPAI_NOT_NORMAL = 0x3, + DESC_ERR_DESC_OFFSET_ERR = 0x4, + DESC_ERR_DESC_READ_FAIL = 0x8, + DESC_ERR_DESC_READ_TIMEOUT = 0x9, + DESC_ERR_DESC_READ_TLP_POISONED = 0xA, + DESC_ERR_CB_READ_FAIL = 0xC, + DESC_ERR_CB_READ_TIMEOUT = 0xD, + DESC_ERR_CB_READ_TLP_POISONED = 0xE +}; + +/* FPGA LTE FEC DMA Encoding Request Descriptor */ +struct __attribute__((__packed__)) fpga_dma_enc_desc { + uint32_t done:1, + rsrvd0:11, + error:4, + rsrvd1:16; + uint32_t ncb:16, + rsrvd2:14, + rv:2; + uint32_t bypass_rm:1, + irq_en:1, + crc_en:1, + rsrvd3:13, + offset:10, + rsrvd4:6; + uint16_t e; + uint16_t k; + uint32_t out_addr_lw; + uint32_t out_addr_hi; + uint32_t in_addr_lw; + uint32_t in_addr_hi; + + union { + struct { + /* Virtual addresses used to retrieve SW context info */ + void *op_addr; + /* Stores information about total number of Code Blocks + * in currently processed Transport Block + */ + uint64_t cbs_in_op; + }; + + uint8_t sw_ctxt[FPGA_RING_DESC_LEN_UNIT_BYTES * + (FPGA_RING_DESC_ENTRY_LENGTH - 1)]; + }; +}; + +/* FPGA LTE FEC DMA Decoding Request Descriptor */ +struct __attribute__((__packed__)) fpga_dma_dec_desc { + uint32_t done:1, + iter:5, + rsrvd0:2, + crc_pass:1, + rsrvd1:3, + error:4, + crc_type:1, + rsrvd2:7, + max_iter:5, + rsrvd3:3; + uint32_t rsrvd4; + uint32_t bypass_rm:1, + irq_en:1, + drop_crc:1, + rsrvd5:13, + offset:10, + rsrvd6:6; + uint16_t k; + uint16_t in_len; + uint32_t out_addr_lw; + uint32_t out_addr_hi; + uint32_t in_addr_lw; + uint32_t in_addr_hi; + + union { + struct { + /* Virtual addresses used to retrieve SW context info */ + void *op_addr; + /* Stores information about total number of Code Blocks + * in currently processed Transport Block + */ + uint8_t cbs_in_op; + }; + + uint32_t sw_ctxt[8 * (FPGA_RING_DESC_ENTRY_LENGTH - 1)]; + }; +}; + +/* FPGA LTE DMA Descriptor */ +union fpga_dma_desc { + struct fpga_dma_enc_desc enc_req; + struct fpga_dma_dec_desc dec_req; +}; + +/* FPGA LTE FEC Ring Control Register */ +struct __attribute__((__packed__)) fpga_ring_ctrl_reg { + uint64_t ring_base_addr; + uint64_t ring_head_addr; + uint16_t ring_size:11; + uint16_t rsrvd0; + union { /* Miscellaneous register */ + uint8_t misc; + uint8_t max_ul_dec:5, + max_ul_dec_en:1, + rsrvd1:2; + }; + uint8_t enable; + uint8_t flush_queue_en; + uint8_t rsrvd2; + uint16_t shadow_tail; + uint16_t rsrvd3; + uint16_t head_point; + uint16_t rsrvd4; + +}; + +/* Private data structure for each FPGA FEC device */ +struct fpga_lte_fec_device { + /** Base address of MMIO registers (BAR0) */ + void *mmio_base; + /** Base address of memory for sw rings */ + void *sw_rings; + /** Physical address of sw_rings */ + rte_iova_t sw_rings_phys; + /** Number of bytes available for each queue in device. */ + uint32_t sw_ring_size; + /** Max number of entries available for each queue in device */ + uint32_t sw_ring_max_depth; + /** Base address of response tail pointer buffer */ + uint32_t *tail_ptrs; + /** Physical address of tail pointers */ + rte_iova_t tail_ptr_phys; + /** Queues flush completion flag */ + uint64_t *flush_queue_status; + /* Bitmap capturing which Queues are bound to the PF/VF */ + uint64_t q_bound_bit_map; + /* Bitmap capturing which Queues have already been assigned */ + uint64_t q_assigned_bit_map; + /** True if this is a PF FPGA FEC device */ + bool pf_device; +}; + +/* Structure associated with each queue. */ +struct __rte_cache_aligned fpga_queue { + struct fpga_ring_ctrl_reg ring_ctrl_reg; /* Ring Control Register */ + union fpga_dma_desc *ring_addr; /* Virtual address of software ring */ + uint64_t *ring_head_addr; /* Virtual address of completion_head */ + uint64_t shadow_completion_head; /* Shadow completion head value */ + uint16_t head_free_desc; /* Ring head */ + uint16_t tail; /* Ring tail */ + /* Mask used to wrap enqueued descriptors on the sw ring */ + uint32_t sw_ring_wrap_mask; + uint32_t irq_enable; /* Enable ops dequeue interrupts if set to 1 */ + uint8_t q_idx; /* Queue index */ + struct fpga_lte_fec_device *d; + /* MMIO register of shadow_tail used to enqueue descriptors */ + void *shadow_tail_addr; +}; + +/* Write to 16 bit MMIO register address */ +static inline void +mmio_write_16(void *addr, uint16_t value) +{ + *((volatile uint16_t *)(addr)) = rte_cpu_to_le_16(value); +} + +/* Write to 32 bit MMIO register address */ +static inline void +mmio_write_32(void *addr, uint32_t value) +{ + *((volatile uint32_t *)(addr)) = rte_cpu_to_le_32(value); +} + +/* Write to 64 bit MMIO register address */ +static inline void +mmio_write_64(void *addr, uint64_t value) +{ + *((volatile uint64_t *)(addr)) = rte_cpu_to_le_64(value); +} + +/* Write a 8 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_8(void *mmio_base, uint32_t offset, uint8_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + *((volatile uint8_t *)(reg_addr)) = payload; +} + +/* Write a 16 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_16(void *mmio_base, uint32_t offset, uint16_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_16(reg_addr, payload); +} + +/* Write a 32 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_32(void *mmio_base, uint32_t offset, uint32_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_32(reg_addr, payload); +} + +/* Write a 64 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_64(void *mmio_base, uint32_t offset, uint64_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_64(reg_addr, payload); +} + +/* Write a ring control register of a FPGA LTE FEC device */ +static inline void +fpga_ring_reg_write(void *mmio_base, uint32_t offset, + struct fpga_ring_ctrl_reg payload) +{ + fpga_reg_write_64(mmio_base, offset, payload.ring_base_addr); + fpga_reg_write_64(mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_ADDR, + payload.ring_head_addr); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_SIZE, + payload.ring_size); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_POINT, + payload.head_point); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN, + payload.flush_queue_en); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_SHADOW_TAIL, + payload.shadow_tail); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_MISC, + payload.misc); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + payload.enable); +} + +/* Read a register of FPGA LTE FEC device */ +static uint32_t +fpga_reg_read_32(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint32_t ret = *((volatile uint32_t *)(reg_addr)); + return rte_le_to_cpu_32(ret); +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Read a register of FPGA LTE FEC device */ +static uint8_t +fpga_reg_read_8(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + return *((volatile uint8_t *)(reg_addr)); +} + +/* Read a register of FPGA LTE FEC device */ +static uint16_t +fpga_reg_read_16(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint16_t ret = *((volatile uint16_t *)(reg_addr)); + return rte_le_to_cpu_16(ret); +} + +/* Read a register of FPGA LTE FEC device */ +static uint64_t +fpga_reg_read_64(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint64_t ret = *((volatile uint64_t *)(reg_addr)); + return rte_le_to_cpu_64(ret); +} + +/* Read Ring Control Register of FPGA LTE FEC device */ +static inline void +print_ring_reg_debug_info(void *mmio_base, uint32_t offset) +{ + rte_bbdev_log_debug( + "FPGA MMIO base address @ %p | Ring Control Register @ offset = 0x%08" + PRIx32, mmio_base, offset); + rte_bbdev_log_debug( + "RING_BASE_ADDR = 0x%016"PRIx64, + fpga_reg_read_64(mmio_base, offset)); + rte_bbdev_log_debug( + "RING_HEAD_ADDR = 0x%016"PRIx64, + fpga_reg_read_64(mmio_base, offset + + FPGA_LTE_FEC_RING_HEAD_ADDR)); + rte_bbdev_log_debug( + "RING_SIZE = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_SIZE)); + rte_bbdev_log_debug( + "RING_MISC = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_MISC)); + rte_bbdev_log_debug( + "RING_ENABLE = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_ENABLE)); + rte_bbdev_log_debug( + "RING_FLUSH_QUEUE_EN = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN)); + rte_bbdev_log_debug( + "RING_SHADOW_TAIL = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_SHADOW_TAIL)); + rte_bbdev_log_debug( + "RING_HEAD_POINT = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_HEAD_POINT)); +} + +/* Read Static Register of FPGA LTE FEC device */ +static inline void +print_static_reg_debug_info(void *mmio_base) +{ + uint16_t config = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_CONFIGURATION); + uint8_t qmap_done = fpga_reg_read_8(mmio_base, + FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE); + uint16_t lb_factor = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_LOAD_BALANCE_FACTOR); + uint16_t ring_desc_len = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_RING_DESC_LEN); + uint16_t flr_time_out = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_FLR_TIME_OUT); + + rte_bbdev_log_debug("UL.DL Weights = %u.%u", + ((uint8_t)config), ((uint8_t)(config >> 8))); + rte_bbdev_log_debug("UL.DL Load Balance = %u.%u", + ((uint8_t)lb_factor), ((uint8_t)(lb_factor >> 8))); + rte_bbdev_log_debug("Queue-PF/VF Mapping Table = %s", + (qmap_done > 0) ? "READY" : "NOT-READY"); + rte_bbdev_log_debug("Ring Descriptor Size = %u bytes", + ring_desc_len*FPGA_RING_DESC_LEN_UNIT_BYTES); + rte_bbdev_log_debug("FLR Timeout = %f usec", + (float)flr_time_out*FPGA_FLR_TIMEOUT_UNIT); +} + +/* Print decode DMA Descriptor of FPGA LTE FEC device */ +static void +print_dma_dec_desc_debug_info(union fpga_dma_desc *desc) +{ + rte_bbdev_log_debug("DMA response desc %p\n" + "\t-- done(%"PRIu32") | iter(%"PRIu32") | crc_pass(%"PRIu32")" + " | error (%"PRIu32") | crc_type(%"PRIu32")\n" + "\t-- max_iter(%"PRIu32") | bypass_rm(%"PRIu32") | " + "irq_en (%"PRIu32") | drop_crc(%"PRIu32") | offset(%"PRIu32")\n" + "\t-- k(%"PRIu32") | in_len (%"PRIu16") | op_add(%p)\n" + "\t-- cbs_in_op(%"PRIu32") | in_add (0x%08"PRIx32"%08"PRIx32") | " + "out_add (0x%08"PRIx32"%08"PRIx32")", + desc, + (uint32_t)desc->dec_req.done, + (uint32_t)desc->dec_req.iter, + (uint32_t)desc->dec_req.crc_pass, + (uint32_t)desc->dec_req.error, + (uint32_t)desc->dec_req.crc_type, + (uint32_t)desc->dec_req.max_iter, + (uint32_t)desc->dec_req.bypass_rm, + (uint32_t)desc->dec_req.irq_en, + (uint32_t)desc->dec_req.drop_crc, + (uint32_t)desc->dec_req.offset, + (uint32_t)desc->dec_req.k, + (uint16_t)desc->dec_req.in_len, + desc->dec_req.op_addr, + (uint32_t)desc->dec_req.cbs_in_op, + (uint32_t)desc->dec_req.in_addr_hi, + (uint32_t)desc->dec_req.in_addr_lw, + (uint32_t)desc->dec_req.out_addr_hi, + (uint32_t)desc->dec_req.out_addr_lw); +} +#endif + +static int +fpga_setup_queues(struct rte_bbdev *dev, uint16_t num_queues, int socket_id) +{ + /* Number of queues bound to a PF/VF */ + uint32_t hw_q_num = 0; + uint32_t ring_size, payload, address, q_id, offset; + rte_iova_t phys_addr; + struct fpga_ring_ctrl_reg ring_reg; + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + + address = FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE; + if (!(fpga_reg_read_32(fpga_dev->mmio_base, address) & 0x1)) { + rte_bbdev_log(ERR, + "Queue-PF/VF mapping is not set! Was PF configured for device (%s) ?", + dev->data->name); + return -EPERM; + } + + /* Clear queue registers structure */ + memset(&ring_reg, 0, sizeof(struct fpga_ring_ctrl_reg)); + + /* Scan queue map. + * If a queue is valid and mapped to a calling PF/VF the read value is + * replaced with a queue ID and if it's not then + * FPGA_INVALID_HW_QUEUE_ID is returned. + */ + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + uint32_t hw_q_id = fpga_reg_read_32(fpga_dev->mmio_base, + FPGA_LTE_FEC_QUEUE_MAP + (q_id << 2)); + + rte_bbdev_log_debug("%s: queue ID: %u, registry queue ID: %u", + dev->device->name, q_id, hw_q_id); + + if (hw_q_id != FPGA_INVALID_HW_QUEUE_ID) { + fpga_dev->q_bound_bit_map |= (1ULL << q_id); + /* Clear queue register of found queue */ + offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q_id); + fpga_ring_reg_write(fpga_dev->mmio_base, + offset, ring_reg); + ++hw_q_num; + } + } + if (hw_q_num == 0) { + rte_bbdev_log(ERR, + "No HW queues assigned to this device. Probably this is a VF configured for PF mode. Check device configuration!"); + return -ENODEV; + } + + if (num_queues > hw_q_num) { + rte_bbdev_log(ERR, + "Not enough queues for device %s! Requested: %u, available: %u", + dev->device->name, num_queues, hw_q_num); + return -EINVAL; + } + + ring_size = FPGA_RING_MAX_SIZE * sizeof(struct fpga_dma_dec_desc); + + /* Enforce 32 byte alignment */ + RTE_BUILD_BUG_ON((RTE_CACHE_LINE_SIZE % 32) != 0); + + /* Allocate memory for SW descriptor rings */ + fpga_dev->sw_rings = rte_zmalloc_socket(dev->device->driver->name, + num_queues * ring_size, RTE_CACHE_LINE_SIZE, + socket_id); + if (fpga_dev->sw_rings == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u sw_rings", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + + fpga_dev->sw_rings_phys = rte_malloc_virt2iova(fpga_dev->sw_rings); + fpga_dev->sw_ring_size = ring_size; + fpga_dev->sw_ring_max_depth = FPGA_RING_MAX_SIZE; + + /* Allocate memory for ring flush status */ + fpga_dev->flush_queue_status = rte_zmalloc_socket(NULL, + sizeof(uint64_t), RTE_CACHE_LINE_SIZE, socket_id); + if (fpga_dev->flush_queue_status == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u flush_queue_status", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + + /* Set the flush status address registers */ + phys_addr = rte_malloc_virt2iova(fpga_dev->flush_queue_status); + + address = FPGA_LTE_FEC_VFQ_FLUSH_STATUS_LW; + payload = (uint32_t)(phys_addr); + fpga_reg_write_32(fpga_dev->mmio_base, address, payload); + + address = FPGA_LTE_FEC_VFQ_FLUSH_STATUS_HI; + payload = (uint32_t)(phys_addr >> 32); + fpga_reg_write_32(fpga_dev->mmio_base, address, payload); + + return 0; +} + +static int +fpga_dev_close(struct rte_bbdev *dev) +{ + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + + rte_free(fpga_dev->sw_rings); + rte_free(fpga_dev->flush_queue_status); + + return 0; +} + +static void +fpga_dev_info_get(struct rte_bbdev *dev, + struct rte_bbdev_driver_info *dev_info) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + uint32_t q_id = 0; + + /* TODO RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN and numbers of buffers are set + * to temporary values as they are required by test application while + * validation phase. + */ + static const struct rte_bbdev_op_cap bbdev_capabilities[] = { + { + .type = RTE_BBDEV_OP_TURBO_DEC, + .cap.turbo_dec = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_TYPE_24B | + RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE | + RTE_BBDEV_TURBO_DEC_INTERRUPTS | + RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN | + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP, + .max_llr_modulus = INT8_MAX, + .num_buffers_src = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_hard_out = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_soft_out = 0 + } + }, + { + .type = RTE_BBDEV_OP_TURBO_ENC, + .cap.turbo_enc = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_24B_ATTACH | + RTE_BBDEV_TURBO_RATE_MATCH | + RTE_BBDEV_TURBO_ENC_INTERRUPTS, + .num_buffers_src = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_dst = + RTE_BBDEV_MAX_CODE_BLOCKS + } + }, + RTE_BBDEV_END_OF_CAPABILITIES_LIST() + }; + + static struct rte_bbdev_queue_conf default_queue_conf; + default_queue_conf.socket = dev->data->socket_id; + default_queue_conf.queue_size = FPGA_RING_MAX_SIZE; + + + dev_info->driver_name = dev->device->driver->name; + dev_info->queue_size_lim = FPGA_RING_MAX_SIZE; + dev_info->hardware_accelerated = true; + dev_info->min_alignment = 64; + dev_info->default_queue_conf = default_queue_conf; + dev_info->capabilities = bbdev_capabilities; + dev_info->cpu_flag_reqs = NULL; + + /* Calculates number of queues assigned to device */ + dev_info->max_num_queues = 0; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + uint32_t hw_q_id = fpga_reg_read_32(d->mmio_base, + FPGA_LTE_FEC_QUEUE_MAP + (q_id << 2)); + if (hw_q_id != FPGA_INVALID_HW_QUEUE_ID) + dev_info->max_num_queues++; + } +} + +/** + * Find index of queue bound to current PF/VF which is unassigned. Return -1 + * when there is no available queue + */ +static int +fpga_find_free_queue_idx(struct rte_bbdev *dev, + const struct rte_bbdev_queue_conf *conf) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + uint64_t q_idx; + uint8_t i = 0; + uint8_t range = FPGA_TOTAL_NUM_QUEUES >> 1; + + if (conf->op_type == RTE_BBDEV_OP_TURBO_ENC) { + i = FPGA_NUM_DL_QUEUES; + range = FPGA_TOTAL_NUM_QUEUES; + } + + for (; i < range; ++i) { + q_idx = 1ULL << i; + /* Check if index of queue is bound to current PF/VF */ + if (d->q_bound_bit_map & q_idx) + /* Check if found queue was not already assigned */ + if (!(d->q_assigned_bit_map & q_idx)) { + d->q_assigned_bit_map |= q_idx; + return i; + } + } + + rte_bbdev_log(INFO, "Failed to find free queue on %s", dev->data->name); + + return -1; +} + +static int +fpga_queue_setup(struct rte_bbdev *dev, uint16_t queue_id, + const struct rte_bbdev_queue_conf *conf) +{ + uint32_t address, ring_offset; + struct fpga_lte_fec_device *d = dev->data->dev_private; + struct fpga_queue *q; + int8_t q_idx; + + /* Check if there is a free queue to assign */ + q_idx = fpga_find_free_queue_idx(dev, conf); + if (q_idx == -1) + return -1; + + /* Allocate the queue data structure. */ + q = rte_zmalloc_socket(dev->device->driver->name, sizeof(*q), + RTE_CACHE_LINE_SIZE, conf->socket); + if (q == NULL) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_bbdev_log(ERR, "Failed to allocate queue memory"); + return -ENOMEM; + } + + q->d = d; + q->q_idx = q_idx; + + /* Set ring_base_addr */ + q->ring_addr = RTE_PTR_ADD(d->sw_rings, (d->sw_ring_size * queue_id)); + q->ring_ctrl_reg.ring_base_addr = d->sw_rings_phys + + (d->sw_ring_size * queue_id); + + /* Allocate memory for Completion Head variable*/ + q->ring_head_addr = rte_zmalloc_socket(dev->device->driver->name, + sizeof(uint64_t), RTE_CACHE_LINE_SIZE, conf->socket); + if (q->ring_head_addr == NULL) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_free(q); + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u completion_head", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + /* Set ring_head_addr */ + q->ring_ctrl_reg.ring_head_addr = + rte_malloc_virt2iova(q->ring_head_addr); + + /* Clear shadow_completion_head */ + q->shadow_completion_head = 0; + + /* Set ring_size */ + if (conf->queue_size > FPGA_RING_MAX_SIZE) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_free(q->ring_head_addr); + rte_free(q); + rte_bbdev_log(ERR, + "Size of queue is too big %d (MAX: %d ) for %s:%u", + conf->queue_size, FPGA_RING_MAX_SIZE, + dev->device->driver->name, dev->data->dev_id); + return -EINVAL; + } + q->ring_ctrl_reg.ring_size = conf->queue_size; + + /* Set Miscellaneous FPGA register*/ + /* Max iteration number for TTI mitigation - todo */ + q->ring_ctrl_reg.max_ul_dec = 0; + /* Enable max iteration number for TTI - todo */ + q->ring_ctrl_reg.max_ul_dec_en = 0; + + /* Enable the ring */ + q->ring_ctrl_reg.enable = 1; + + /* Set FPGA head_point and tail registers */ + q->ring_ctrl_reg.head_point = q->tail = 0; + + /* Set FPGA shadow_tail register */ + q->ring_ctrl_reg.shadow_tail = q->tail; + + /* Calculates the ring offset for found queue */ + ring_offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q_idx); + + /* Set FPGA Ring Control Registers */ + fpga_ring_reg_write(d->mmio_base, ring_offset, q->ring_ctrl_reg); + + /* Store MMIO register of shadow_tail */ + address = ring_offset + FPGA_LTE_FEC_RING_SHADOW_TAIL; + q->shadow_tail_addr = RTE_PTR_ADD(d->mmio_base, address); + + q->head_free_desc = q->tail; + + /* Set wrap mask */ + q->sw_ring_wrap_mask = conf->queue_size - 1; + + rte_bbdev_log_debug("Setup dev%u q%u: queue_idx=%u", + dev->data->dev_id, queue_id, q->q_idx); + + dev->data->queues[queue_id].queue_private = q; + + rte_bbdev_log_debug("BBDEV queue[%d] set up for FPGA queue[%d]", + queue_id, q_idx); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Read FPGA Ring Control Registers after configuration*/ + print_ring_reg_debug_info(d->mmio_base, ring_offset); +#endif + return 0; +} + +static int +fpga_queue_release(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + struct fpga_ring_ctrl_reg ring_reg; + uint32_t offset; + + rte_bbdev_log_debug("FPGA Queue[%d] released", queue_id); + + if (q != NULL) { + memset(&ring_reg, 0, sizeof(struct fpga_ring_ctrl_reg)); + offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + /* Disable queue */ + fpga_reg_write_8(d->mmio_base, + offset + FPGA_LTE_FEC_RING_ENABLE, 0x00); + /* Clear queue registers */ + fpga_ring_reg_write(d->mmio_base, offset, ring_reg); + + /* Mark the Queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q->q_idx)); + rte_free(q->ring_head_addr); + rte_free(q); + dev->data->queues[queue_id].queue_private = NULL; + } + + return 0; +} + +/* Function starts a device queue. */ +static int +fpga_queue_start(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (d == NULL) { + rte_bbdev_log(ERR, "Invalid device pointer"); + return -1; + } +#endif + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + uint32_t offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + uint8_t enable = 0x01; + uint16_t zero = 0x0000; + + /* Clear queue head and tail variables */ + q->tail = q->head_free_desc = 0; + + /* Clear FPGA head_point and tail registers */ + fpga_reg_write_16(d->mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_POINT, + zero); + fpga_reg_write_16(d->mmio_base, offset + FPGA_LTE_FEC_RING_SHADOW_TAIL, + zero); + + /* Enable queue */ + fpga_reg_write_8(d->mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + enable); + + rte_bbdev_log_debug("FPGA Queue[%d] started", queue_id); + return 0; +} + +/* Function stops a device queue. */ +static int +fpga_queue_stop(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (d == NULL) { + rte_bbdev_log(ERR, "Invalid device pointer"); + return -1; + } +#endif + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + uint32_t offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + uint8_t payload = 0x01; + uint8_t counter = 0; + uint8_t timeout = FPGA_QUEUE_FLUSH_TIMEOUT_US / + FPGA_TIMEOUT_CHECK_INTERVAL; + + /* Set flush_queue_en bit to trigger queue flushing */ + fpga_reg_write_8(d->mmio_base, + offset + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN, payload); + + /** Check if queue flush is completed. + * FPGA will update the completion flag after queue flushing is + * completed. If completion flag is not updated within 1ms it is + * considered as a failure. + */ + while (!(*((uint8_t *)d->flush_queue_status + q->q_idx) & payload)) { + if (counter > timeout) { + rte_bbdev_log(ERR, "FPGA Queue Flush failed for queue %d", + queue_id); + return -1; + } + usleep(FPGA_TIMEOUT_CHECK_INTERVAL); + counter++; + } + + /* Disable queue */ + payload = 0x00; + fpga_reg_write_8(d->mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + payload); + + rte_bbdev_log_debug("FPGA Queue[%d] stopped", queue_id); + return 0; +} + +static inline uint16_t +get_queue_id(struct rte_bbdev_data *data, uint8_t q_idx) +{ + uint16_t queue_id; + + for (queue_id = 0; queue_id < data->num_queues; ++queue_id) { + struct fpga_queue *q = data->queues[queue_id].queue_private; + if (q != NULL && q->q_idx == q_idx) + return queue_id; + } + + return -1; +} + +/* Interrupt handler triggered by FPGA dev for handling specific interrupt */ +static void +fpga_dev_interrupt_handler(void *cb_arg) +{ + struct rte_bbdev *dev = cb_arg; + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + struct fpga_queue *q; + uint64_t ring_head; + uint64_t q_idx; + uint16_t queue_id; + uint8_t i; + + /* Scan queue assigned to this device */ + for (i = 0; i < FPGA_TOTAL_NUM_QUEUES; ++i) { + q_idx = 1ULL << i; + if (fpga_dev->q_bound_bit_map & q_idx) { + queue_id = get_queue_id(dev->data, i); + if (queue_id == (uint16_t) -1) + continue; + + /* Check if completion head was changed */ + q = dev->data->queues[queue_id].queue_private; + ring_head = *q->ring_head_addr; + if (q->shadow_completion_head != ring_head && + q->irq_enable == 1) { + q->shadow_completion_head = ring_head; + rte_bbdev_pmd_callback_process( + dev, + RTE_BBDEV_EVENT_DEQUEUE, + &queue_id); + } + } + } +} + +static int +fpga_queue_intr_enable(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + + if (!rte_intr_cap_multiple(dev->intr_handle)) + return -ENOTSUP; + + q->irq_enable = 1; + + return 0; +} + +static int +fpga_queue_intr_disable(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + q->irq_enable = 0; + + return 0; +} + +static int +fpga_intr_enable(struct rte_bbdev *dev) +{ + int ret; + uint8_t i; + + if (!rte_intr_cap_multiple(dev->intr_handle)) { + rte_bbdev_log(ERR, "Multiple intr vector is not supported by FPGA (%s)", + dev->data->name); + return -ENOTSUP; + } + + /* Create event file descriptors for each of 64 queue. Event fds will be + * mapped to FPGA IRQs in rte_intr_enable(). This is a 1:1 mapping where + * the IRQ number is a direct translation to the queue number. + * + * 63 (FPGA_NUM_INTR_VEC) event fds are created as rte_intr_enable() + * mapped the first IRQ to already created interrupt event file + * descriptor (intr_handle->fd). + */ + if (rte_intr_efd_enable(dev->intr_handle, FPGA_NUM_INTR_VEC)) { + rte_bbdev_log(ERR, "Failed to create fds for %u queues", + dev->data->num_queues); + return -1; + } + + /* TODO Each event file descriptor is overwritten by interrupt event + * file descriptor. That descriptor is added to epoll observed list. + * It ensures that callback function assigned to that descriptor will + * invoked when any FPGA queue issues interrupt. + */ + for (i = 0; i < FPGA_NUM_INTR_VEC; ++i) + dev->intr_handle->efds[i] = dev->intr_handle->fd; + + if (!dev->intr_handle->intr_vec) { + dev->intr_handle->intr_vec = rte_zmalloc("intr_vec", + dev->data->num_queues * sizeof(int), 0); + if (!dev->intr_handle->intr_vec) { + rte_bbdev_log(ERR, "Failed to allocate %u vectors", + dev->data->num_queues); + return -ENOMEM; + } + } + + ret = rte_intr_enable(dev->intr_handle); + if (ret < 0) { + rte_bbdev_log(ERR, + "Couldn't enable interrupts for device: %s", + dev->data->name); + return ret; + } + + ret = rte_intr_callback_register(dev->intr_handle, + fpga_dev_interrupt_handler, dev); + if (ret < 0) { + rte_bbdev_log(ERR, + "Couldn't register interrupt callback for device: %s", + dev->data->name); + return ret; + } + + return 0; +} + +static const struct rte_bbdev_ops fpga_ops = { + .setup_queues = fpga_setup_queues, + .intr_enable = fpga_intr_enable, + .close = fpga_dev_close, + .info_get = fpga_dev_info_get, + .queue_setup = fpga_queue_setup, + .queue_stop = fpga_queue_stop, + .queue_start = fpga_queue_start, + .queue_release = fpga_queue_release, + .queue_intr_enable = fpga_queue_intr_enable, + .queue_intr_disable = fpga_queue_intr_disable +}; + +static inline void +fpga_dma_enqueue(struct fpga_queue *q, uint16_t num_desc, + struct rte_bbdev_stats *queue_stats) +{ +#ifdef RTE_BBDEV_OFFLOAD_COST + uint64_t start_time = 0; + queue_stats->acc_offload_cycles = 0; +#else + RTE_SET_USED(queue_stats); +#endif + + /* Update tail and shadow_tail register */ + q->tail = (q->tail + num_desc) & q->sw_ring_wrap_mask; + + rte_wmb(); + +#ifdef RTE_BBDEV_OFFLOAD_COST + /* Start time measurement for enqueue function offload. */ + start_time = rte_rdtsc_precise(); +#endif + mmio_write_16(q->shadow_tail_addr, q->tail); + +#ifdef RTE_BBDEV_OFFLOAD_COST + rte_wmb(); + queue_stats->acc_offload_cycles += rte_rdtsc_precise() - start_time; +#endif +} + +/* Calculates number of CBs in processed encoder TB based on 'r' and input + * length. + */ +static inline uint8_t +get_num_cbs_in_op_enc(struct rte_bbdev_op_turbo_enc *turbo_enc) +{ + uint8_t c, c_neg, r, crc24_bits = 0; + uint16_t k, k_neg, k_pos; + uint8_t cbs_in_op = 0; + int32_t length; + + length = turbo_enc->input.length; + r = turbo_enc->tb_params.r; + c = turbo_enc->tb_params.c; + c_neg = turbo_enc->tb_params.c_neg; + k_neg = turbo_enc->tb_params.k_neg; + k_pos = turbo_enc->tb_params.k_pos; + crc24_bits = 24; + while (length > 0 && r < c) { + k = (r < c_neg) ? k_neg : k_pos; + length -= (k - crc24_bits) >> 3; + r++; + cbs_in_op++; + } + + return cbs_in_op; +} + +/* Calculates number of CBs in processed decoder TB based on 'r' and input + * length. + */ +static inline uint16_t +get_num_cbs_in_op_dec(struct rte_bbdev_op_turbo_dec *turbo_dec) +{ + uint8_t c, c_neg, r = 0; + uint16_t kw, k, k_neg, k_pos, cbs_in_op = 0; + int32_t length; + + length = turbo_dec->input.length; + r = turbo_dec->tb_params.r; + c = turbo_dec->tb_params.c; + c_neg = turbo_dec->tb_params.c_neg; + k_neg = turbo_dec->tb_params.k_neg; + k_pos = turbo_dec->tb_params.k_pos; + while (length > 0 && r < c) { + k = (r < c_neg) ? k_neg : k_pos; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + length -= kw; + r++; + cbs_in_op++; + } + + return cbs_in_op; +} + +/* Read flag value 0/1/ from bitmap */ +static inline bool +check_bit(uint32_t bitmap, uint32_t bitmask) +{ + return bitmap & bitmask; +} + +/* Print an error if a descriptor error has occurred. + * Return 0 on success, 1 on failure + */ +static inline int +check_desc_error(uint32_t error_code) { + switch (error_code) { + case DESC_ERR_NO_ERR: + return 0; + case DESC_ERR_K_OUT_OF_RANGE: + rte_bbdev_log(ERR, "Block_size_k is out of range (k<40 or k>6144)"); + break; + case DESC_ERR_K_NOT_NORMAL: + rte_bbdev_log(ERR, "Block_size_k is not a normal value within normal range"); + break; + case DESC_ERR_KPAI_NOT_NORMAL: + rte_bbdev_log(ERR, "Three_kpai is not a normal value for UL only"); + break; + case DESC_ERR_DESC_OFFSET_ERR: + rte_bbdev_log(ERR, "Queue offset does not meet the expectation in the FPGA"); + break; + case (DESC_ERR_K_OUT_OF_RANGE | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Block_size_k is out of range (k<40 or k>6144) and queue offset error"); + break; + case (DESC_ERR_K_NOT_NORMAL | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Block_size_k is not a normal value within normal range and queue offset error"); + break; + case (DESC_ERR_KPAI_NOT_NORMAL | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Three_kpai is not a normal value for UL only and queue offset error"); + break; + case DESC_ERR_DESC_READ_FAIL: + rte_bbdev_log(ERR, "Unsuccessful completion for descriptor read"); + break; + case DESC_ERR_DESC_READ_TIMEOUT: + rte_bbdev_log(ERR, "Descriptor read time-out"); + break; + case DESC_ERR_DESC_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Descriptor read TLP poisoned"); + break; + case DESC_ERR_CB_READ_FAIL: + rte_bbdev_log(ERR, "Unsuccessful completion for code block"); + break; + case DESC_ERR_CB_READ_TIMEOUT: + rte_bbdev_log(ERR, "Code block read time-out"); + break; + case DESC_ERR_CB_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Code block read TLP poisoned"); + break; + default: + rte_bbdev_log(ERR, "Descriptor error unknown error code %u", + error_code); + break; + } + return 1; +} + +/** + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param k + * K value (length of input in bits). + * @param e + * E value (length of output in bits). + * @param ncb + * Ncb value (size of the soft buffer). + * @param out_length + * Length of output buffer + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +fpga_dma_desc_te_fill(struct rte_bbdev_enc_op *op, + struct fpga_dma_enc_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t k, uint16_t e, uint16_t ncb, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) + +{ + /* reset */ + desc->done = 0; + desc->crc_en = check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_CRC_24B_ATTACH); + desc->bypass_rm = !check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_RATE_MATCH); + desc->k = k; + desc->e = e; + desc->ncb = ncb; + desc->rv = op->turbo_enc.rv_index; + desc->offset = desc_offset; + /* Set inbound data buffer address */ + desc->in_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset) >> 32); + desc->in_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset)); + + desc->out_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset) >> 32); + desc->out_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset)); + + /* Save software context needed for dequeue */ + desc->op_addr = op; + + /* Set total number of CBs in an op */ + desc->cbs_in_op = cbs_in_op; + + return 0; +} + +/** + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param in_length + * Length of an input. + * @param k + * K value (length of an output in bits). + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +fpga_dma_desc_td_fill(struct rte_bbdev_dec_op *op, + struct fpga_dma_dec_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t in_length, uint16_t k, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) +{ + /* reset */ + desc->done = 0; + /* Set inbound data buffer address */ + desc->in_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset) >> 32); + desc->in_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset)); + desc->in_len = in_length; + desc->k = k; + desc->crc_type = !check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_CRC_TYPE_24B); + if ((op->turbo_dec.code_block_mode == 0) + && !check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP)) + desc->drop_crc = 1; + desc->max_iter = op->turbo_dec.iter_max * 2; + desc->offset = desc_offset; + desc->out_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset) >> 32); + desc->out_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset)); + + /* Save software context needed for dequeue */ + desc->op_addr = op; + + /* Set total number of CBs in an op */ + desc->cbs_in_op = cbs_in_op; + + return 0; +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Validates turbo encoder parameters */ +static int +validate_enc_op(struct rte_bbdev_enc_op *op) +{ + struct rte_bbdev_op_turbo_enc *turbo_enc = &op->turbo_enc; + struct rte_bbdev_op_enc_turbo_cb_params *cb = NULL; + struct rte_bbdev_op_enc_turbo_tb_params *tb = NULL; + uint16_t kw, kw_neg, kw_pos; + + if (turbo_enc->input.length > + RTE_BBDEV_MAX_TB_SIZE >> 3) { + rte_bbdev_log(ERR, "TB size (%u) is too big, max: %d", + turbo_enc->input.length, RTE_BBDEV_MAX_TB_SIZE); + op->status = 1 << RTE_BBDEV_DATA_ERROR; + return -1; + } + + if (op->mempool == NULL) { + rte_bbdev_log(ERR, "Invalid mempool pointer"); + return -1; + } + if (turbo_enc->input.data == NULL) { + rte_bbdev_log(ERR, "Invalid input pointer"); + return -1; + } + if (turbo_enc->output.data == NULL) { + rte_bbdev_log(ERR, "Invalid output pointer"); + return -1; + } + if (turbo_enc->rv_index > 3) { + rte_bbdev_log(ERR, + "rv_index (%u) is out of range 0 <= value <= 3", + turbo_enc->rv_index); + return -1; + } + if (turbo_enc->code_block_mode != 0 && + turbo_enc->code_block_mode != 1) { + rte_bbdev_log(ERR, + "code_block_mode (%u) is out of range 0 <= value <= 1", + turbo_enc->code_block_mode); + return -1; + } + + if (turbo_enc->code_block_mode == 0) { + tb = &turbo_enc->tb_params; + if ((tb->k_neg < RTE_BBDEV_MIN_CB_SIZE + || tb->k_neg > RTE_BBDEV_MAX_CB_SIZE) + && tb->c_neg > 0) { + rte_bbdev_log(ERR, + "k_neg (%u) is out of range %u <= value <= %u", + tb->k_neg, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->k_pos < RTE_BBDEV_MIN_CB_SIZE + || tb->k_pos > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k_pos (%u) is out of range %u <= value <= %u", + tb->k_pos, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->c_neg > (RTE_BBDEV_MAX_CODE_BLOCKS - 1)) + rte_bbdev_log(ERR, + "c_neg (%u) is out of range 0 <= value <= %u", + tb->c_neg, + RTE_BBDEV_MAX_CODE_BLOCKS - 1); + if (tb->c < 1 || tb->c > RTE_BBDEV_MAX_CODE_BLOCKS) { + rte_bbdev_log(ERR, + "c (%u) is out of range 1 <= value <= %u", + tb->c, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + if (tb->cab > tb->c) { + rte_bbdev_log(ERR, + "cab (%u) is greater than c (%u)", + tb->cab, tb->c); + return -1; + } + if ((tb->ea < RTE_BBDEV_MIN_CB_SIZE || (tb->ea % 2)) + && tb->r < tb->cab) { + rte_bbdev_log(ERR, + "ea (%u) is less than %u or it is not even", + tb->ea, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + if ((tb->eb < RTE_BBDEV_MIN_CB_SIZE || (tb->eb % 2)) + && tb->c > tb->cab) { + rte_bbdev_log(ERR, + "eb (%u) is less than %u or it is not even", + tb->eb, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + + kw_neg = 3 * RTE_ALIGN_CEIL(tb->k_neg + 4, + RTE_BBDEV_TURBO_C_SUBBLOCK); + if (tb->ncb_neg < tb->k_neg || tb->ncb_neg > kw_neg) { + rte_bbdev_log(ERR, + "ncb_neg (%u) is out of range (%u) k_neg <= value <= (%u) kw_neg", + tb->ncb_neg, tb->k_neg, kw_neg); + return -1; + } + + kw_pos = 3 * RTE_ALIGN_CEIL(tb->k_pos + 4, + RTE_BBDEV_TURBO_C_SUBBLOCK); + if (tb->ncb_pos < tb->k_pos || tb->ncb_pos > kw_pos) { + rte_bbdev_log(ERR, + "ncb_pos (%u) is out of range (%u) k_pos <= value <= (%u) kw_pos", + tb->ncb_pos, tb->k_pos, kw_pos); + return -1; + } + if (tb->r > (tb->c - 1)) { + rte_bbdev_log(ERR, + "r (%u) is greater than c - 1 (%u)", + tb->r, tb->c - 1); + return -1; + } + } else { + cb = &turbo_enc->cb_params; + if (cb->k < RTE_BBDEV_MIN_CB_SIZE + || cb->k > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k (%u) is out of range %u <= value <= %u", + cb->k, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + + if (cb->e < RTE_BBDEV_MIN_CB_SIZE || (cb->e % 2)) { + rte_bbdev_log(ERR, + "e (%u) is less than %u or it is not even", + cb->e, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + + kw = RTE_ALIGN_CEIL(cb->k + 4, RTE_BBDEV_TURBO_C_SUBBLOCK) * 3; + if (cb->ncb < cb->k || cb->ncb > kw) { + rte_bbdev_log(ERR, + "ncb (%u) is out of range (%u) k <= value <= (%u) kw", + cb->ncb, cb->k, kw); + return -1; + } + } + + return 0; +} +#endif + +static inline char * +mbuf_append(struct rte_mbuf *m_head, struct rte_mbuf *m, uint16_t len) +{ + if (unlikely(len > rte_pktmbuf_tailroom(m))) + return NULL; + + char *tail = (char *)m->buf_addr + m->data_off + m->data_len; + m->data_len = (uint16_t)(m->data_len + len); + m_head->pkt_len = (m_head->pkt_len + len); + return tail; +} + +static inline int +enqueue_enc_one_op_cb(struct fpga_queue *q, struct rte_bbdev_enc_op *op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input; + struct rte_mbuf *output; + int ret; + uint16_t k, e, ncb, ring_offset; + uint32_t total_left, in_length, out_length, in_offset, out_offset; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_enc_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo encoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_enc.input.data; + output = op->turbo_enc.output.data; + in_offset = op->turbo_enc.input.offset; + out_offset = op->turbo_enc.output.offset; + total_left = op->turbo_enc.input.length; + k = op->turbo_enc.cb_params.k; + e = op->turbo_enc.cb_params.e; + ncb = op->turbo_enc.cb_params.ncb; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_CRC_24B_ATTACH)) + in_length = ((k - 24) >> 3); + else + in_length = k >> 3; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_RATE_MATCH)) + out_length = (e + 7) >> 3; + else + out_length = (k >> 3) * 3 + 2; + + mbuf_append(output, output, out_length); + + /* Offset into the ring */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + /* Setup DMA Descriptor */ + desc = q->ring_addr + ring_offset; + + ret = fpga_dma_desc_te_fill(op, &desc->enc_req, input, output, k, e, + ncb, in_offset, out_offset, ring_offset, 1); + if (unlikely(ret < 0)) + return ret; + + /* Update lengths */ + total_left -= in_length; + op->turbo_enc.output.length += out_length; + + if (total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + total_left, in_length); + return -1; + } + + return 1; +} + +static inline int +enqueue_enc_one_op_tb(struct fpga_queue *q, struct rte_bbdev_enc_op *op, + uint16_t desc_offset, uint8_t cbs_in_op) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input, *output_head, *output; + int ret; + uint8_t r, c, crc24_bits = 0; + uint16_t k, e, ncb, ring_offset; + uint32_t mbuf_total_left, in_length, out_length, in_offset, out_offset; + uint32_t seg_total_left; + uint16_t current_enqueued_cbs = 0; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_enc_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo encoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_enc.input.data; + output_head = output = op->turbo_enc.output.data; + in_offset = op->turbo_enc.input.offset; + out_offset = op->turbo_enc.output.offset; + mbuf_total_left = op->turbo_enc.input.length; + + c = op->turbo_enc.tb_params.c; + r = op->turbo_enc.tb_params.r; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_CRC_24B_ATTACH)) + crc24_bits = 24; + + while (mbuf_total_left > 0 && r < c && input != NULL) { + seg_total_left = rte_pktmbuf_data_len(input) - in_offset; + + e = (r < op->turbo_enc.tb_params.cab) ? + op->turbo_enc.tb_params.ea : + op->turbo_enc.tb_params.eb; + k = (r < op->turbo_enc.tb_params.c_neg) ? + op->turbo_enc.tb_params.k_neg : + op->turbo_enc.tb_params.k_pos; + ncb = (r < op->turbo_enc.tb_params.c_neg) ? + op->turbo_enc.tb_params.ncb_neg : + op->turbo_enc.tb_params.ncb_pos; + + in_length = ((k - crc24_bits) >> 3); + + if (check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_RATE_MATCH)) + out_length = (e + 7) >> 3; + else + out_length = (k >> 3) * 3 + 2; + + mbuf_append(output_head, output, out_length); + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_te_fill(op, &desc->enc_req, input, output, + k, e, ncb, in_offset, out_offset, ring_offset, + cbs_in_op); + if (unlikely(ret < 0)) + return ret; + + rte_bbdev_log_debug("DMA request desc %p", desc); + + /* Update lengths */ + op->turbo_enc.output.length += out_length; + mbuf_total_left -= in_length; + + /* Update offsets */ + if (seg_total_left == in_length) { + /* Go to the next mbuf */ + input = input->next; + output = output->next; + in_offset = 0; + out_offset = 0; + } else { + in_offset += in_length; + out_offset += out_length; + } + + r++; + desc_offset++; + current_enqueued_cbs++; + } + + if (mbuf_total_left > 0) { + rte_bbdev_log(ERR, + "Some date still left for processing: mbuf_total_left = %u", + mbuf_total_left); + return -1; + } + + return current_enqueued_cbs; +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Validates turbo decoder parameters */ +static int +validate_dec_op(struct rte_bbdev_dec_op *op) +{ + struct rte_bbdev_op_turbo_dec *turbo_dec = &op->turbo_dec; + struct rte_bbdev_op_dec_turbo_cb_params *cb = NULL; + struct rte_bbdev_op_dec_turbo_tb_params *tb = NULL; + + if (op->mempool == NULL) { + rte_bbdev_log(ERR, "Invalid mempool pointer"); + return -1; + } + if (turbo_dec->input.data == NULL) { + rte_bbdev_log(ERR, "Invalid input pointer"); + return -1; + } + if (turbo_dec->hard_output.data == NULL) { + rte_bbdev_log(ERR, "Invalid hard_output pointer"); + return -1; + } + if (turbo_dec->rv_index > 3) { + rte_bbdev_log(ERR, + "rv_index (%u) is out of range 0 <= value <= 3", + turbo_dec->rv_index); + return -1; + } + if (turbo_dec->iter_min < 1) { + rte_bbdev_log(ERR, + "iter_min (%u) is less than 1", + turbo_dec->iter_min); + return -1; + } + if (turbo_dec->iter_max <= 2) { + rte_bbdev_log(ERR, + "iter_max (%u) is less than or equal to 2", + turbo_dec->iter_max); + return -1; + } + if (turbo_dec->iter_min > turbo_dec->iter_max) { + rte_bbdev_log(ERR, + "iter_min (%u) is greater than iter_max (%u)", + turbo_dec->iter_min, turbo_dec->iter_max); + return -1; + } + if (turbo_dec->code_block_mode != 0 && + turbo_dec->code_block_mode != 1) { + rte_bbdev_log(ERR, + "code_block_mode (%u) is out of range 0 <= value <= 1", + turbo_dec->code_block_mode); + return -1; + } + + if (turbo_dec->code_block_mode == 0) { + + if ((turbo_dec->op_flags & + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP) && + !(turbo_dec->op_flags & RTE_BBDEV_TURBO_CRC_TYPE_24B)) { + rte_bbdev_log(ERR, + "RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP should accompany RTE_BBDEV_TURBO_CRC_TYPE_24B"); + return -1; + } + + tb = &turbo_dec->tb_params; + if ((tb->k_neg < RTE_BBDEV_MIN_CB_SIZE + || tb->k_neg > RTE_BBDEV_MAX_CB_SIZE) + && tb->c_neg > 0) { + rte_bbdev_log(ERR, + "k_neg (%u) is out of range %u <= value <= %u", + tb->k_neg, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if ((tb->k_pos < RTE_BBDEV_MIN_CB_SIZE + || tb->k_pos > RTE_BBDEV_MAX_CB_SIZE) + && tb->c > tb->c_neg) { + rte_bbdev_log(ERR, + "k_pos (%u) is out of range %u <= value <= %u", + tb->k_pos, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->c_neg > (RTE_BBDEV_MAX_CODE_BLOCKS - 1)) + rte_bbdev_log(ERR, + "c_neg (%u) is out of range 0 <= value <= %u", + tb->c_neg, + RTE_BBDEV_MAX_CODE_BLOCKS - 1); + if (tb->c < 1 || tb->c > RTE_BBDEV_MAX_CODE_BLOCKS) { + rte_bbdev_log(ERR, + "c (%u) is out of range 1 <= value <= %u", + tb->c, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + if (tb->cab > tb->c) { + rte_bbdev_log(ERR, + "cab (%u) is greater than c (%u)", + tb->cab, tb->c); + return -1; + } + } else { + + if (turbo_dec->op_flags & RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP) { + rte_bbdev_log(ERR, + "RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP is invalid in CB-mode"); + return -1; + } + + cb = &turbo_dec->cb_params; + if (cb->k < RTE_BBDEV_MIN_CB_SIZE + || cb->k > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k (%u) is out of range %u <= value <= %u", + cb->k, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + } + + return 0; +} +#endif + +static inline int +enqueue_dec_one_op_cb(struct fpga_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input; + struct rte_mbuf *output; + int ret; + uint16_t k, kw, ring_offset; + uint32_t total_left, in_length, out_length, in_offset, out_offset; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_dec_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo decoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_dec.input.data; + output = op->turbo_dec.hard_output.data; + total_left = op->turbo_dec.input.length; + in_offset = op->turbo_dec.input.offset; + out_offset = op->turbo_dec.hard_output.offset; + + k = op->turbo_dec.cb_params.k; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + in_length = kw; + out_length = k >> 3; + + mbuf_append(output, output, out_length); + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_td_fill(op, &desc->dec_req, input, output, + in_length, k, in_offset, out_offset, ring_offset, 1); + if (unlikely(ret < 0)) + return ret; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_dma_dec_desc_debug_info(desc); +#endif + + /* Update lengths */ + total_left -= in_length; + op->turbo_dec.hard_output.length += out_length; + + if (total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + total_left, in_length); + return -1; + } + + return 1; +} + + +static inline int +enqueue_dec_one_op_tb(struct fpga_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset, uint8_t cbs_in_op) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input, *output_head, *output; + int ret; + uint8_t r, c; + uint16_t k, kw, in_length, out_length, ring_offset; + uint32_t mbuf_total_left, seg_total_left, in_offset, out_offset; + uint16_t current_enqueued_cbs = 0; + uint16_t crc24_overlap = 0; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_dec_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo decoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_dec.input.data; + output_head = output = op->turbo_dec.hard_output.data; + mbuf_total_left = op->turbo_dec.input.length; + in_offset = op->turbo_dec.input.offset; + out_offset = op->turbo_dec.hard_output.offset; + + if (!check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP)) + crc24_overlap = 24; + + c = op->turbo_dec.tb_params.c; + r = op->turbo_dec.tb_params.r; + + while (mbuf_total_left > 0 && r < c && input != NULL) { + seg_total_left = rte_pktmbuf_data_len(input) - in_offset; + k = (r < op->turbo_dec.tb_params.c_neg) ? + op->turbo_dec.tb_params.k_neg : + op->turbo_dec.tb_params.k_pos; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + + in_length = kw; + out_length = (k - crc24_overlap) >> 3; + + mbuf_append(output_head, output, out_length); + + if (seg_total_left < in_length) { + rte_bbdev_log(ERR, + "Partial CB found in a TB. FPGA Driver doesn't support scatter-gather operations!"); + return -1; + } + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_td_fill(op, &desc->dec_req, input, output, + in_length, k, in_offset, out_offset, + ring_offset, cbs_in_op); + if (unlikely(ret < 0)) + return ret; + + /* Update lengths */ + ret = rte_pktmbuf_trim(op->turbo_dec.hard_output.data, + (crc24_overlap >> 3)); +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (ret < 0) { + rte_bbdev_log(ERR, + "The length to remove is greater than the length of the last segment"); + return -EINVAL; + } +#endif + op->turbo_dec.hard_output.length += out_length; + mbuf_total_left -= in_length; + + /* Update offsets */ + if (seg_total_left == in_length) { + /* Go to the next mbuf */ + input = input->next; + output = output->next; + in_offset = 0; + out_offset = 0; + } else { + in_offset += in_length; + out_offset += out_length; + } + + r++; + desc_offset++; + current_enqueued_cbs++; + } + + if (mbuf_total_left > 0) { + rte_bbdev_log(ERR, + "Some date still left for processing: mbuf_total_left = %u", + mbuf_total_left); + return -1; + } + + return current_enqueued_cbs; +} + +static uint16_t +fpga_enqueue_enc(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num) +{ + uint8_t cbs_in_op; + uint16_t i, total_enqueued_cbs = 0; + int32_t avail; + int enqueued_cbs; + struct fpga_queue *q = q_data->queue_private; + union fpga_dma_desc *desc; + + /* Check if queue is not full */ + if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == + q->head_free_desc)) + return 0; + + /* Calculates available space */ + avail = (q->head_free_desc > q->tail) ? + q->head_free_desc - q->tail - 1 : + q->ring_ctrl_reg.ring_size + q->head_free_desc - q->tail - 1; + + for (i = 0; i < num; ++i) { + if (ops[i]->turbo_enc.code_block_mode == 0) { + cbs_in_op = get_num_cbs_in_op_enc(&ops[i]->turbo_enc); + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - cbs_in_op < 0)) + break; + avail -= cbs_in_op; + enqueued_cbs = enqueue_enc_one_op_tb(q, ops[i], + total_enqueued_cbs, cbs_in_op); + } else { + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - 1 < 0)) + break; + avail -= 1; + enqueued_cbs = enqueue_enc_one_op_cb(q, ops[i], + total_enqueued_cbs); + } + + if (enqueued_cbs < 0) + break; + + total_enqueued_cbs += enqueued_cbs; + + rte_bbdev_log_debug("enqueuing enc ops [%d/%d] | head %d | tail %d", + total_enqueued_cbs, num, + q->head_free_desc, q->tail); + } + + /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt + * only when all previous CBs were already processed. + */ + desc = q->ring_addr + ((q->tail + total_enqueued_cbs - 1) + & q->sw_ring_wrap_mask); + desc->enc_req.irq_en = q->irq_enable; + + fpga_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); + + /* Update stats */ + q_data->queue_stats.enqueued_count += i; + q_data->queue_stats.enqueue_err_count += num - i; + + return i; +} + +static uint16_t +fpga_enqueue_dec(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num) +{ + uint8_t cbs_in_op; + uint16_t i, total_enqueued_cbs = 0; + int32_t avail; + int enqueued_cbs; + struct fpga_queue *q = q_data->queue_private; + union fpga_dma_desc *desc; + + /* Check if queue is not full */ + if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == + q->head_free_desc)) + return 0; + + /* Calculates available space */ + avail = (q->head_free_desc > q->tail) ? + q->head_free_desc - q->tail - 1 : + q->ring_ctrl_reg.ring_size + q->head_free_desc - q->tail - 1; + + for (i = 0; i < num; ++i) { + if (ops[i]->turbo_dec.code_block_mode == 0) { + cbs_in_op = get_num_cbs_in_op_dec(&ops[i]->turbo_dec); + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - cbs_in_op < 0)) + break; + avail -= cbs_in_op; + enqueued_cbs = enqueue_dec_one_op_tb(q, ops[i], + total_enqueued_cbs, cbs_in_op); + } else { + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - 1 < 0)) + break; + avail -= 1; + enqueued_cbs = enqueue_dec_one_op_cb(q, ops[i], + total_enqueued_cbs); + } + + if (enqueued_cbs < 0) + break; + + total_enqueued_cbs += enqueued_cbs; + + rte_bbdev_log_debug("enqueuing dec ops [%d/%d] | head %d | tail %d", + total_enqueued_cbs, num, + q->head_free_desc, q->tail); + } + + /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt + * only when all previous CBs were already processed. + */ + desc = q->ring_addr + ((q->tail + total_enqueued_cbs - 1) + & q->sw_ring_wrap_mask); + desc->dec_req.irq_en = q->irq_enable; + + fpga_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); + + /* Update stats */ + q_data->queue_stats.enqueued_count += i; + q_data->queue_stats.enqueue_err_count += num - i; + + return i; +} + +static inline int +dequeue_enc_one_op_cb(struct fpga_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + int desc_error = 0; + + /* Set current desc */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /*check if done */ + if (desc->enc_req.done == 0) + return -1; + + /* make sure the response is read atomically */ + rte_smp_rmb(); + + rte_bbdev_log_debug("DMA response desc %p", desc); + + *op = desc->enc_req.op_addr; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + (*op)->status = desc_error << RTE_BBDEV_DATA_ERROR; + + return 1; +} + +static inline int +dequeue_enc_one_op_tb(struct fpga_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + uint8_t cbs_in_op, cb_idx; + int desc_error = 0; + int status = 0; + + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify if done bit is set */ + if (desc->enc_req.done == 0) + return -1; + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + /* Verify if done bit in all CBs is set */ + cbs_in_op = desc->enc_req.cbs_in_op; + for (cb_idx = 1; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + if (desc->enc_req.done == 0) + return -1; + } + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + for (cb_idx = 0; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + status |= desc_error << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log_debug("DMA response desc %p", desc); + } + + *op = desc->enc_req.op_addr; + (*op)->status = status; + return cbs_in_op; +} + +static inline int +dequeue_dec_one_op_cb(struct fpga_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + int desc_error = 0; + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify done bit is set */ + if (desc->dec_req.done == 0) + return -1; + + /* make sure the response is read atomically */ + rte_smp_rmb(); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_dma_dec_desc_debug_info(desc); + +#endif + + *op = desc->dec_req.op_addr; + /* FPGA reports in half-iterations, from 0 to 31. get ceiling */ + (*op)->turbo_dec.iter_count = (desc->dec_req.iter + 2) >> 1; + /* crc_pass = 0 when decoder fails */ + (*op)->status = !(desc->dec_req.crc_pass) << RTE_BBDEV_CRC_ERROR; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + (*op)->status |= desc_error << RTE_BBDEV_DATA_ERROR; + return 1; +} + +static inline int +dequeue_dec_one_op_tb(struct fpga_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + uint8_t cbs_in_op, cb_idx, iter_count = 0; + int status = 0; + int desc_error = 0; + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify if done bit is set */ + if (desc->dec_req.done == 0) + return -1; + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + /* Verify if done bit in all CBs is set */ + cbs_in_op = desc->dec_req.cbs_in_op; + for (cb_idx = 1; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + if (desc->dec_req.done == 0) + return -1; + } + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + for (cb_idx = 0; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + /* get max iter_count for all CBs in op */ + iter_count = RTE_MAX(iter_count, (uint8_t) desc->dec_req.iter); + /* crc_pass = 0 when decoder fails, one fails all */ + status |= !(desc->dec_req.crc_pass) << RTE_BBDEV_CRC_ERROR; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + status |= desc_error << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log_debug("DMA response desc %p", desc); + } + + *op = desc->dec_req.op_addr; + + /* FPGA reports in half-iterations, get ceiling */ + (*op)->turbo_dec.iter_count = (iter_count + 2) >> 1; + (*op)->status = status; + return cbs_in_op; +} + +static uint16_t +fpga_dequeue_enc(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num) +{ + struct fpga_queue *q = q_data->queue_private; + uint32_t avail = (q->tail - q->head_free_desc) & q->sw_ring_wrap_mask; + uint16_t i; + uint16_t dequeued_cbs = 0; + struct rte_bbdev_enc_op *op; + int ret; + + for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { + op = (q->ring_addr + ((q->head_free_desc + dequeued_cbs) + & q->sw_ring_wrap_mask))->enc_req.op_addr; + if (op->turbo_enc.code_block_mode == 0) + ret = dequeue_enc_one_op_tb(q, &ops[i], dequeued_cbs); + else + ret = dequeue_enc_one_op_cb(q, &ops[i], dequeued_cbs); + + if (ret < 0) + break; + + dequeued_cbs += ret; + + rte_bbdev_log_debug("dequeuing enc ops [%d/%d] | head %d | tail %d", + dequeued_cbs, num, q->head_free_desc, q->tail); + } + + /* Update head */ + q->head_free_desc = (q->head_free_desc + dequeued_cbs) & + q->sw_ring_wrap_mask; + + /* Update stats */ + q_data->queue_stats.dequeued_count += i; + + return i; +} + +static uint16_t +fpga_dequeue_dec(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num) +{ + struct fpga_queue *q = q_data->queue_private; + uint32_t avail = (q->tail - q->head_free_desc) & q->sw_ring_wrap_mask; + uint16_t i; + uint16_t dequeued_cbs = 0; + struct rte_bbdev_dec_op *op; + int ret; + + for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { + op = (q->ring_addr + ((q->head_free_desc + dequeued_cbs) + & q->sw_ring_wrap_mask))->dec_req.op_addr; + if (op->turbo_dec.code_block_mode == 0) + ret = dequeue_dec_one_op_tb(q, &ops[i], dequeued_cbs); + else + ret = dequeue_dec_one_op_cb(q, &ops[i], dequeued_cbs); + + if (ret < 0) + break; + + dequeued_cbs += ret; + + rte_bbdev_log_debug("dequeuing dec ops [%d/%d] | head %d | tail %d", + dequeued_cbs, num, q->head_free_desc, q->tail); + } + + /* Update head */ + q->head_free_desc = (q->head_free_desc + dequeued_cbs) & + q->sw_ring_wrap_mask; + + /* Update stats */ + q_data->queue_stats.dequeued_count += i; + + return i; +} + +/* Initialization Function */ +static void +fpga_lte_fec_init(struct rte_bbdev *dev) +{ + struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev->device); + + dev->dev_ops = &fpga_ops; + dev->enqueue_enc_ops = fpga_enqueue_enc; + dev->enqueue_dec_ops = fpga_enqueue_dec; + dev->dequeue_enc_ops = fpga_dequeue_enc; + dev->dequeue_dec_ops = fpga_dequeue_dec; + + ((struct fpga_lte_fec_device *) dev->data->dev_private)->pf_device = + !strcmp(dev->device->driver->name, + RTE_STR(FPGA_LTE_FEC_PF_DRIVER_NAME)); + ((struct fpga_lte_fec_device *) dev->data->dev_private)->mmio_base = + pci_dev->mem_resource[0].addr; + + rte_bbdev_log_debug( + "Init device %s [%s] @ virtaddr %p phyaddr %#"PRIx64, + dev->device->driver->name, dev->data->name, + (void *)pci_dev->mem_resource[0].addr, + pci_dev->mem_resource[0].phys_addr); +} + +static int +fpga_lte_fec_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct rte_bbdev *bbdev = NULL; + char dev_name[RTE_BBDEV_NAME_MAX_LEN]; + + if (pci_dev == NULL) { + rte_bbdev_log(ERR, "NULL PCI device"); + return -EINVAL; + } + + rte_pci_device_name(&pci_dev->addr, dev_name, sizeof(dev_name)); + + /* Allocate memory to be used privately by drivers */ + bbdev = rte_bbdev_allocate(pci_dev->device.name); + if (bbdev == NULL) + return -ENODEV; + + /* allocate device private memory */ + bbdev->data->dev_private = rte_zmalloc_socket(dev_name, + sizeof(struct fpga_lte_fec_device), RTE_CACHE_LINE_SIZE, + pci_dev->device.numa_node); + + if (bbdev->data->dev_private == NULL) { + rte_bbdev_log(CRIT, + "Allocate of %zu bytes for device \"%s\" failed", + sizeof(struct fpga_lte_fec_device), dev_name); + rte_bbdev_release(bbdev); + return -ENOMEM; + } + + /* Fill HW specific part of device structure */ + bbdev->device = &pci_dev->device; + bbdev->intr_handle = &pci_dev->intr_handle; + bbdev->data->socket_id = pci_dev->device.numa_node; + + /* Invoke FEC FPGA device initialization function */ + fpga_lte_fec_init(bbdev); + + rte_bbdev_log_debug("bbdev id = %u [%s]", + bbdev->data->dev_id, dev_name); + + struct fpga_lte_fec_device *d = bbdev->data->dev_private; + uint32_t version_id = fpga_reg_read_32(d->mmio_base, + FPGA_LTE_FEC_VERSION_ID); + rte_bbdev_log(INFO, "FEC FPGA RTL v%u.%u", + ((uint16_t)(version_id >> 16)), ((uint16_t)version_id)); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (!strcmp(bbdev->device->driver->name, + RTE_STR(FPGA_LTE_FEC_PF_DRIVER_NAME))) + print_static_reg_debug_info(d->mmio_base); +#endif + return 0; +} + +static int +fpga_lte_fec_remove(struct rte_pci_device *pci_dev) +{ + struct rte_bbdev *bbdev; + int ret; + uint8_t dev_id; + + if (pci_dev == NULL) + return -EINVAL; + + /* Find device */ + bbdev = rte_bbdev_get_named_dev(pci_dev->device.name); + if (bbdev == NULL) { + rte_bbdev_log(CRIT, + "Couldn't find HW dev \"%s\" to uninitialise it", + pci_dev->device.name); + return -ENODEV; + } + dev_id = bbdev->data->dev_id; + + /* free device private memory before close */ + rte_free(bbdev->data->dev_private); + + /* Close device */ + ret = rte_bbdev_close(dev_id); + if (ret < 0) + rte_bbdev_log(ERR, + "Device %i failed to close during uninit: %i", + dev_id, ret); + + /* release bbdev from library */ + ret = rte_bbdev_release(bbdev); + if (ret) + rte_bbdev_log(ERR, "Device %i failed to uninit: %i", dev_id, + ret); + + rte_bbdev_log_debug("Destroyed bbdev = %u", dev_id); + + return 0; +} + +static inline void +set_default_fpga_conf(struct fpga_lte_fec_conf *def_conf) +{ + /* clear default configuration before initialization */ + memset(def_conf, 0, sizeof(struct fpga_lte_fec_conf)); + /* Set pf mode to true */ + def_conf->pf_mode_en = true; + + /* Set ratio between UL and DL to 1:1 (unit of weight is 3 CBs) */ + def_conf->ul_bandwidth = 3; + def_conf->dl_bandwidth = 3; + + /* Set Load Balance Factor to 64 */ + def_conf->dl_load_balance = 64; + def_conf->ul_load_balance = 64; +} + +/* Initial configuration of FPGA LTE FEC device */ +int +fpga_lte_fec_configure(const char *dev_name, + const struct fpga_lte_fec_conf *conf) +{ + uint32_t payload_32, address; + uint16_t payload_16; + uint8_t payload_8; + uint16_t q_id, vf_id, total_q_id, total_ul_q_id, total_dl_q_id; + struct rte_bbdev *bbdev = rte_bbdev_get_named_dev(dev_name); + struct fpga_lte_fec_conf def_conf; + + if (bbdev == NULL) { + rte_bbdev_log(ERR, + "Invalid dev_name (%s), or device is not yet initialised", + dev_name); + return -ENODEV; + } + + struct fpga_lte_fec_device *d = bbdev->data->dev_private; + + if (conf == NULL) { + rte_bbdev_log(ERR, + "FPGA Configuration was not provided. Default configuration will be loaded."); + set_default_fpga_conf(&def_conf); + conf = &def_conf; + } + + /* + * Configure UL:DL ratio. + * [7:0]: UL weight + * [15:8]: DL weight + */ + payload_16 = (conf->dl_bandwidth << 8) | conf->ul_bandwidth; + address = FPGA_LTE_FEC_CONFIGURATION; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Clear all queues registers */ + payload_32 = FPGA_INVALID_HW_QUEUE_ID; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + address = (q_id << 2) + FPGA_LTE_FEC_QUEUE_MAP; + fpga_reg_write_32(d->mmio_base, address, payload_32); + } + + /* + * If PF mode is enabled allocate all queues for PF only. + * + * For VF mode each VF can have different number of UL and DL queues. + * Total number of queues to configure cannot exceed FPGA + * capabilities - 64 queues - 32 queues for UL and 32 queues for DL. + * Queues mapping is done according to configuration: + * + * UL queues: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | ... | 0 | + * | conf->vf_dl_queues_number[0] - 1 | 0 | + * | conf->vf_dl_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_dl_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_dl_queues_number[7] - 1 | 7 | + * + * DL queues: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | ... | 0 | + * | conf->vf_ul_queues_number[0] - 1 | 0 | + * | conf->vf_ul_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_ul_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_ul_queues_number[7] - 1 | 7 | + * + * Example of configuration: + * conf->vf_ul_queues_number[0] = 4; -> 4 UL queues for VF0 + * conf->vf_dl_queues_number[0] = 4; -> 4 DL queues for VF0 + * conf->vf_ul_queues_number[1] = 2; -> 2 UL queues for VF1 + * conf->vf_dl_queues_number[1] = 2; -> 2 DL queues for VF1 + * + * UL: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | 1 | 0 | + * | 2 | 0 | + * | 3 | 0 | + * | 4 | 1 | + * | 5 | 1 | + * + * DL: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | 33 | 0 | + * | 34 | 0 | + * | 35 | 0 | + * | 36 | 1 | + * | 37 | 1 | + */ + if (conf->pf_mode_en) { + payload_32 = 0x1; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + address = (q_id << 2) + FPGA_LTE_FEC_QUEUE_MAP; + fpga_reg_write_32(d->mmio_base, address, payload_32); + } + } else { + /* Calculate total number of UL and DL queues to configure */ + total_ul_q_id = total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + total_ul_q_id += conf->vf_ul_queues_number[vf_id]; + total_dl_q_id += conf->vf_dl_queues_number[vf_id]; + } + total_q_id = total_dl_q_id + total_ul_q_id; + /* + * Check if total number of queues to configure does not exceed + * FPGA capabilities (64 queues - 32 UL and 32 DL queues) + */ + if ((total_ul_q_id > FPGA_NUM_UL_QUEUES) || + (total_dl_q_id > FPGA_NUM_DL_QUEUES) || + (total_q_id > FPGA_TOTAL_NUM_QUEUES)) { + rte_bbdev_log(ERR, + "FPGA Configuration failed. Too many queues to configure: UL_Q %u, DL_Q %u, FPGA_Q %u", + total_ul_q_id, total_dl_q_id, + FPGA_TOTAL_NUM_QUEUES); + return -EINVAL; + } + total_ul_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_ul_queues_number[vf_id]; + ++q_id, ++total_ul_q_id) { + address = (total_ul_q_id << 2) + + FPGA_LTE_FEC_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_reg_write_32(d->mmio_base, address, + payload_32); + } + } + total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_dl_queues_number[vf_id]; + ++q_id, ++total_dl_q_id) { + address = ((total_dl_q_id + FPGA_NUM_UL_QUEUES) + << 2) + FPGA_LTE_FEC_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_reg_write_32(d->mmio_base, address, + payload_32); + } + } + } + + /* Setting Load Balance Factor */ + payload_16 = (conf->dl_load_balance << 8) | (conf->ul_load_balance); + address = FPGA_LTE_FEC_LOAD_BALANCE_FACTOR; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting length of ring descriptor entry */ + payload_16 = FPGA_RING_DESC_ENTRY_LENGTH; + address = FPGA_LTE_FEC_RING_DESC_LEN; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting FLR timeout value */ + payload_16 = conf->flr_time_out; + address = FPGA_LTE_FEC_FLR_TIME_OUT; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Queue PF/VF mapping table is ready */ + payload_8 = 0x1; + address = FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE; + fpga_reg_write_8(d->mmio_base, address, payload_8); + + rte_bbdev_log_debug("PF FPGA LTE FEC configuration complete for %s", + dev_name); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_static_reg_debug_info(d->mmio_base); +#endif + return 0; +} + +/* FPGA LTE FEC PCI PF address map */ +static struct rte_pci_id pci_id_fpga_lte_fec_pf_map[] = { + { + RTE_PCI_DEVICE(FPGA_LTE_FEC_VENDOR_ID, + FPGA_LTE_FEC_PF_DEVICE_ID) + }, + {.device_id = 0}, +}; + +static struct rte_pci_driver fpga_lte_fec_pci_pf_driver = { + .probe = fpga_lte_fec_probe, + .remove = fpga_lte_fec_remove, + .id_table = pci_id_fpga_lte_fec_pf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING +}; + +/* FPGA LTE FEC PCI VF address map */ +static struct rte_pci_id pci_id_fpga_lte_fec_vf_map[] = { + { + RTE_PCI_DEVICE(FPGA_LTE_FEC_VENDOR_ID, + FPGA_LTE_FEC_VF_DEVICE_ID) + }, + {.device_id = 0}, +}; + +static struct rte_pci_driver fpga_lte_fec_pci_vf_driver = { + .probe = fpga_lte_fec_probe, + .remove = fpga_lte_fec_remove, + .id_table = pci_id_fpga_lte_fec_vf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING +}; + + +RTE_PMD_REGISTER_PCI(FPGA_LTE_FEC_PF_DRIVER_NAME, fpga_lte_fec_pci_pf_driver); +RTE_PMD_REGISTER_PCI_TABLE(FPGA_LTE_FEC_PF_DRIVER_NAME, + pci_id_fpga_lte_fec_pf_map); +RTE_PMD_REGISTER_PCI(FPGA_LTE_FEC_VF_DRIVER_NAME, fpga_lte_fec_pci_vf_driver); +RTE_PMD_REGISTER_PCI_TABLE(FPGA_LTE_FEC_VF_DRIVER_NAME, + pci_id_fpga_lte_fec_vf_map); + +RTE_INIT(fpga_lte_fec_init_log) +{ + fpga_lte_fec_logtype = rte_log_register("pmd.bb.fpga_lte_fec"); + if (fpga_lte_fec_logtype >= 0) +#ifdef RTE_LIBRTE_BBDEV_DEBUG + rte_log_set_level(fpga_lte_fec_logtype, RTE_LOG_DEBUG); +#else + rte_log_set_level(fpga_lte_fec_logtype, RTE_LOG_NOTICE); +#endif +} diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h new file mode 100644 index 0000000..2c6952e --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2018 Intel Corporation + */ + +#ifndef _FPGA_LTE_FEC_H_ +#define _FPGA_LTE_FEC_H_ + +#include <stdint.h> +#include <stdbool.h> + +/** + * @file fpga_lte_fec.h + * + * Interface for Intel(R) FGPA LTE FEC device configuration at the host level, + * directly accessible by the application. + * Configuration related to LTE Turbo coding functionality is done through + * librte_bbdev library. + * + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + */ + +#ifdef __cplusplus +extern "C" { +#endif + +/**< Number of Virtual Functions FGPA 4G FEC supports */ +#define FPGA_LTE_FEC_NUM_VFS 8 + +/** + * Structure to pass FPGA 4G FEC configuration. + */ +struct fpga_lte_fec_conf { + /**< 1 if PF is used for dataplane, 0 for VFs */ + bool pf_mode_en; + /**< Number of UL queues per VF */ + uint8_t vf_ul_queues_number[FPGA_LTE_FEC_NUM_VFS]; + /**< Number of DL queues per VF */ + uint8_t vf_dl_queues_number[FPGA_LTE_FEC_NUM_VFS]; + /**< UL bandwidth. Needed for schedule algorithm */ + uint8_t ul_bandwidth; + /**< DL bandwidth. Needed for schedule algorithm */ + uint8_t dl_bandwidth; + /**< UL Load Balance */ + uint8_t ul_load_balance; + /**< DL Load Balance */ + uint8_t dl_load_balance; + /**< FLR timeout value */ + uint16_t flr_time_out; +}; + +/** + * Configure Intel(R) FPGA LTE FEC device + * + * @param dev_name + * The name of the device. This is the short form of PCI BDF, e.g. 00:01.0. + * It can also be retrieved for a bbdev device from the dev_name field in the + * rte_bbdev_info structure returned by rte_bbdev_info_get(). + * @param conf + * Configuration to apply to FPGA 4G FEC. + * + * @return + * Zero on success, negative value on failure. + */ +int +fpga_lte_fec_configure(const char *dev_name, + const struct fpga_lte_fec_conf *conf); + +#ifdef __cplusplus +} +#endif + +#endif /* _FPGA_LTE_FEC_H_ */ diff --git a/drivers/baseband/fpga_lte_fec/meson.build b/drivers/baseband/fpga_lte_fec/meson.build new file mode 100644 index 0000000..bf44e6b --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +deps += ['bbdev', 'bus_vdev', 'ring', 'pci', 'bus_pci'] +name = 'bbdev_fpga_lte_fec' +allow_experimental_apis = true +sources = files('fpga_lte_fec.c') diff --git a/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map new file mode 100644 index 0000000..1a99f8d --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map @@ -0,0 +1,3 @@ +DPDK_18.08 { + local: *; +}; diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build index 52489df..a17d549 100644 --- a/drivers/baseband/meson.build +++ b/drivers/baseband/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <bluca@debian.org> -drivers = ['null'] +drivers = ['null', 'fpga_lte_fec'] config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' driver_name_fmt = 'rte_pmd_@0@' diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 7c9b4b5..21d32a2 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -218,6 +218,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_NETVSC_PMD) += -lrte_pmd_netvsc ifeq ($(CONFIG_RTE_LIBRTE_BBDEV),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += -lrte_pmd_fpga_lte_fec # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v4] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-06 9:04 ` Nicolas Chautru 2019-06-08 0:17 ` [dpdk-dev] [PATCH v4] " Nicolas Chautru @ 2019-06-10 17:01 ` Nicolas Chautru 2019-06-10 17:01 ` Nicolas Chautru 1 sibling, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-10 17:01 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Update v4: Fix warning for the DEBUG configuration. Update v3: Squashing 3 previous patches into one as recommended. This is adding a new PMD driver for BBDEV device based on FPGA implementation on PAC N3000 HW to provide FEC 4G acceleration. v1 was shared earlier on this patchwork : https://patches.dpdk.org/patch/53409/ The existing BBDEV test framework is still supported. Main updates based on feedback during the last v1 were: - correct the documentation which was arguably misleading in hinting that ICC was required. There is no dependency whatsoever. - add meson build support Note that a number of other BBDEV patches are in v1 here and will be resubmitted for 19.08 rc1 https://patches.dpdk.org/project/dpdk/list/?series=4657 Nicolas Chautru (1): baseband/fpga_lte_fec: adding driver for FEC on FPGA config/common_base | 6 + doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++ doc/guides/bbdevs/index.rst | 1 + drivers/baseband/Makefile | 2 + drivers/baseband/fpga_lte_fec/Makefile | 29 + drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 ++++++++++++++++++++ drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + drivers/baseband/fpga_lte_fec/meson.build | 7 + .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + drivers/baseband/meson.build | 2 +- mk/rte.app.mk | 1 + 11 files changed, 3115 insertions(+), 1 deletion(-) create mode 100644 doc/guides/bbdevs/fpga_lte_fec.rst create mode 100644 drivers/baseband/fpga_lte_fec/Makefile create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h create mode 100644 drivers/baseband/fpga_lte_fec/meson.build create mode 100644 drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v4] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-10 17:01 ` Nicolas Chautru @ 2019-06-10 17:01 ` Nicolas Chautru [not found] ` <EEA9FF629BF25B47BD67ADE995041EE2496A888B@IRSMSX103.ger.corp.intel.com> ` (3 more replies) 0 siblings, 4 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-10 17:01 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Supports for FEC 4G PMD Driver on FPGA card PAC N3000 Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 6 + doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++ doc/guides/bbdevs/index.rst | 1 + drivers/baseband/Makefile | 2 + drivers/baseband/fpga_lte_fec/Makefile | 29 + drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 ++++++++++++++++++++ drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + drivers/baseband/fpga_lte_fec/meson.build | 7 + .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + drivers/baseband/meson.build | 2 +- mk/rte.app.mk | 1 + 11 files changed, 3115 insertions(+), 1 deletion(-) create mode 100644 doc/guides/bbdevs/fpga_lte_fec.rst create mode 100644 drivers/baseband/fpga_lte_fec/Makefile create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h create mode 100644 drivers/baseband/fpga_lte_fec/meson.build create mode 100644 drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map diff --git a/config/common_base b/config/common_base index 6b96e0e..ae52028 100644 --- a/config/common_base +++ b/config/common_base @@ -521,6 +521,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y # EXPERIMENTAL: API may change without prior notice # CONFIG_RTE_LIBRTE_BBDEV=y +CONFIG_RTE_LIBRTE_BBDEV_DEBUG=n CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y @@ -535,6 +536,11 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n # +# Compile PMD for Intel FPGA LTE FEC bbdev device +# +CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC=y + +# # Compile generic crypto device library # CONFIG_RTE_LIBRTE_CRYPTODEV=y diff --git a/doc/guides/bbdevs/fpga_lte_fec.rst b/doc/guides/bbdevs/fpga_lte_fec.rst new file mode 100644 index 0000000..71b058c --- /dev/null +++ b/doc/guides/bbdevs/fpga_lte_fec.rst @@ -0,0 +1,318 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation + +Intel(R) FPGA LTE FEC Poll Mode Driver +====================================== + +The BBDEV FPGA LTE FEC poll mode driver (PMD) supports an FPGA implementation of a VRAN +Turbo Encode / Decode LTE wireless acceleration function, using Intel's PCI-e and FPGA +based Vista Creek device. + +Features +-------- + +FPGA LTE FEC PMD supports the following features: + +- Turbo Encode in the DL with total throughput of 4.5 Gbits/s +- Turbo Decode in the UL with total throughput of 1.5 Gbits/s assuming 8 decoder iterations +- 8 VFs per PF (physical device) +- Maximum of 32 UL queues per VF +- Maximum of 32 DL queues per VF +- PCIe Gen-3 x8 Interface +- MSI-X +- SR-IOV + + +FPGA LTE FEC PMD supports the following BBDEV capabilities: + +* For the turbo encode operation: + - ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` : set to attach CRC24B to CB(s) + - ``RTE_BBDEV_TURBO_RATE_MATCH`` : if set then do not do Rate Match bypass + - ``RTE_BBDEV_TURBO_ENC_INTERRUPTS`` : set for encoder dequeue interrupts + + +* For the turbo decode operation: + - ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` : check CRC24B from CB(s) + - ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` : perform subblock de-interleave + - ``RTE_BBDEV_TURBO_DEC_INTERRUPTS`` : set for decoder dequeue interrupts + - ``RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN`` : set if negative LLR encoder i/p is supported + - ``RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP`` : keep CRC24B bits appended while decoding + + +Limitations +----------- + +FPGA LTE FEC does not support the following: + +- Scatter-Gather function + + +Installation +-------------- + +Section 3 of the DPDK manual provides instuctions on installing and compiling DPDK. The +default set of bbdev compile flags may be found in config/common_base, where for example +the flag to build the FPGA LTE FEC device, ``CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC``, is already +set. It is assumed DPDK has been compiled using for instance: + +.. code-block:: console + + make install T=x86_64-native-linuxapp-gcc + + +DPDK requires hugepages to be configured as detailed in section 2 of the DPDK manual. +The bbdev test application has been tested with a configuration 40 x 1GB hugepages. The +hugepage configuration of a server may be examined using: + +.. code-block:: console + + grep Huge* /proc/meminfo + + +Initialization +-------------- + +When the device first powers up, its PCI Physical Functions (PF) can be listed through this command: + +.. code-block:: console + + sudo lspci -vd1172:5052 + +The physical and virtual functions are compatible with Linux UIO drivers: +``vfio`` and ``igb_uio``. However, in order to work the FPGA LTE FEC device firstly needs +to be bound to one of these linux drivers through DPDK. + + +Bind PF UIO driver(s) +~~~~~~~~~~~~~~~~~~~~~ + +Install the DPDK igb_uio driver, bind it with the PF PCI device ID and use +``lspci`` to confirm the PF device is under use by ``igb_uio`` DPDK UIO driver. + +The igb_uio driver may be bound to the PF PCI device using one of three methods: + + +1. PCI functions (physical or virtual, depending on the use case) can be bound to +the UIO driver by repeating this command for every function. + +.. code-block:: console + + cd <dpdk-top-level-directory> + insmod ./build/kmod/igb_uio.ko + echo "1172 5052" > /sys/bus/pci/drivers/igb_uio/new_id + lspci -vd1172: + + +2. Another way to bind PF with DPDK UIO driver is by using the ``dpdk-devbind.py`` tool + +.. code-block:: console + + cd <dpdk-top-level-directory> + ./usertools/dpdk-devbind.py -b igb_uio 0000:06:00.0 + +where the PCI device ID (example: 0000:06:00.0) is obtained using lspci -vd1172: + + +3. A third way to bind is to use ``dpdk-setup.sh`` tool + +.. code-block:: console + + cd <dpdk-top-level-directory> + ./usertools/dpdk-setup.sh + + select 'Bind Ethernet/Crypto/Baseband device to IGB UIO module' + or + select 'Bind Ethernet/Crypto/Baseband device to VFIO module' depending on driver required + enter PCI device ID + select 'Display current Ethernet/Crypto/Baseband device settings' to confirm binding + + +In the same way the FPGA LTE FEC PF can be bound with vfio, but vfio driver does not +support SR-IOV configuration right out of the box, so it will need to be patched. + + +Enable Virtual Functions +~~~~~~~~~~~~~~~~~~~~~~~~ + +Now, it should be visible in the printouts that PCI PF is under igb_uio control +"``Kernel driver in use: igb_uio``" + +To show the number of available VFs on the device, read ``sriov_totalvfs`` file.. + +.. code-block:: console + + cat /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_totalvfs + + where 0000\:<b>\:<d>.<f> is the PCI device ID + + +To enable VFs via igb_uio, echo the number of virtual functions intended to +enable to ``max_vfs`` file.. + +.. code-block:: console + + echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/max_vfs + + +Afterwards, all VFs must be bound to appropriate UIO drivers as required, same +way it was done with the physical function previously. + +Enabling SR-IOV via vfio driver is pretty much the same, except that the file +name is different: + +.. code-block:: console + + echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_numvfs + + +Configure the VFs through PF +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The PCI virtual functions must be configured before working or getting assigned +to VMs/Containers. The configuration involves allocating the number of hardware +queues, priorities, load balance, bandwidth and other settings necessary for the +device to perform FEC functions. + +This configuration needs to be executed at least once after reboot or PCI FLR and can +be achieved by using the function ``fpga_lte_fec_configure()``, which sets up the +parameters defined in ``fpga_lte_fec_conf`` structure: + +.. code-block:: c + + struct fpga_lte_fec_conf { + bool pf_mode_en; + uint8_t vf_ul_queues_number[FPGA_LTE_FEC_NUM_VFS]; + uint8_t vf_dl_queues_number[FPGA_LTE_FEC_NUM_VFS]; + uint8_t ul_bandwidth; + uint8_t dl_bandwidth; + uint8_t ul_load_balance; + uint8_t dl_load_balance; + uint16_t flr_time_out; + }; + +- ``pf_mode_en``: identifies whether only PF is to be used, or the VFs. PF and + VFs are mutually exclusive and cannot run simultaneously. + Set to 1 for PF mode enabled. + If PF mode is enabled all queues available in the device are assigned + exclusively to PF and 0 queues given to VFs. + +- ``vf_*l_queues_number``: defines the hardware queue mapping for every VF. + +- ``*l_bandwidth``: in case of congestion on PCIe interface. The device + allocates different bandwidth to UL and DL. The weight is configured by this + setting. The unit of weight is 3 code blocks. For example, if the code block + cbps (code block per second) ratio between UL and DL is 12:1, then the + configuration value should be set to 36:3. The schedule algorithm is based + on code block regardless the length of each block. + +- ``*l_load_balance``: hardware queues are load-balanced in a round-robin + fashion. Queues get filled first-in first-out until they reach a pre-defined + watermark level, if exceeded, they won't get assigned new code blocks.. + This watermark is defined by this setting. + + If all hardware queues exceeds the watermark, no code blocks will be + streamed in from UL/DL code block FIFO. + +- ``flr_time_out``: specifies how many 16.384us to be FLR time out. The + time_out = flr_time_out x 16.384us. For instance, if you want to set 10ms for + the FLR time out then set this setting to 0x262=610. + + +An example configuration code calling the function ``fpga_lte_fec_configure()`` is shown +below: + +.. code-block:: c + + struct fpga_lte_fec_conf conf; + unsigned int i; + + memset(&conf, 0, sizeof(struct fpga_lte_fec_conf)); + conf.pf_mode_en = 1; + + for (i = 0; i < FPGA_LTE_FEC_NUM_VFS; ++i) { + conf.vf_ul_queues_number[i] = 4; + conf.vf_dl_queues_number[i] = 4; + } + conf.ul_bandwidth = 12; + conf.dl_bandwidth = 5; + conf.dl_load_balance = 64; + conf.ul_load_balance = 64; + + /* setup FPGA PF */ + ret = fpga_lte_fec_configure(info->dev_name, &conf); + TEST_ASSERT_SUCCESS(ret, + "Failed to configure 4G FPGA PF for bbdev %s", + info->dev_name); + + +Test Application +---------------- + +BBDEV provides a test application, ``test-bbdev.py`` and range of test data for testing +the functionality of FPGA LTE FEC turbo encode and turbo decode, depending on the device's +capabilities. The test application is located under app->test-bbdev folder and has the +following options: + +.. code-block:: console + + "-p", "--testapp-path": specifies path to the bbdev test app. + "-e", "--eal-params" : EAL arguments which are passed to the test app. + "-t", "--timeout" : Timeout in seconds (default=300). + "-c", "--test-cases" : Defines test cases to run. Run all if not specified. + "-v", "--test-vector" : Test vector path (default=dpdk_path+/app/test-bbdev/test_vectors/bbdev_null.data). + "-n", "--num-ops" : Number of operations to process on device (default=32). + "-b", "--burst-size" : Operations enqueue/dequeue burst size (default=32). + "-l", "--num-lcores" : Number of lcores to run (default=16). + "-i", "--init-device" : Initialise PF device with default values. + + +To execute the test application tool using simple turbo decode or turbo encode data, +type one of the following: + +.. code-block:: console + + ./test-bbdev.py -c validation -n 64 -b 8 -v ./turbo_dec_default.data + ./test-bbdev.py -c validation -n 64 -b 8 -v ./turbo_enc_default.data + + +The test application ``test-bbdev.py``, supports the ability to configure the PF device with +a default set of values, if the "-i" or "- -init-device" option is included. The default values +are defined in test_bbdev_perf.c as: + +- VF_UL_QUEUE_VALUE 4 +- VF_DL_QUEUE_VALUE 4 +- UL_BANDWIDTH 3 +- DL_BANDWIDTH 3 +- UL_LOAD_BALANCE 128 +- DL_LOAD_BALANCE 128 +- FLR_TIMEOUT 610 + + +Test Vectors +~~~~~~~~~~~~ + +In addition to the simple turbo decoder and turbo encoder tests, bbdev also provides +a range of additional tests under the test_vectors folder, which may be useful. The results +of these tests will depend on the FPGA LTE FEC capabilities: + +* turbo decoder tests: + - ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_high_snr.data`` + - ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_low_snr.data`` + - ``turbo_dec_c1_k6144_r0_e34560_negllr.data`` + - ``turbo_dec_c1_k6144_r0_e34560_sbd_negllr.data`` + - ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr_crc24b.data`` + - ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr.data`` + + +* turbo encoder tests: + - ``turbo_enc_c1_k40_r0_e1190_rm.data`` + - ``turbo_enc_c1_k40_r0_e1194_rm.data`` + - ``turbo_enc_c1_k40_r0_e1196_rm.data`` + - ``turbo_enc_c1_k40_r0_e272_rm.data`` + - ``turbo_enc_c1_k6144_r0_e18444.data`` + - ``turbo_enc_c1_k6144_r0_e32256_crc24b_rm.data`` + - ``turbo_enc_c2_k5952_r0_e17868_crc24b.data`` + - ``turbo_enc_c3_k4800_r2_e14412_crc24b.data`` + - ``turbo_enc_c4_k4800_r2_e14412_crc24b.data`` + + diff --git a/doc/guides/bbdevs/index.rst b/doc/guides/bbdevs/index.rst index 93276ed..005b95e 100644 --- a/doc/guides/bbdevs/index.rst +++ b/doc/guides/bbdevs/index.rst @@ -10,3 +10,4 @@ Baseband Device Drivers null turbo_sw + fpga_lte_fec diff --git a/drivers/baseband/Makefile b/drivers/baseband/Makefile index 4ec83b0..ceffc7d 100644 --- a/drivers/baseband/Makefile +++ b/drivers/baseband/Makefile @@ -10,5 +10,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += null DEPDIRS-null = $(core-libs) DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += turbo_sw DEPDIRS-turbo_sw = $(core-libs) +DIRS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += fpga_lte_fec +DEPDIRS-fpga_lte_fec = $(core-libs) include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/drivers/baseband/fpga_lte_fec/Makefile b/drivers/baseband/fpga_lte_fec/Makefile new file mode 100644 index 0000000..e478d84 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/Makefile @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2018 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_pmd_fpga_lte_fec.a + +# build flags +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring +LDLIBS += -lrte_bbdev +LDLIBS += -lrte_pci -lrte_bus_pci + +# versioning export map +EXPORT_MAP := rte_pmd_bbdev_fpga_lte_fec_version.map + +# library version +LIBABIVER := 1 + +# library source files +SRCS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += fpga_lte_fec.c + +# export include files +SYMLINK-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC)-include += fpga_lte_fec.h + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c new file mode 100644 index 0000000..4dc6e98 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c @@ -0,0 +1,2674 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2018 Intel Corporation + */ + +#include <unistd.h> + +#include <rte_common.h> +#include <rte_log.h> +#include <rte_dev.h> +#include <rte_malloc.h> +#include <rte_mempool.h> +#include <rte_errno.h> +#include <rte_pci.h> +#include <rte_bus_pci.h> +#include <rte_byteorder.h> +#ifdef RTE_BBDEV_OFFLOAD_COST +#include <rte_cycles.h> +#endif + +#include <rte_bbdev.h> +#include <rte_bbdev_pmd.h> + +#include "fpga_lte_fec.h" + +/* Turbo SW PMD logging ID */ +static int fpga_lte_fec_logtype; + +/* Helper macro for logging */ +#define rte_bbdev_log(level, fmt, ...) \ + rte_log(RTE_LOG_ ## level, fpga_lte_fec_logtype, fmt "\n", \ + ##__VA_ARGS__) + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +#define rte_bbdev_log_debug(fmt, ...) \ + rte_bbdev_log(DEBUG, "fpga_lte_fec: " fmt, \ + ##__VA_ARGS__) +#else +#define rte_bbdev_log_debug(fmt, ...) +#endif + +/* FPGA LTE FEC driver names */ +#define FPGA_LTE_FEC_PF_DRIVER_NAME intel_fpga_lte_fec_pf +#define FPGA_LTE_FEC_VF_DRIVER_NAME intel_fpga_lte_fec_vf + +/* FPGA LTE FEC PCI vendor & device IDs */ +#define FPGA_LTE_FEC_VENDOR_ID (0x1172) +#define FPGA_LTE_FEC_PF_DEVICE_ID (0x5052) +#define FPGA_LTE_FEC_VF_DEVICE_ID (0x5050) + +/* Align DMA descriptors to 256 bytes - cache-aligned */ +#define FPGA_RING_DESC_ENTRY_LENGTH (8) +/* Ring size is in 256 bits (32 bytes) units */ +#define FPGA_RING_DESC_LEN_UNIT_BYTES (32) +/* Maximum size of queue */ +#define FPGA_RING_MAX_SIZE (1024) +#define FPGA_FLR_TIMEOUT_UNIT (16.384) + +#define FPGA_NUM_UL_QUEUES (32) +#define FPGA_NUM_DL_QUEUES (32) +#define FPGA_TOTAL_NUM_QUEUES (FPGA_NUM_UL_QUEUES + FPGA_NUM_DL_QUEUES) +#define FPGA_NUM_INTR_VEC (FPGA_TOTAL_NUM_QUEUES - RTE_INTR_VEC_RXTX_OFFSET) + +#define FPGA_INVALID_HW_QUEUE_ID (0xFFFFFFFF) + +#define FPGA_QUEUE_FLUSH_TIMEOUT_US (1000) +#define FPGA_TIMEOUT_CHECK_INTERVAL (5) + +/* FPGA LTE FEC Register mapping on BAR0 */ +enum { + FPGA_LTE_FEC_VERSION_ID = 0x00000000, /* len: 4B */ + FPGA_LTE_FEC_CONFIGURATION = 0x00000004, /* len: 2B */ + FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE = 0x00000008, /* len: 1B */ + FPGA_LTE_FEC_LOAD_BALANCE_FACTOR = 0x0000000a, /* len: 2B */ + FPGA_LTE_FEC_RING_DESC_LEN = 0x0000000c, /* len: 2B */ + FPGA_LTE_FEC_FLR_TIME_OUT = 0x0000000e, /* len: 2B */ + FPGA_LTE_FEC_VFQ_FLUSH_STATUS_LW = 0x00000018, /* len: 4B */ + FPGA_LTE_FEC_VFQ_FLUSH_STATUS_HI = 0x0000001c, /* len: 4B */ + FPGA_LTE_FEC_VF0_DEBUG = 0x00000020, /* len: 4B */ + FPGA_LTE_FEC_VF1_DEBUG = 0x00000024, /* len: 4B */ + FPGA_LTE_FEC_VF2_DEBUG = 0x00000028, /* len: 4B */ + FPGA_LTE_FEC_VF3_DEBUG = 0x0000002c, /* len: 4B */ + FPGA_LTE_FEC_VF4_DEBUG = 0x00000030, /* len: 4B */ + FPGA_LTE_FEC_VF5_DEBUG = 0x00000034, /* len: 4B */ + FPGA_LTE_FEC_VF6_DEBUG = 0x00000038, /* len: 4B */ + FPGA_LTE_FEC_VF7_DEBUG = 0x0000003c, /* len: 4B */ + FPGA_LTE_FEC_QUEUE_MAP = 0x00000040, /* len: 256B */ + FPGA_LTE_FEC_RING_CTRL_REGS = 0x00000200 /* len: 2048B */ +}; + +/* FPGA LTE FEC Ring Control Registers */ +enum { + FPGA_LTE_FEC_RING_HEAD_ADDR = 0x00000008, + FPGA_LTE_FEC_RING_SIZE = 0x00000010, + FPGA_LTE_FEC_RING_MISC = 0x00000014, + FPGA_LTE_FEC_RING_ENABLE = 0x00000015, + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN = 0x00000016, + FPGA_LTE_FEC_RING_SHADOW_TAIL = 0x00000018, + FPGA_LTE_FEC_RING_HEAD_POINT = 0x0000001C +}; + +/* FPGA LTE FEC DESCRIPTOR ERROR */ +enum { + DESC_ERR_NO_ERR = 0x0, + DESC_ERR_K_OUT_OF_RANGE = 0x1, + DESC_ERR_K_NOT_NORMAL = 0x2, + DESC_ERR_KPAI_NOT_NORMAL = 0x3, + DESC_ERR_DESC_OFFSET_ERR = 0x4, + DESC_ERR_DESC_READ_FAIL = 0x8, + DESC_ERR_DESC_READ_TIMEOUT = 0x9, + DESC_ERR_DESC_READ_TLP_POISONED = 0xA, + DESC_ERR_CB_READ_FAIL = 0xC, + DESC_ERR_CB_READ_TIMEOUT = 0xD, + DESC_ERR_CB_READ_TLP_POISONED = 0xE +}; + +/* FPGA LTE FEC DMA Encoding Request Descriptor */ +struct __attribute__((__packed__)) fpga_dma_enc_desc { + uint32_t done:1, + rsrvd0:11, + error:4, + rsrvd1:16; + uint32_t ncb:16, + rsrvd2:14, + rv:2; + uint32_t bypass_rm:1, + irq_en:1, + crc_en:1, + rsrvd3:13, + offset:10, + rsrvd4:6; + uint16_t e; + uint16_t k; + uint32_t out_addr_lw; + uint32_t out_addr_hi; + uint32_t in_addr_lw; + uint32_t in_addr_hi; + + union { + struct { + /* Virtual addresses used to retrieve SW context info */ + void *op_addr; + /* Stores information about total number of Code Blocks + * in currently processed Transport Block + */ + uint64_t cbs_in_op; + }; + + uint8_t sw_ctxt[FPGA_RING_DESC_LEN_UNIT_BYTES * + (FPGA_RING_DESC_ENTRY_LENGTH - 1)]; + }; +}; + +/* FPGA LTE FEC DMA Decoding Request Descriptor */ +struct __attribute__((__packed__)) fpga_dma_dec_desc { + uint32_t done:1, + iter:5, + rsrvd0:2, + crc_pass:1, + rsrvd1:3, + error:4, + crc_type:1, + rsrvd2:7, + max_iter:5, + rsrvd3:3; + uint32_t rsrvd4; + uint32_t bypass_rm:1, + irq_en:1, + drop_crc:1, + rsrvd5:13, + offset:10, + rsrvd6:6; + uint16_t k; + uint16_t in_len; + uint32_t out_addr_lw; + uint32_t out_addr_hi; + uint32_t in_addr_lw; + uint32_t in_addr_hi; + + union { + struct { + /* Virtual addresses used to retrieve SW context info */ + void *op_addr; + /* Stores information about total number of Code Blocks + * in currently processed Transport Block + */ + uint8_t cbs_in_op; + }; + + uint32_t sw_ctxt[8 * (FPGA_RING_DESC_ENTRY_LENGTH - 1)]; + }; +}; + +/* FPGA LTE DMA Descriptor */ +union fpga_dma_desc { + struct fpga_dma_enc_desc enc_req; + struct fpga_dma_dec_desc dec_req; +}; + +/* FPGA LTE FEC Ring Control Register */ +struct __attribute__((__packed__)) fpga_ring_ctrl_reg { + uint64_t ring_base_addr; + uint64_t ring_head_addr; + uint16_t ring_size:11; + uint16_t rsrvd0; + union { /* Miscellaneous register */ + uint8_t misc; + uint8_t max_ul_dec:5, + max_ul_dec_en:1, + rsrvd1:2; + }; + uint8_t enable; + uint8_t flush_queue_en; + uint8_t rsrvd2; + uint16_t shadow_tail; + uint16_t rsrvd3; + uint16_t head_point; + uint16_t rsrvd4; + +}; + +/* Private data structure for each FPGA FEC device */ +struct fpga_lte_fec_device { + /** Base address of MMIO registers (BAR0) */ + void *mmio_base; + /** Base address of memory for sw rings */ + void *sw_rings; + /** Physical address of sw_rings */ + rte_iova_t sw_rings_phys; + /** Number of bytes available for each queue in device. */ + uint32_t sw_ring_size; + /** Max number of entries available for each queue in device */ + uint32_t sw_ring_max_depth; + /** Base address of response tail pointer buffer */ + uint32_t *tail_ptrs; + /** Physical address of tail pointers */ + rte_iova_t tail_ptr_phys; + /** Queues flush completion flag */ + uint64_t *flush_queue_status; + /* Bitmap capturing which Queues are bound to the PF/VF */ + uint64_t q_bound_bit_map; + /* Bitmap capturing which Queues have already been assigned */ + uint64_t q_assigned_bit_map; + /** True if this is a PF FPGA FEC device */ + bool pf_device; +}; + +/* Structure associated with each queue. */ +struct __rte_cache_aligned fpga_queue { + struct fpga_ring_ctrl_reg ring_ctrl_reg; /* Ring Control Register */ + union fpga_dma_desc *ring_addr; /* Virtual address of software ring */ + uint64_t *ring_head_addr; /* Virtual address of completion_head */ + uint64_t shadow_completion_head; /* Shadow completion head value */ + uint16_t head_free_desc; /* Ring head */ + uint16_t tail; /* Ring tail */ + /* Mask used to wrap enqueued descriptors on the sw ring */ + uint32_t sw_ring_wrap_mask; + uint32_t irq_enable; /* Enable ops dequeue interrupts if set to 1 */ + uint8_t q_idx; /* Queue index */ + struct fpga_lte_fec_device *d; + /* MMIO register of shadow_tail used to enqueue descriptors */ + void *shadow_tail_addr; +}; + +/* Write to 16 bit MMIO register address */ +static inline void +mmio_write_16(void *addr, uint16_t value) +{ + *((volatile uint16_t *)(addr)) = rte_cpu_to_le_16(value); +} + +/* Write to 32 bit MMIO register address */ +static inline void +mmio_write_32(void *addr, uint32_t value) +{ + *((volatile uint32_t *)(addr)) = rte_cpu_to_le_32(value); +} + +/* Write to 64 bit MMIO register address */ +static inline void +mmio_write_64(void *addr, uint64_t value) +{ + *((volatile uint64_t *)(addr)) = rte_cpu_to_le_64(value); +} + +/* Write a 8 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_8(void *mmio_base, uint32_t offset, uint8_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + *((volatile uint8_t *)(reg_addr)) = payload; +} + +/* Write a 16 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_16(void *mmio_base, uint32_t offset, uint16_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_16(reg_addr, payload); +} + +/* Write a 32 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_32(void *mmio_base, uint32_t offset, uint32_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_32(reg_addr, payload); +} + +/* Write a 64 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_64(void *mmio_base, uint32_t offset, uint64_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_64(reg_addr, payload); +} + +/* Write a ring control register of a FPGA LTE FEC device */ +static inline void +fpga_ring_reg_write(void *mmio_base, uint32_t offset, + struct fpga_ring_ctrl_reg payload) +{ + fpga_reg_write_64(mmio_base, offset, payload.ring_base_addr); + fpga_reg_write_64(mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_ADDR, + payload.ring_head_addr); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_SIZE, + payload.ring_size); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_POINT, + payload.head_point); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN, + payload.flush_queue_en); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_SHADOW_TAIL, + payload.shadow_tail); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_MISC, + payload.misc); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + payload.enable); +} + +/* Read a register of FPGA LTE FEC device */ +static uint32_t +fpga_reg_read_32(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint32_t ret = *((volatile uint32_t *)(reg_addr)); + return rte_le_to_cpu_32(ret); +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Read a register of FPGA LTE FEC device */ +static uint8_t +fpga_reg_read_8(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + return *((volatile uint8_t *)(reg_addr)); +} + +/* Read a register of FPGA LTE FEC device */ +static uint16_t +fpga_reg_read_16(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint16_t ret = *((volatile uint16_t *)(reg_addr)); + return rte_le_to_cpu_16(ret); +} + +/* Read a register of FPGA LTE FEC device */ +static uint64_t +fpga_reg_read_64(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint64_t ret = *((volatile uint64_t *)(reg_addr)); + return rte_le_to_cpu_64(ret); +} + +/* Read Ring Control Register of FPGA LTE FEC device */ +static inline void +print_ring_reg_debug_info(void *mmio_base, uint32_t offset) +{ + rte_bbdev_log_debug( + "FPGA MMIO base address @ %p | Ring Control Register @ offset = 0x%08" + PRIx32, mmio_base, offset); + rte_bbdev_log_debug( + "RING_BASE_ADDR = 0x%016"PRIx64, + fpga_reg_read_64(mmio_base, offset)); + rte_bbdev_log_debug( + "RING_HEAD_ADDR = 0x%016"PRIx64, + fpga_reg_read_64(mmio_base, offset + + FPGA_LTE_FEC_RING_HEAD_ADDR)); + rte_bbdev_log_debug( + "RING_SIZE = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_SIZE)); + rte_bbdev_log_debug( + "RING_MISC = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_MISC)); + rte_bbdev_log_debug( + "RING_ENABLE = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_ENABLE)); + rte_bbdev_log_debug( + "RING_FLUSH_QUEUE_EN = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN)); + rte_bbdev_log_debug( + "RING_SHADOW_TAIL = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_SHADOW_TAIL)); + rte_bbdev_log_debug( + "RING_HEAD_POINT = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_HEAD_POINT)); +} + +/* Read Static Register of FPGA LTE FEC device */ +static inline void +print_static_reg_debug_info(void *mmio_base) +{ + uint16_t config = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_CONFIGURATION); + uint8_t qmap_done = fpga_reg_read_8(mmio_base, + FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE); + uint16_t lb_factor = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_LOAD_BALANCE_FACTOR); + uint16_t ring_desc_len = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_RING_DESC_LEN); + uint16_t flr_time_out = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_FLR_TIME_OUT); + + rte_bbdev_log_debug("UL.DL Weights = %u.%u", + ((uint8_t)config), ((uint8_t)(config >> 8))); + rte_bbdev_log_debug("UL.DL Load Balance = %u.%u", + ((uint8_t)lb_factor), ((uint8_t)(lb_factor >> 8))); + rte_bbdev_log_debug("Queue-PF/VF Mapping Table = %s", + (qmap_done > 0) ? "READY" : "NOT-READY"); + rte_bbdev_log_debug("Ring Descriptor Size = %u bytes", + ring_desc_len*FPGA_RING_DESC_LEN_UNIT_BYTES); + rte_bbdev_log_debug("FLR Timeout = %f usec", + (float)flr_time_out*FPGA_FLR_TIMEOUT_UNIT); +} + +/* Print decode DMA Descriptor of FPGA LTE FEC device */ +static void +print_dma_dec_desc_debug_info(union fpga_dma_desc *desc) +{ + rte_bbdev_log_debug("DMA response desc %p\n" + "\t-- done(%"PRIu32") | iter(%"PRIu32") | crc_pass(%"PRIu32")" + " | error (%"PRIu32") | crc_type(%"PRIu32")\n" + "\t-- max_iter(%"PRIu32") | bypass_rm(%"PRIu32") | " + "irq_en (%"PRIu32") | drop_crc(%"PRIu32") | offset(%"PRIu32")\n" + "\t-- k(%"PRIu32") | in_len (%"PRIu16") | op_add(%p)\n" + "\t-- cbs_in_op(%"PRIu32") | in_add (0x%08"PRIx32"%08"PRIx32") | " + "out_add (0x%08"PRIx32"%08"PRIx32")", + desc, + (uint32_t)desc->dec_req.done, + (uint32_t)desc->dec_req.iter, + (uint32_t)desc->dec_req.crc_pass, + (uint32_t)desc->dec_req.error, + (uint32_t)desc->dec_req.crc_type, + (uint32_t)desc->dec_req.max_iter, + (uint32_t)desc->dec_req.bypass_rm, + (uint32_t)desc->dec_req.irq_en, + (uint32_t)desc->dec_req.drop_crc, + (uint32_t)desc->dec_req.offset, + (uint32_t)desc->dec_req.k, + (uint16_t)desc->dec_req.in_len, + desc->dec_req.op_addr, + (uint32_t)desc->dec_req.cbs_in_op, + (uint32_t)desc->dec_req.in_addr_hi, + (uint32_t)desc->dec_req.in_addr_lw, + (uint32_t)desc->dec_req.out_addr_hi, + (uint32_t)desc->dec_req.out_addr_lw); +} +#endif + +static int +fpga_setup_queues(struct rte_bbdev *dev, uint16_t num_queues, int socket_id) +{ + /* Number of queues bound to a PF/VF */ + uint32_t hw_q_num = 0; + uint32_t ring_size, payload, address, q_id, offset; + rte_iova_t phys_addr; + struct fpga_ring_ctrl_reg ring_reg; + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + + address = FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE; + if (!(fpga_reg_read_32(fpga_dev->mmio_base, address) & 0x1)) { + rte_bbdev_log(ERR, + "Queue-PF/VF mapping is not set! Was PF configured for device (%s) ?", + dev->data->name); + return -EPERM; + } + + /* Clear queue registers structure */ + memset(&ring_reg, 0, sizeof(struct fpga_ring_ctrl_reg)); + + /* Scan queue map. + * If a queue is valid and mapped to a calling PF/VF the read value is + * replaced with a queue ID and if it's not then + * FPGA_INVALID_HW_QUEUE_ID is returned. + */ + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + uint32_t hw_q_id = fpga_reg_read_32(fpga_dev->mmio_base, + FPGA_LTE_FEC_QUEUE_MAP + (q_id << 2)); + + rte_bbdev_log_debug("%s: queue ID: %u, registry queue ID: %u", + dev->device->name, q_id, hw_q_id); + + if (hw_q_id != FPGA_INVALID_HW_QUEUE_ID) { + fpga_dev->q_bound_bit_map |= (1ULL << q_id); + /* Clear queue register of found queue */ + offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q_id); + fpga_ring_reg_write(fpga_dev->mmio_base, + offset, ring_reg); + ++hw_q_num; + } + } + if (hw_q_num == 0) { + rte_bbdev_log(ERR, + "No HW queues assigned to this device. Probably this is a VF configured for PF mode. Check device configuration!"); + return -ENODEV; + } + + if (num_queues > hw_q_num) { + rte_bbdev_log(ERR, + "Not enough queues for device %s! Requested: %u, available: %u", + dev->device->name, num_queues, hw_q_num); + return -EINVAL; + } + + ring_size = FPGA_RING_MAX_SIZE * sizeof(struct fpga_dma_dec_desc); + + /* Enforce 32 byte alignment */ + RTE_BUILD_BUG_ON((RTE_CACHE_LINE_SIZE % 32) != 0); + + /* Allocate memory for SW descriptor rings */ + fpga_dev->sw_rings = rte_zmalloc_socket(dev->device->driver->name, + num_queues * ring_size, RTE_CACHE_LINE_SIZE, + socket_id); + if (fpga_dev->sw_rings == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u sw_rings", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + + fpga_dev->sw_rings_phys = rte_malloc_virt2iova(fpga_dev->sw_rings); + fpga_dev->sw_ring_size = ring_size; + fpga_dev->sw_ring_max_depth = FPGA_RING_MAX_SIZE; + + /* Allocate memory for ring flush status */ + fpga_dev->flush_queue_status = rte_zmalloc_socket(NULL, + sizeof(uint64_t), RTE_CACHE_LINE_SIZE, socket_id); + if (fpga_dev->flush_queue_status == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u flush_queue_status", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + + /* Set the flush status address registers */ + phys_addr = rte_malloc_virt2iova(fpga_dev->flush_queue_status); + + address = FPGA_LTE_FEC_VFQ_FLUSH_STATUS_LW; + payload = (uint32_t)(phys_addr); + fpga_reg_write_32(fpga_dev->mmio_base, address, payload); + + address = FPGA_LTE_FEC_VFQ_FLUSH_STATUS_HI; + payload = (uint32_t)(phys_addr >> 32); + fpga_reg_write_32(fpga_dev->mmio_base, address, payload); + + return 0; +} + +static int +fpga_dev_close(struct rte_bbdev *dev) +{ + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + + rte_free(fpga_dev->sw_rings); + rte_free(fpga_dev->flush_queue_status); + + return 0; +} + +static void +fpga_dev_info_get(struct rte_bbdev *dev, + struct rte_bbdev_driver_info *dev_info) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + uint32_t q_id = 0; + + /* TODO RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN and numbers of buffers are set + * to temporary values as they are required by test application while + * validation phase. + */ + static const struct rte_bbdev_op_cap bbdev_capabilities[] = { + { + .type = RTE_BBDEV_OP_TURBO_DEC, + .cap.turbo_dec = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_TYPE_24B | + RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE | + RTE_BBDEV_TURBO_DEC_INTERRUPTS | + RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN | + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP, + .max_llr_modulus = INT8_MAX, + .num_buffers_src = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_hard_out = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_soft_out = 0 + } + }, + { + .type = RTE_BBDEV_OP_TURBO_ENC, + .cap.turbo_enc = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_24B_ATTACH | + RTE_BBDEV_TURBO_RATE_MATCH | + RTE_BBDEV_TURBO_ENC_INTERRUPTS, + .num_buffers_src = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_dst = + RTE_BBDEV_MAX_CODE_BLOCKS + } + }, + RTE_BBDEV_END_OF_CAPABILITIES_LIST() + }; + + static struct rte_bbdev_queue_conf default_queue_conf; + default_queue_conf.socket = dev->data->socket_id; + default_queue_conf.queue_size = FPGA_RING_MAX_SIZE; + + + dev_info->driver_name = dev->device->driver->name; + dev_info->queue_size_lim = FPGA_RING_MAX_SIZE; + dev_info->hardware_accelerated = true; + dev_info->min_alignment = 64; + dev_info->default_queue_conf = default_queue_conf; + dev_info->capabilities = bbdev_capabilities; + dev_info->cpu_flag_reqs = NULL; + + /* Calculates number of queues assigned to device */ + dev_info->max_num_queues = 0; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + uint32_t hw_q_id = fpga_reg_read_32(d->mmio_base, + FPGA_LTE_FEC_QUEUE_MAP + (q_id << 2)); + if (hw_q_id != FPGA_INVALID_HW_QUEUE_ID) + dev_info->max_num_queues++; + } +} + +/** + * Find index of queue bound to current PF/VF which is unassigned. Return -1 + * when there is no available queue + */ +static int +fpga_find_free_queue_idx(struct rte_bbdev *dev, + const struct rte_bbdev_queue_conf *conf) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + uint64_t q_idx; + uint8_t i = 0; + uint8_t range = FPGA_TOTAL_NUM_QUEUES >> 1; + + if (conf->op_type == RTE_BBDEV_OP_TURBO_ENC) { + i = FPGA_NUM_DL_QUEUES; + range = FPGA_TOTAL_NUM_QUEUES; + } + + for (; i < range; ++i) { + q_idx = 1ULL << i; + /* Check if index of queue is bound to current PF/VF */ + if (d->q_bound_bit_map & q_idx) + /* Check if found queue was not already assigned */ + if (!(d->q_assigned_bit_map & q_idx)) { + d->q_assigned_bit_map |= q_idx; + return i; + } + } + + rte_bbdev_log(INFO, "Failed to find free queue on %s", dev->data->name); + + return -1; +} + +static int +fpga_queue_setup(struct rte_bbdev *dev, uint16_t queue_id, + const struct rte_bbdev_queue_conf *conf) +{ + uint32_t address, ring_offset; + struct fpga_lte_fec_device *d = dev->data->dev_private; + struct fpga_queue *q; + int8_t q_idx; + + /* Check if there is a free queue to assign */ + q_idx = fpga_find_free_queue_idx(dev, conf); + if (q_idx == -1) + return -1; + + /* Allocate the queue data structure. */ + q = rte_zmalloc_socket(dev->device->driver->name, sizeof(*q), + RTE_CACHE_LINE_SIZE, conf->socket); + if (q == NULL) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_bbdev_log(ERR, "Failed to allocate queue memory"); + return -ENOMEM; + } + + q->d = d; + q->q_idx = q_idx; + + /* Set ring_base_addr */ + q->ring_addr = RTE_PTR_ADD(d->sw_rings, (d->sw_ring_size * queue_id)); + q->ring_ctrl_reg.ring_base_addr = d->sw_rings_phys + + (d->sw_ring_size * queue_id); + + /* Allocate memory for Completion Head variable*/ + q->ring_head_addr = rte_zmalloc_socket(dev->device->driver->name, + sizeof(uint64_t), RTE_CACHE_LINE_SIZE, conf->socket); + if (q->ring_head_addr == NULL) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_free(q); + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u completion_head", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + /* Set ring_head_addr */ + q->ring_ctrl_reg.ring_head_addr = + rte_malloc_virt2iova(q->ring_head_addr); + + /* Clear shadow_completion_head */ + q->shadow_completion_head = 0; + + /* Set ring_size */ + if (conf->queue_size > FPGA_RING_MAX_SIZE) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_free(q->ring_head_addr); + rte_free(q); + rte_bbdev_log(ERR, + "Size of queue is too big %d (MAX: %d ) for %s:%u", + conf->queue_size, FPGA_RING_MAX_SIZE, + dev->device->driver->name, dev->data->dev_id); + return -EINVAL; + } + q->ring_ctrl_reg.ring_size = conf->queue_size; + + /* Set Miscellaneous FPGA register*/ + /* Max iteration number for TTI mitigation - todo */ + q->ring_ctrl_reg.max_ul_dec = 0; + /* Enable max iteration number for TTI - todo */ + q->ring_ctrl_reg.max_ul_dec_en = 0; + + /* Enable the ring */ + q->ring_ctrl_reg.enable = 1; + + /* Set FPGA head_point and tail registers */ + q->ring_ctrl_reg.head_point = q->tail = 0; + + /* Set FPGA shadow_tail register */ + q->ring_ctrl_reg.shadow_tail = q->tail; + + /* Calculates the ring offset for found queue */ + ring_offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q_idx); + + /* Set FPGA Ring Control Registers */ + fpga_ring_reg_write(d->mmio_base, ring_offset, q->ring_ctrl_reg); + + /* Store MMIO register of shadow_tail */ + address = ring_offset + FPGA_LTE_FEC_RING_SHADOW_TAIL; + q->shadow_tail_addr = RTE_PTR_ADD(d->mmio_base, address); + + q->head_free_desc = q->tail; + + /* Set wrap mask */ + q->sw_ring_wrap_mask = conf->queue_size - 1; + + rte_bbdev_log_debug("Setup dev%u q%u: queue_idx=%u", + dev->data->dev_id, queue_id, q->q_idx); + + dev->data->queues[queue_id].queue_private = q; + + rte_bbdev_log_debug("BBDEV queue[%d] set up for FPGA queue[%d]", + queue_id, q_idx); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Read FPGA Ring Control Registers after configuration*/ + print_ring_reg_debug_info(d->mmio_base, ring_offset); +#endif + return 0; +} + +static int +fpga_queue_release(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + struct fpga_ring_ctrl_reg ring_reg; + uint32_t offset; + + rte_bbdev_log_debug("FPGA Queue[%d] released", queue_id); + + if (q != NULL) { + memset(&ring_reg, 0, sizeof(struct fpga_ring_ctrl_reg)); + offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + /* Disable queue */ + fpga_reg_write_8(d->mmio_base, + offset + FPGA_LTE_FEC_RING_ENABLE, 0x00); + /* Clear queue registers */ + fpga_ring_reg_write(d->mmio_base, offset, ring_reg); + + /* Mark the Queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q->q_idx)); + rte_free(q->ring_head_addr); + rte_free(q); + dev->data->queues[queue_id].queue_private = NULL; + } + + return 0; +} + +/* Function starts a device queue. */ +static int +fpga_queue_start(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (d == NULL) { + rte_bbdev_log(ERR, "Invalid device pointer"); + return -1; + } +#endif + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + uint32_t offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + uint8_t enable = 0x01; + uint16_t zero = 0x0000; + + /* Clear queue head and tail variables */ + q->tail = q->head_free_desc = 0; + + /* Clear FPGA head_point and tail registers */ + fpga_reg_write_16(d->mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_POINT, + zero); + fpga_reg_write_16(d->mmio_base, offset + FPGA_LTE_FEC_RING_SHADOW_TAIL, + zero); + + /* Enable queue */ + fpga_reg_write_8(d->mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + enable); + + rte_bbdev_log_debug("FPGA Queue[%d] started", queue_id); + return 0; +} + +/* Function stops a device queue. */ +static int +fpga_queue_stop(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (d == NULL) { + rte_bbdev_log(ERR, "Invalid device pointer"); + return -1; + } +#endif + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + uint32_t offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + uint8_t payload = 0x01; + uint8_t counter = 0; + uint8_t timeout = FPGA_QUEUE_FLUSH_TIMEOUT_US / + FPGA_TIMEOUT_CHECK_INTERVAL; + + /* Set flush_queue_en bit to trigger queue flushing */ + fpga_reg_write_8(d->mmio_base, + offset + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN, payload); + + /** Check if queue flush is completed. + * FPGA will update the completion flag after queue flushing is + * completed. If completion flag is not updated within 1ms it is + * considered as a failure. + */ + while (!(*((uint8_t *)d->flush_queue_status + q->q_idx) & payload)) { + if (counter > timeout) { + rte_bbdev_log(ERR, "FPGA Queue Flush failed for queue %d", + queue_id); + return -1; + } + usleep(FPGA_TIMEOUT_CHECK_INTERVAL); + counter++; + } + + /* Disable queue */ + payload = 0x00; + fpga_reg_write_8(d->mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + payload); + + rte_bbdev_log_debug("FPGA Queue[%d] stopped", queue_id); + return 0; +} + +static inline uint16_t +get_queue_id(struct rte_bbdev_data *data, uint8_t q_idx) +{ + uint16_t queue_id; + + for (queue_id = 0; queue_id < data->num_queues; ++queue_id) { + struct fpga_queue *q = data->queues[queue_id].queue_private; + if (q != NULL && q->q_idx == q_idx) + return queue_id; + } + + return -1; +} + +/* Interrupt handler triggered by FPGA dev for handling specific interrupt */ +static void +fpga_dev_interrupt_handler(void *cb_arg) +{ + struct rte_bbdev *dev = cb_arg; + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + struct fpga_queue *q; + uint64_t ring_head; + uint64_t q_idx; + uint16_t queue_id; + uint8_t i; + + /* Scan queue assigned to this device */ + for (i = 0; i < FPGA_TOTAL_NUM_QUEUES; ++i) { + q_idx = 1ULL << i; + if (fpga_dev->q_bound_bit_map & q_idx) { + queue_id = get_queue_id(dev->data, i); + if (queue_id == (uint16_t) -1) + continue; + + /* Check if completion head was changed */ + q = dev->data->queues[queue_id].queue_private; + ring_head = *q->ring_head_addr; + if (q->shadow_completion_head != ring_head && + q->irq_enable == 1) { + q->shadow_completion_head = ring_head; + rte_bbdev_pmd_callback_process( + dev, + RTE_BBDEV_EVENT_DEQUEUE, + &queue_id); + } + } + } +} + +static int +fpga_queue_intr_enable(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + + if (!rte_intr_cap_multiple(dev->intr_handle)) + return -ENOTSUP; + + q->irq_enable = 1; + + return 0; +} + +static int +fpga_queue_intr_disable(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + q->irq_enable = 0; + + return 0; +} + +static int +fpga_intr_enable(struct rte_bbdev *dev) +{ + int ret; + uint8_t i; + + if (!rte_intr_cap_multiple(dev->intr_handle)) { + rte_bbdev_log(ERR, "Multiple intr vector is not supported by FPGA (%s)", + dev->data->name); + return -ENOTSUP; + } + + /* Create event file descriptors for each of 64 queue. Event fds will be + * mapped to FPGA IRQs in rte_intr_enable(). This is a 1:1 mapping where + * the IRQ number is a direct translation to the queue number. + * + * 63 (FPGA_NUM_INTR_VEC) event fds are created as rte_intr_enable() + * mapped the first IRQ to already created interrupt event file + * descriptor (intr_handle->fd). + */ + if (rte_intr_efd_enable(dev->intr_handle, FPGA_NUM_INTR_VEC)) { + rte_bbdev_log(ERR, "Failed to create fds for %u queues", + dev->data->num_queues); + return -1; + } + + /* TODO Each event file descriptor is overwritten by interrupt event + * file descriptor. That descriptor is added to epoll observed list. + * It ensures that callback function assigned to that descriptor will + * invoked when any FPGA queue issues interrupt. + */ + for (i = 0; i < FPGA_NUM_INTR_VEC; ++i) + dev->intr_handle->efds[i] = dev->intr_handle->fd; + + if (!dev->intr_handle->intr_vec) { + dev->intr_handle->intr_vec = rte_zmalloc("intr_vec", + dev->data->num_queues * sizeof(int), 0); + if (!dev->intr_handle->intr_vec) { + rte_bbdev_log(ERR, "Failed to allocate %u vectors", + dev->data->num_queues); + return -ENOMEM; + } + } + + ret = rte_intr_enable(dev->intr_handle); + if (ret < 0) { + rte_bbdev_log(ERR, + "Couldn't enable interrupts for device: %s", + dev->data->name); + return ret; + } + + ret = rte_intr_callback_register(dev->intr_handle, + fpga_dev_interrupt_handler, dev); + if (ret < 0) { + rte_bbdev_log(ERR, + "Couldn't register interrupt callback for device: %s", + dev->data->name); + return ret; + } + + return 0; +} + +static const struct rte_bbdev_ops fpga_ops = { + .setup_queues = fpga_setup_queues, + .intr_enable = fpga_intr_enable, + .close = fpga_dev_close, + .info_get = fpga_dev_info_get, + .queue_setup = fpga_queue_setup, + .queue_stop = fpga_queue_stop, + .queue_start = fpga_queue_start, + .queue_release = fpga_queue_release, + .queue_intr_enable = fpga_queue_intr_enable, + .queue_intr_disable = fpga_queue_intr_disable +}; + +static inline void +fpga_dma_enqueue(struct fpga_queue *q, uint16_t num_desc, + struct rte_bbdev_stats *queue_stats) +{ +#ifdef RTE_BBDEV_OFFLOAD_COST + uint64_t start_time = 0; + queue_stats->acc_offload_cycles = 0; +#else + RTE_SET_USED(queue_stats); +#endif + + /* Update tail and shadow_tail register */ + q->tail = (q->tail + num_desc) & q->sw_ring_wrap_mask; + + rte_wmb(); + +#ifdef RTE_BBDEV_OFFLOAD_COST + /* Start time measurement for enqueue function offload. */ + start_time = rte_rdtsc_precise(); +#endif + mmio_write_16(q->shadow_tail_addr, q->tail); + +#ifdef RTE_BBDEV_OFFLOAD_COST + rte_wmb(); + queue_stats->acc_offload_cycles += rte_rdtsc_precise() - start_time; +#endif +} + +/* Calculates number of CBs in processed encoder TB based on 'r' and input + * length. + */ +static inline uint8_t +get_num_cbs_in_op_enc(struct rte_bbdev_op_turbo_enc *turbo_enc) +{ + uint8_t c, c_neg, r, crc24_bits = 0; + uint16_t k, k_neg, k_pos; + uint8_t cbs_in_op = 0; + int32_t length; + + length = turbo_enc->input.length; + r = turbo_enc->tb_params.r; + c = turbo_enc->tb_params.c; + c_neg = turbo_enc->tb_params.c_neg; + k_neg = turbo_enc->tb_params.k_neg; + k_pos = turbo_enc->tb_params.k_pos; + crc24_bits = 24; + while (length > 0 && r < c) { + k = (r < c_neg) ? k_neg : k_pos; + length -= (k - crc24_bits) >> 3; + r++; + cbs_in_op++; + } + + return cbs_in_op; +} + +/* Calculates number of CBs in processed decoder TB based on 'r' and input + * length. + */ +static inline uint16_t +get_num_cbs_in_op_dec(struct rte_bbdev_op_turbo_dec *turbo_dec) +{ + uint8_t c, c_neg, r = 0; + uint16_t kw, k, k_neg, k_pos, cbs_in_op = 0; + int32_t length; + + length = turbo_dec->input.length; + r = turbo_dec->tb_params.r; + c = turbo_dec->tb_params.c; + c_neg = turbo_dec->tb_params.c_neg; + k_neg = turbo_dec->tb_params.k_neg; + k_pos = turbo_dec->tb_params.k_pos; + while (length > 0 && r < c) { + k = (r < c_neg) ? k_neg : k_pos; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + length -= kw; + r++; + cbs_in_op++; + } + + return cbs_in_op; +} + +/* Read flag value 0/1/ from bitmap */ +static inline bool +check_bit(uint32_t bitmap, uint32_t bitmask) +{ + return bitmap & bitmask; +} + +/* Print an error if a descriptor error has occurred. + * Return 0 on success, 1 on failure + */ +static inline int +check_desc_error(uint32_t error_code) { + switch (error_code) { + case DESC_ERR_NO_ERR: + return 0; + case DESC_ERR_K_OUT_OF_RANGE: + rte_bbdev_log(ERR, "Block_size_k is out of range (k<40 or k>6144)"); + break; + case DESC_ERR_K_NOT_NORMAL: + rte_bbdev_log(ERR, "Block_size_k is not a normal value within normal range"); + break; + case DESC_ERR_KPAI_NOT_NORMAL: + rte_bbdev_log(ERR, "Three_kpai is not a normal value for UL only"); + break; + case DESC_ERR_DESC_OFFSET_ERR: + rte_bbdev_log(ERR, "Queue offset does not meet the expectation in the FPGA"); + break; + case (DESC_ERR_K_OUT_OF_RANGE | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Block_size_k is out of range (k<40 or k>6144) and queue offset error"); + break; + case (DESC_ERR_K_NOT_NORMAL | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Block_size_k is not a normal value within normal range and queue offset error"); + break; + case (DESC_ERR_KPAI_NOT_NORMAL | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Three_kpai is not a normal value for UL only and queue offset error"); + break; + case DESC_ERR_DESC_READ_FAIL: + rte_bbdev_log(ERR, "Unsuccessful completion for descriptor read"); + break; + case DESC_ERR_DESC_READ_TIMEOUT: + rte_bbdev_log(ERR, "Descriptor read time-out"); + break; + case DESC_ERR_DESC_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Descriptor read TLP poisoned"); + break; + case DESC_ERR_CB_READ_FAIL: + rte_bbdev_log(ERR, "Unsuccessful completion for code block"); + break; + case DESC_ERR_CB_READ_TIMEOUT: + rte_bbdev_log(ERR, "Code block read time-out"); + break; + case DESC_ERR_CB_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Code block read TLP poisoned"); + break; + default: + rte_bbdev_log(ERR, "Descriptor error unknown error code %u", + error_code); + break; + } + return 1; +} + +/** + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param k + * K value (length of input in bits). + * @param e + * E value (length of output in bits). + * @param ncb + * Ncb value (size of the soft buffer). + * @param out_length + * Length of output buffer + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +fpga_dma_desc_te_fill(struct rte_bbdev_enc_op *op, + struct fpga_dma_enc_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t k, uint16_t e, uint16_t ncb, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) + +{ + /* reset */ + desc->done = 0; + desc->crc_en = check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_CRC_24B_ATTACH); + desc->bypass_rm = !check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_RATE_MATCH); + desc->k = k; + desc->e = e; + desc->ncb = ncb; + desc->rv = op->turbo_enc.rv_index; + desc->offset = desc_offset; + /* Set inbound data buffer address */ + desc->in_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset) >> 32); + desc->in_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset)); + + desc->out_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset) >> 32); + desc->out_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset)); + + /* Save software context needed for dequeue */ + desc->op_addr = op; + + /* Set total number of CBs in an op */ + desc->cbs_in_op = cbs_in_op; + + return 0; +} + +/** + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param in_length + * Length of an input. + * @param k + * K value (length of an output in bits). + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +fpga_dma_desc_td_fill(struct rte_bbdev_dec_op *op, + struct fpga_dma_dec_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t in_length, uint16_t k, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) +{ + /* reset */ + desc->done = 0; + /* Set inbound data buffer address */ + desc->in_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset) >> 32); + desc->in_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset)); + desc->in_len = in_length; + desc->k = k; + desc->crc_type = !check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_CRC_TYPE_24B); + if ((op->turbo_dec.code_block_mode == 0) + && !check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP)) + desc->drop_crc = 1; + desc->max_iter = op->turbo_dec.iter_max * 2; + desc->offset = desc_offset; + desc->out_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset) >> 32); + desc->out_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset)); + + /* Save software context needed for dequeue */ + desc->op_addr = op; + + /* Set total number of CBs in an op */ + desc->cbs_in_op = cbs_in_op; + + return 0; +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Validates turbo encoder parameters */ +static int +validate_enc_op(struct rte_bbdev_enc_op *op) +{ + struct rte_bbdev_op_turbo_enc *turbo_enc = &op->turbo_enc; + struct rte_bbdev_op_enc_cb_params *cb = NULL; + struct rte_bbdev_op_enc_tb_params *tb = NULL; + uint16_t kw, kw_neg, kw_pos; + + if (turbo_enc->input.length > + RTE_BBDEV_MAX_TB_SIZE >> 3) { + rte_bbdev_log(ERR, "TB size (%u) is too big, max: %d", + turbo_enc->input.length, RTE_BBDEV_MAX_TB_SIZE); + op->status = 1 << RTE_BBDEV_DATA_ERROR; + return -1; + } + + if (op->mempool == NULL) { + rte_bbdev_log(ERR, "Invalid mempool pointer"); + return -1; + } + if (turbo_enc->input.data == NULL) { + rte_bbdev_log(ERR, "Invalid input pointer"); + return -1; + } + if (turbo_enc->output.data == NULL) { + rte_bbdev_log(ERR, "Invalid output pointer"); + return -1; + } + if (turbo_enc->rv_index > 3) { + rte_bbdev_log(ERR, + "rv_index (%u) is out of range 0 <= value <= 3", + turbo_enc->rv_index); + return -1; + } + if (turbo_enc->code_block_mode != 0 && + turbo_enc->code_block_mode != 1) { + rte_bbdev_log(ERR, + "code_block_mode (%u) is out of range 0 <= value <= 1", + turbo_enc->code_block_mode); + return -1; + } + + if (turbo_enc->code_block_mode == 0) { + tb = &turbo_enc->tb_params; + if ((tb->k_neg < RTE_BBDEV_MIN_CB_SIZE + || tb->k_neg > RTE_BBDEV_MAX_CB_SIZE) + && tb->c_neg > 0) { + rte_bbdev_log(ERR, + "k_neg (%u) is out of range %u <= value <= %u", + tb->k_neg, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->k_pos < RTE_BBDEV_MIN_CB_SIZE + || tb->k_pos > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k_pos (%u) is out of range %u <= value <= %u", + tb->k_pos, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->c_neg > (RTE_BBDEV_MAX_CODE_BLOCKS - 1)) + rte_bbdev_log(ERR, + "c_neg (%u) is out of range 0 <= value <= %u", + tb->c_neg, + RTE_BBDEV_MAX_CODE_BLOCKS - 1); + if (tb->c < 1 || tb->c > RTE_BBDEV_MAX_CODE_BLOCKS) { + rte_bbdev_log(ERR, + "c (%u) is out of range 1 <= value <= %u", + tb->c, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + if (tb->cab > tb->c) { + rte_bbdev_log(ERR, + "cab (%u) is greater than c (%u)", + tb->cab, tb->c); + return -1; + } + if ((tb->ea < RTE_BBDEV_MIN_CB_SIZE || (tb->ea % 2)) + && tb->r < tb->cab) { + rte_bbdev_log(ERR, + "ea (%u) is less than %u or it is not even", + tb->ea, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + if ((tb->eb < RTE_BBDEV_MIN_CB_SIZE || (tb->eb % 2)) + && tb->c > tb->cab) { + rte_bbdev_log(ERR, + "eb (%u) is less than %u or it is not even", + tb->eb, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + + kw_neg = 3 * RTE_ALIGN_CEIL(tb->k_neg + 4, + RTE_BBDEV_C_SUBBLOCK); + if (tb->ncb_neg < tb->k_neg || tb->ncb_neg > kw_neg) { + rte_bbdev_log(ERR, + "ncb_neg (%u) is out of range (%u) k_neg <= value <= (%u) kw_neg", + tb->ncb_neg, tb->k_neg, kw_neg); + return -1; + } + + kw_pos = 3 * RTE_ALIGN_CEIL(tb->k_pos + 4, + RTE_BBDEV_C_SUBBLOCK); + if (tb->ncb_pos < tb->k_pos || tb->ncb_pos > kw_pos) { + rte_bbdev_log(ERR, + "ncb_pos (%u) is out of range (%u) k_pos <= value <= (%u) kw_pos", + tb->ncb_pos, tb->k_pos, kw_pos); + return -1; + } + if (tb->r > (tb->c - 1)) { + rte_bbdev_log(ERR, + "r (%u) is greater than c - 1 (%u)", + tb->r, tb->c - 1); + return -1; + } + } else { + cb = &turbo_enc->cb_params; + if (cb->k < RTE_BBDEV_MIN_CB_SIZE + || cb->k > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k (%u) is out of range %u <= value <= %u", + cb->k, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + + if (cb->e < RTE_BBDEV_MIN_CB_SIZE || (cb->e % 2)) { + rte_bbdev_log(ERR, + "e (%u) is less than %u or it is not even", + cb->e, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + + kw = RTE_ALIGN_CEIL(cb->k + 4, RTE_BBDEV_C_SUBBLOCK) * 3; + if (cb->ncb < cb->k || cb->ncb > kw) { + rte_bbdev_log(ERR, + "ncb (%u) is out of range (%u) k <= value <= (%u) kw", + cb->ncb, cb->k, kw); + return -1; + } + } + + return 0; +} +#endif + +static inline char * +mbuf_append(struct rte_mbuf *m_head, struct rte_mbuf *m, uint16_t len) +{ + if (unlikely(len > rte_pktmbuf_tailroom(m))) + return NULL; + + char *tail = (char *)m->buf_addr + m->data_off + m->data_len; + m->data_len = (uint16_t)(m->data_len + len); + m_head->pkt_len = (m_head->pkt_len + len); + return tail; +} + +static inline int +enqueue_enc_one_op_cb(struct fpga_queue *q, struct rte_bbdev_enc_op *op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input; + struct rte_mbuf *output; + int ret; + uint16_t k, e, ncb, ring_offset; + uint32_t total_left, in_length, out_length, in_offset, out_offset; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_enc_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo encoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_enc.input.data; + output = op->turbo_enc.output.data; + in_offset = op->turbo_enc.input.offset; + out_offset = op->turbo_enc.output.offset; + total_left = op->turbo_enc.input.length; + k = op->turbo_enc.cb_params.k; + e = op->turbo_enc.cb_params.e; + ncb = op->turbo_enc.cb_params.ncb; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_CRC_24B_ATTACH)) + in_length = ((k - 24) >> 3); + else + in_length = k >> 3; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_RATE_MATCH)) + out_length = (e + 7) >> 3; + else + out_length = (k >> 3) * 3 + 2; + + mbuf_append(output, output, out_length); + + /* Offset into the ring */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + /* Setup DMA Descriptor */ + desc = q->ring_addr + ring_offset; + + ret = fpga_dma_desc_te_fill(op, &desc->enc_req, input, output, k, e, + ncb, in_offset, out_offset, ring_offset, 1); + if (unlikely(ret < 0)) + return ret; + + /* Update lengths */ + total_left -= in_length; + op->turbo_enc.output.length += out_length; + + if (total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + total_left, in_length); + return -1; + } + + return 1; +} + +static inline int +enqueue_enc_one_op_tb(struct fpga_queue *q, struct rte_bbdev_enc_op *op, + uint16_t desc_offset, uint8_t cbs_in_op) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input, *output_head, *output; + int ret; + uint8_t r, c, crc24_bits = 0; + uint16_t k, e, ncb, ring_offset; + uint32_t mbuf_total_left, in_length, out_length, in_offset, out_offset; + uint32_t seg_total_left; + uint16_t current_enqueued_cbs = 0; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_enc_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo encoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_enc.input.data; + output_head = output = op->turbo_enc.output.data; + in_offset = op->turbo_enc.input.offset; + out_offset = op->turbo_enc.output.offset; + mbuf_total_left = op->turbo_enc.input.length; + + c = op->turbo_enc.tb_params.c; + r = op->turbo_enc.tb_params.r; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_CRC_24B_ATTACH)) + crc24_bits = 24; + + while (mbuf_total_left > 0 && r < c && input != NULL) { + seg_total_left = rte_pktmbuf_data_len(input) - in_offset; + + e = (r < op->turbo_enc.tb_params.cab) ? + op->turbo_enc.tb_params.ea : + op->turbo_enc.tb_params.eb; + k = (r < op->turbo_enc.tb_params.c_neg) ? + op->turbo_enc.tb_params.k_neg : + op->turbo_enc.tb_params.k_pos; + ncb = (r < op->turbo_enc.tb_params.c_neg) ? + op->turbo_enc.tb_params.ncb_neg : + op->turbo_enc.tb_params.ncb_pos; + + in_length = ((k - crc24_bits) >> 3); + + if (check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_RATE_MATCH)) + out_length = (e + 7) >> 3; + else + out_length = (k >> 3) * 3 + 2; + + mbuf_append(output_head, output, out_length); + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_te_fill(op, &desc->enc_req, input, output, + k, e, ncb, in_offset, out_offset, ring_offset, + cbs_in_op); + if (unlikely(ret < 0)) + return ret; + + rte_bbdev_log_debug("DMA request desc %p", desc); + + /* Update lengths */ + op->turbo_enc.output.length += out_length; + mbuf_total_left -= in_length; + + /* Update offsets */ + if (seg_total_left == in_length) { + /* Go to the next mbuf */ + input = input->next; + output = output->next; + in_offset = 0; + out_offset = 0; + } else { + in_offset += in_length; + out_offset += out_length; + } + + r++; + desc_offset++; + current_enqueued_cbs++; + } + + if (mbuf_total_left > 0) { + rte_bbdev_log(ERR, + "Some date still left for processing: mbuf_total_left = %u", + mbuf_total_left); + return -1; + } + + return current_enqueued_cbs; +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Validates turbo decoder parameters */ +static int +validate_dec_op(struct rte_bbdev_dec_op *op) +{ + struct rte_bbdev_op_turbo_dec *turbo_dec = &op->turbo_dec; + struct rte_bbdev_op_dec_cb_params *cb = NULL; + struct rte_bbdev_op_dec_tb_params *tb = NULL; + + if (op->mempool == NULL) { + rte_bbdev_log(ERR, "Invalid mempool pointer"); + return -1; + } + if (turbo_dec->input.data == NULL) { + rte_bbdev_log(ERR, "Invalid input pointer"); + return -1; + } + if (turbo_dec->hard_output.data == NULL) { + rte_bbdev_log(ERR, "Invalid hard_output pointer"); + return -1; + } + if (turbo_dec->rv_index > 3) { + rte_bbdev_log(ERR, + "rv_index (%u) is out of range 0 <= value <= 3", + turbo_dec->rv_index); + return -1; + } + if (turbo_dec->iter_min < 1) { + rte_bbdev_log(ERR, + "iter_min (%u) is less than 1", + turbo_dec->iter_min); + return -1; + } + if (turbo_dec->iter_max <= 2) { + rte_bbdev_log(ERR, + "iter_max (%u) is less than or equal to 2", + turbo_dec->iter_max); + return -1; + } + if (turbo_dec->iter_min > turbo_dec->iter_max) { + rte_bbdev_log(ERR, + "iter_min (%u) is greater than iter_max (%u)", + turbo_dec->iter_min, turbo_dec->iter_max); + return -1; + } + if (turbo_dec->code_block_mode != 0 && + turbo_dec->code_block_mode != 1) { + rte_bbdev_log(ERR, + "code_block_mode (%u) is out of range 0 <= value <= 1", + turbo_dec->code_block_mode); + return -1; + } + + if (turbo_dec->code_block_mode == 0) { + + if ((turbo_dec->op_flags & + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP) && + !(turbo_dec->op_flags & RTE_BBDEV_TURBO_CRC_TYPE_24B)) { + rte_bbdev_log(ERR, + "RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP should accompany RTE_BBDEV_TURBO_CRC_TYPE_24B"); + return -1; + } + + tb = &turbo_dec->tb_params; + if ((tb->k_neg < RTE_BBDEV_MIN_CB_SIZE + || tb->k_neg > RTE_BBDEV_MAX_CB_SIZE) + && tb->c_neg > 0) { + rte_bbdev_log(ERR, + "k_neg (%u) is out of range %u <= value <= %u", + tb->k_neg, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if ((tb->k_pos < RTE_BBDEV_MIN_CB_SIZE + || tb->k_pos > RTE_BBDEV_MAX_CB_SIZE) + && tb->c > tb->c_neg) { + rte_bbdev_log(ERR, + "k_pos (%u) is out of range %u <= value <= %u", + tb->k_pos, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->c_neg > (RTE_BBDEV_MAX_CODE_BLOCKS - 1)) + rte_bbdev_log(ERR, + "c_neg (%u) is out of range 0 <= value <= %u", + tb->c_neg, + RTE_BBDEV_MAX_CODE_BLOCKS - 1); + if (tb->c < 1 || tb->c > RTE_BBDEV_MAX_CODE_BLOCKS) { + rte_bbdev_log(ERR, + "c (%u) is out of range 1 <= value <= %u", + tb->c, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + if (tb->cab > tb->c) { + rte_bbdev_log(ERR, + "cab (%u) is greater than c (%u)", + tb->cab, tb->c); + return -1; + } + } else { + + if (turbo_dec->op_flags & RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP) { + rte_bbdev_log(ERR, + "RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP is invalid in CB-mode"); + return -1; + } + + cb = &turbo_dec->cb_params; + if (cb->k < RTE_BBDEV_MIN_CB_SIZE + || cb->k > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k (%u) is out of range %u <= value <= %u", + cb->k, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + } + + return 0; +} +#endif + +static inline int +enqueue_dec_one_op_cb(struct fpga_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input; + struct rte_mbuf *output; + int ret; + uint16_t k, kw, ring_offset; + uint32_t total_left, in_length, out_length, in_offset, out_offset; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_dec_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo decoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_dec.input.data; + output = op->turbo_dec.hard_output.data; + total_left = op->turbo_dec.input.length; + in_offset = op->turbo_dec.input.offset; + out_offset = op->turbo_dec.hard_output.offset; + + k = op->turbo_dec.cb_params.k; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + in_length = kw; + out_length = k >> 3; + + mbuf_append(output, output, out_length); + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_td_fill(op, &desc->dec_req, input, output, + in_length, k, in_offset, out_offset, ring_offset, 1); + if (unlikely(ret < 0)) + return ret; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_dma_dec_desc_debug_info(desc); +#endif + + /* Update lengths */ + total_left -= in_length; + op->turbo_dec.hard_output.length += out_length; + + if (total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + total_left, in_length); + return -1; + } + + return 1; +} + + +static inline int +enqueue_dec_one_op_tb(struct fpga_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset, uint8_t cbs_in_op) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input, *output_head, *output; + int ret; + uint8_t r, c; + uint16_t k, kw, in_length, out_length, ring_offset; + uint32_t mbuf_total_left, seg_total_left, in_offset, out_offset; + uint16_t current_enqueued_cbs = 0; + uint16_t crc24_overlap = 0; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_dec_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo decoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_dec.input.data; + output_head = output = op->turbo_dec.hard_output.data; + mbuf_total_left = op->turbo_dec.input.length; + in_offset = op->turbo_dec.input.offset; + out_offset = op->turbo_dec.hard_output.offset; + + if (!check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP)) + crc24_overlap = 24; + + c = op->turbo_dec.tb_params.c; + r = op->turbo_dec.tb_params.r; + + while (mbuf_total_left > 0 && r < c && input != NULL) { + seg_total_left = rte_pktmbuf_data_len(input) - in_offset; + k = (r < op->turbo_dec.tb_params.c_neg) ? + op->turbo_dec.tb_params.k_neg : + op->turbo_dec.tb_params.k_pos; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + + in_length = kw; + out_length = (k - crc24_overlap) >> 3; + + mbuf_append(output_head, output, out_length); + + if (seg_total_left < in_length) { + rte_bbdev_log(ERR, + "Partial CB found in a TB. FPGA Driver doesn't support scatter-gather operations!"); + return -1; + } + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_td_fill(op, &desc->dec_req, input, output, + in_length, k, in_offset, out_offset, + ring_offset, cbs_in_op); + if (unlikely(ret < 0)) + return ret; + + /* Update lengths */ + ret = rte_pktmbuf_trim(op->turbo_dec.hard_output.data, + (crc24_overlap >> 3)); +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (ret < 0) { + rte_bbdev_log(ERR, + "The length to remove is greater than the length of the last segment"); + return -EINVAL; + } +#endif + op->turbo_dec.hard_output.length += out_length; + mbuf_total_left -= in_length; + + /* Update offsets */ + if (seg_total_left == in_length) { + /* Go to the next mbuf */ + input = input->next; + output = output->next; + in_offset = 0; + out_offset = 0; + } else { + in_offset += in_length; + out_offset += out_length; + } + + r++; + desc_offset++; + current_enqueued_cbs++; + } + + if (mbuf_total_left > 0) { + rte_bbdev_log(ERR, + "Some date still left for processing: mbuf_total_left = %u", + mbuf_total_left); + return -1; + } + + return current_enqueued_cbs; +} + +static uint16_t +fpga_enqueue_enc(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num) +{ + uint8_t cbs_in_op; + uint16_t i, total_enqueued_cbs = 0; + int32_t avail; + int enqueued_cbs; + struct fpga_queue *q = q_data->queue_private; + union fpga_dma_desc *desc; + + /* Check if queue is not full */ + if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == + q->head_free_desc)) + return 0; + + /* Calculates available space */ + avail = (q->head_free_desc > q->tail) ? + q->head_free_desc - q->tail - 1 : + q->ring_ctrl_reg.ring_size + q->head_free_desc - q->tail - 1; + + for (i = 0; i < num; ++i) { + if (ops[i]->turbo_enc.code_block_mode == 0) { + cbs_in_op = get_num_cbs_in_op_enc(&ops[i]->turbo_enc); + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - cbs_in_op < 0)) + break; + avail -= cbs_in_op; + enqueued_cbs = enqueue_enc_one_op_tb(q, ops[i], + total_enqueued_cbs, cbs_in_op); + } else { + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - 1 < 0)) + break; + avail -= 1; + enqueued_cbs = enqueue_enc_one_op_cb(q, ops[i], + total_enqueued_cbs); + } + + if (enqueued_cbs < 0) + break; + + total_enqueued_cbs += enqueued_cbs; + + rte_bbdev_log_debug("enqueuing enc ops [%d/%d] | head %d | tail %d", + total_enqueued_cbs, num, + q->head_free_desc, q->tail); + } + + /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt + * only when all previous CBs were already processed. + */ + desc = q->ring_addr + ((q->tail + total_enqueued_cbs - 1) + & q->sw_ring_wrap_mask); + desc->enc_req.irq_en = q->irq_enable; + + fpga_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); + + /* Update stats */ + q_data->queue_stats.enqueued_count += i; + q_data->queue_stats.enqueue_err_count += num - i; + + return i; +} + +static uint16_t +fpga_enqueue_dec(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num) +{ + uint8_t cbs_in_op; + uint16_t i, total_enqueued_cbs = 0; + int32_t avail; + int enqueued_cbs; + struct fpga_queue *q = q_data->queue_private; + union fpga_dma_desc *desc; + + /* Check if queue is not full */ + if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == + q->head_free_desc)) + return 0; + + /* Calculates available space */ + avail = (q->head_free_desc > q->tail) ? + q->head_free_desc - q->tail - 1 : + q->ring_ctrl_reg.ring_size + q->head_free_desc - q->tail - 1; + + for (i = 0; i < num; ++i) { + if (ops[i]->turbo_dec.code_block_mode == 0) { + cbs_in_op = get_num_cbs_in_op_dec(&ops[i]->turbo_dec); + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - cbs_in_op < 0)) + break; + avail -= cbs_in_op; + enqueued_cbs = enqueue_dec_one_op_tb(q, ops[i], + total_enqueued_cbs, cbs_in_op); + } else { + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - 1 < 0)) + break; + avail -= 1; + enqueued_cbs = enqueue_dec_one_op_cb(q, ops[i], + total_enqueued_cbs); + } + + if (enqueued_cbs < 0) + break; + + total_enqueued_cbs += enqueued_cbs; + + rte_bbdev_log_debug("enqueuing dec ops [%d/%d] | head %d | tail %d", + total_enqueued_cbs, num, + q->head_free_desc, q->tail); + } + + /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt + * only when all previous CBs were already processed. + */ + desc = q->ring_addr + ((q->tail + total_enqueued_cbs - 1) + & q->sw_ring_wrap_mask); + desc->dec_req.irq_en = q->irq_enable; + + fpga_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); + + /* Update stats */ + q_data->queue_stats.enqueued_count += i; + q_data->queue_stats.enqueue_err_count += num - i; + + return i; +} + +static inline int +dequeue_enc_one_op_cb(struct fpga_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + int desc_error = 0; + + /* Set current desc */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /*check if done */ + if (desc->enc_req.done == 0) + return -1; + + /* make sure the response is read atomically */ + rte_smp_rmb(); + + rte_bbdev_log_debug("DMA response desc %p", desc); + + *op = desc->enc_req.op_addr; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + (*op)->status = desc_error << RTE_BBDEV_DATA_ERROR; + + return 1; +} + +static inline int +dequeue_enc_one_op_tb(struct fpga_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + uint8_t cbs_in_op, cb_idx; + int desc_error = 0; + int status = 0; + + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify if done bit is set */ + if (desc->enc_req.done == 0) + return -1; + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + /* Verify if done bit in all CBs is set */ + cbs_in_op = desc->enc_req.cbs_in_op; + for (cb_idx = 1; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + if (desc->enc_req.done == 0) + return -1; + } + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + for (cb_idx = 0; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + status |= desc_error << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log_debug("DMA response desc %p", desc); + } + + *op = desc->enc_req.op_addr; + (*op)->status = status; + return cbs_in_op; +} + +static inline int +dequeue_dec_one_op_cb(struct fpga_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + int desc_error = 0; + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify done bit is set */ + if (desc->dec_req.done == 0) + return -1; + + /* make sure the response is read atomically */ + rte_smp_rmb(); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_dma_dec_desc_debug_info(desc); + +#endif + + *op = desc->dec_req.op_addr; + /* FPGA reports in half-iterations, from 0 to 31. get ceiling */ + (*op)->turbo_dec.iter_count = (desc->dec_req.iter + 2) >> 1; + /* crc_pass = 0 when decoder fails */ + (*op)->status = !(desc->dec_req.crc_pass) << RTE_BBDEV_CRC_ERROR; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + (*op)->status |= desc_error << RTE_BBDEV_DATA_ERROR; + return 1; +} + +static inline int +dequeue_dec_one_op_tb(struct fpga_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + uint8_t cbs_in_op, cb_idx, iter_count = 0; + int status = 0; + int desc_error = 0; + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify if done bit is set */ + if (desc->dec_req.done == 0) + return -1; + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + /* Verify if done bit in all CBs is set */ + cbs_in_op = desc->dec_req.cbs_in_op; + for (cb_idx = 1; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + if (desc->dec_req.done == 0) + return -1; + } + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + for (cb_idx = 0; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + /* get max iter_count for all CBs in op */ + iter_count = RTE_MAX(iter_count, (uint8_t) desc->dec_req.iter); + /* crc_pass = 0 when decoder fails, one fails all */ + status |= !(desc->dec_req.crc_pass) << RTE_BBDEV_CRC_ERROR; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + status |= desc_error << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log_debug("DMA response desc %p", desc); + } + + *op = desc->dec_req.op_addr; + + /* FPGA reports in half-iterations, get ceiling */ + (*op)->turbo_dec.iter_count = (iter_count + 2) >> 1; + (*op)->status = status; + return cbs_in_op; +} + +static uint16_t +fpga_dequeue_enc(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num) +{ + struct fpga_queue *q = q_data->queue_private; + uint32_t avail = (q->tail - q->head_free_desc) & q->sw_ring_wrap_mask; + uint16_t i; + uint16_t dequeued_cbs = 0; + struct rte_bbdev_enc_op *op; + int ret; + + for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { + op = (q->ring_addr + ((q->head_free_desc + dequeued_cbs) + & q->sw_ring_wrap_mask))->enc_req.op_addr; + if (op->turbo_enc.code_block_mode == 0) + ret = dequeue_enc_one_op_tb(q, &ops[i], dequeued_cbs); + else + ret = dequeue_enc_one_op_cb(q, &ops[i], dequeued_cbs); + + if (ret < 0) + break; + + dequeued_cbs += ret; + + rte_bbdev_log_debug("dequeuing enc ops [%d/%d] | head %d | tail %d", + dequeued_cbs, num, q->head_free_desc, q->tail); + } + + /* Update head */ + q->head_free_desc = (q->head_free_desc + dequeued_cbs) & + q->sw_ring_wrap_mask; + + /* Update stats */ + q_data->queue_stats.dequeued_count += i; + + return i; +} + +static uint16_t +fpga_dequeue_dec(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num) +{ + struct fpga_queue *q = q_data->queue_private; + uint32_t avail = (q->tail - q->head_free_desc) & q->sw_ring_wrap_mask; + uint16_t i; + uint16_t dequeued_cbs = 0; + struct rte_bbdev_dec_op *op; + int ret; + + for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { + op = (q->ring_addr + ((q->head_free_desc + dequeued_cbs) + & q->sw_ring_wrap_mask))->dec_req.op_addr; + if (op->turbo_dec.code_block_mode == 0) + ret = dequeue_dec_one_op_tb(q, &ops[i], dequeued_cbs); + else + ret = dequeue_dec_one_op_cb(q, &ops[i], dequeued_cbs); + + if (ret < 0) + break; + + dequeued_cbs += ret; + + rte_bbdev_log_debug("dequeuing dec ops [%d/%d] | head %d | tail %d", + dequeued_cbs, num, q->head_free_desc, q->tail); + } + + /* Update head */ + q->head_free_desc = (q->head_free_desc + dequeued_cbs) & + q->sw_ring_wrap_mask; + + /* Update stats */ + q_data->queue_stats.dequeued_count += i; + + return i; +} + +/* Initialization Function */ +static void +fpga_lte_fec_init(struct rte_bbdev *dev) +{ + struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev->device); + + dev->dev_ops = &fpga_ops; + dev->enqueue_enc_ops = fpga_enqueue_enc; + dev->enqueue_dec_ops = fpga_enqueue_dec; + dev->dequeue_enc_ops = fpga_dequeue_enc; + dev->dequeue_dec_ops = fpga_dequeue_dec; + + ((struct fpga_lte_fec_device *) dev->data->dev_private)->pf_device = + !strcmp(dev->device->driver->name, + RTE_STR(FPGA_LTE_FEC_PF_DRIVER_NAME)); + ((struct fpga_lte_fec_device *) dev->data->dev_private)->mmio_base = + pci_dev->mem_resource[0].addr; + + rte_bbdev_log_debug( + "Init device %s [%s] @ virtaddr %p phyaddr %#"PRIx64, + dev->device->driver->name, dev->data->name, + (void *)pci_dev->mem_resource[0].addr, + pci_dev->mem_resource[0].phys_addr); +} + +static int +fpga_lte_fec_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct rte_bbdev *bbdev = NULL; + char dev_name[RTE_BBDEV_NAME_MAX_LEN]; + + if (pci_dev == NULL) { + rte_bbdev_log(ERR, "NULL PCI device"); + return -EINVAL; + } + + rte_pci_device_name(&pci_dev->addr, dev_name, sizeof(dev_name)); + + /* Allocate memory to be used privately by drivers */ + bbdev = rte_bbdev_allocate(pci_dev->device.name); + if (bbdev == NULL) + return -ENODEV; + + /* allocate device private memory */ + bbdev->data->dev_private = rte_zmalloc_socket(dev_name, + sizeof(struct fpga_lte_fec_device), RTE_CACHE_LINE_SIZE, + pci_dev->device.numa_node); + + if (bbdev->data->dev_private == NULL) { + rte_bbdev_log(CRIT, + "Allocate of %zu bytes for device \"%s\" failed", + sizeof(struct fpga_lte_fec_device), dev_name); + rte_bbdev_release(bbdev); + return -ENOMEM; + } + + /* Fill HW specific part of device structure */ + bbdev->device = &pci_dev->device; + bbdev->intr_handle = &pci_dev->intr_handle; + bbdev->data->socket_id = pci_dev->device.numa_node; + + /* Invoke FEC FPGA device initialization function */ + fpga_lte_fec_init(bbdev); + + rte_bbdev_log_debug("bbdev id = %u [%s]", + bbdev->data->dev_id, dev_name); + + struct fpga_lte_fec_device *d = bbdev->data->dev_private; + uint32_t version_id = fpga_reg_read_32(d->mmio_base, + FPGA_LTE_FEC_VERSION_ID); + rte_bbdev_log(INFO, "FEC FPGA RTL v%u.%u", + ((uint16_t)(version_id >> 16)), ((uint16_t)version_id)); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (!strcmp(bbdev->device->driver->name, + RTE_STR(FPGA_LTE_FEC_PF_DRIVER_NAME))) + print_static_reg_debug_info(d->mmio_base); +#endif + return 0; +} + +static int +fpga_lte_fec_remove(struct rte_pci_device *pci_dev) +{ + struct rte_bbdev *bbdev; + int ret; + uint8_t dev_id; + + if (pci_dev == NULL) + return -EINVAL; + + /* Find device */ + bbdev = rte_bbdev_get_named_dev(pci_dev->device.name); + if (bbdev == NULL) { + rte_bbdev_log(CRIT, + "Couldn't find HW dev \"%s\" to uninitialise it", + pci_dev->device.name); + return -ENODEV; + } + dev_id = bbdev->data->dev_id; + + /* free device private memory before close */ + rte_free(bbdev->data->dev_private); + + /* Close device */ + ret = rte_bbdev_close(dev_id); + if (ret < 0) + rte_bbdev_log(ERR, + "Device %i failed to close during uninit: %i", + dev_id, ret); + + /* release bbdev from library */ + ret = rte_bbdev_release(bbdev); + if (ret) + rte_bbdev_log(ERR, "Device %i failed to uninit: %i", dev_id, + ret); + + rte_bbdev_log_debug("Destroyed bbdev = %u", dev_id); + + return 0; +} + +static inline void +set_default_fpga_conf(struct fpga_lte_fec_conf *def_conf) +{ + /* clear default configuration before initialization */ + memset(def_conf, 0, sizeof(struct fpga_lte_fec_conf)); + /* Set pf mode to true */ + def_conf->pf_mode_en = true; + + /* Set ratio between UL and DL to 1:1 (unit of weight is 3 CBs) */ + def_conf->ul_bandwidth = 3; + def_conf->dl_bandwidth = 3; + + /* Set Load Balance Factor to 64 */ + def_conf->dl_load_balance = 64; + def_conf->ul_load_balance = 64; +} + +/* Initial configuration of FPGA LTE FEC device */ +int +fpga_lte_fec_configure(const char *dev_name, + const struct fpga_lte_fec_conf *conf) +{ + uint32_t payload_32, address; + uint16_t payload_16; + uint8_t payload_8; + uint16_t q_id, vf_id, total_q_id, total_ul_q_id, total_dl_q_id; + struct rte_bbdev *bbdev = rte_bbdev_get_named_dev(dev_name); + struct fpga_lte_fec_conf def_conf; + + if (bbdev == NULL) { + rte_bbdev_log(ERR, + "Invalid dev_name (%s), or device is not yet initialised", + dev_name); + return -ENODEV; + } + + struct fpga_lte_fec_device *d = bbdev->data->dev_private; + + if (conf == NULL) { + rte_bbdev_log(ERR, + "FPGA Configuration was not provided. Default configuration will be loaded."); + set_default_fpga_conf(&def_conf); + conf = &def_conf; + } + + /* + * Configure UL:DL ratio. + * [7:0]: UL weight + * [15:8]: DL weight + */ + payload_16 = (conf->dl_bandwidth << 8) | conf->ul_bandwidth; + address = FPGA_LTE_FEC_CONFIGURATION; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Clear all queues registers */ + payload_32 = FPGA_INVALID_HW_QUEUE_ID; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + address = (q_id << 2) + FPGA_LTE_FEC_QUEUE_MAP; + fpga_reg_write_32(d->mmio_base, address, payload_32); + } + + /* + * If PF mode is enabled allocate all queues for PF only. + * + * For VF mode each VF can have different number of UL and DL queues. + * Total number of queues to configure cannot exceed FPGA + * capabilities - 64 queues - 32 queues for UL and 32 queues for DL. + * Queues mapping is done according to configuration: + * + * UL queues: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | ... | 0 | + * | conf->vf_dl_queues_number[0] - 1 | 0 | + * | conf->vf_dl_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_dl_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_dl_queues_number[7] - 1 | 7 | + * + * DL queues: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | ... | 0 | + * | conf->vf_ul_queues_number[0] - 1 | 0 | + * | conf->vf_ul_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_ul_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_ul_queues_number[7] - 1 | 7 | + * + * Example of configuration: + * conf->vf_ul_queues_number[0] = 4; -> 4 UL queues for VF0 + * conf->vf_dl_queues_number[0] = 4; -> 4 DL queues for VF0 + * conf->vf_ul_queues_number[1] = 2; -> 2 UL queues for VF1 + * conf->vf_dl_queues_number[1] = 2; -> 2 DL queues for VF1 + * + * UL: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | 1 | 0 | + * | 2 | 0 | + * | 3 | 0 | + * | 4 | 1 | + * | 5 | 1 | + * + * DL: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | 33 | 0 | + * | 34 | 0 | + * | 35 | 0 | + * | 36 | 1 | + * | 37 | 1 | + */ + if (conf->pf_mode_en) { + payload_32 = 0x1; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + address = (q_id << 2) + FPGA_LTE_FEC_QUEUE_MAP; + fpga_reg_write_32(d->mmio_base, address, payload_32); + } + } else { + /* Calculate total number of UL and DL queues to configure */ + total_ul_q_id = total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + total_ul_q_id += conf->vf_ul_queues_number[vf_id]; + total_dl_q_id += conf->vf_dl_queues_number[vf_id]; + } + total_q_id = total_dl_q_id + total_ul_q_id; + /* + * Check if total number of queues to configure does not exceed + * FPGA capabilities (64 queues - 32 UL and 32 DL queues) + */ + if ((total_ul_q_id > FPGA_NUM_UL_QUEUES) || + (total_dl_q_id > FPGA_NUM_DL_QUEUES) || + (total_q_id > FPGA_TOTAL_NUM_QUEUES)) { + rte_bbdev_log(ERR, + "FPGA Configuration failed. Too many queues to configure: UL_Q %u, DL_Q %u, FPGA_Q %u", + total_ul_q_id, total_dl_q_id, + FPGA_TOTAL_NUM_QUEUES); + return -EINVAL; + } + total_ul_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_ul_queues_number[vf_id]; + ++q_id, ++total_ul_q_id) { + address = (total_ul_q_id << 2) + + FPGA_LTE_FEC_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_reg_write_32(d->mmio_base, address, + payload_32); + } + } + total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_dl_queues_number[vf_id]; + ++q_id, ++total_dl_q_id) { + address = ((total_dl_q_id + FPGA_NUM_UL_QUEUES) + << 2) + FPGA_LTE_FEC_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_reg_write_32(d->mmio_base, address, + payload_32); + } + } + } + + /* Setting Load Balance Factor */ + payload_16 = (conf->dl_load_balance << 8) | (conf->ul_load_balance); + address = FPGA_LTE_FEC_LOAD_BALANCE_FACTOR; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting length of ring descriptor entry */ + payload_16 = FPGA_RING_DESC_ENTRY_LENGTH; + address = FPGA_LTE_FEC_RING_DESC_LEN; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting FLR timeout value */ + payload_16 = conf->flr_time_out; + address = FPGA_LTE_FEC_FLR_TIME_OUT; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Queue PF/VF mapping table is ready */ + payload_8 = 0x1; + address = FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE; + fpga_reg_write_8(d->mmio_base, address, payload_8); + + rte_bbdev_log_debug("PF FPGA LTE FEC configuration complete for %s", + dev_name); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_static_reg_debug_info(d->mmio_base); +#endif + return 0; +} + +/* FPGA LTE FEC PCI PF address map */ +static struct rte_pci_id pci_id_fpga_lte_fec_pf_map[] = { + { + RTE_PCI_DEVICE(FPGA_LTE_FEC_VENDOR_ID, + FPGA_LTE_FEC_PF_DEVICE_ID) + }, + {.device_id = 0}, +}; + +static struct rte_pci_driver fpga_lte_fec_pci_pf_driver = { + .probe = fpga_lte_fec_probe, + .remove = fpga_lte_fec_remove, + .id_table = pci_id_fpga_lte_fec_pf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING +}; + +/* FPGA LTE FEC PCI VF address map */ +static struct rte_pci_id pci_id_fpga_lte_fec_vf_map[] = { + { + RTE_PCI_DEVICE(FPGA_LTE_FEC_VENDOR_ID, + FPGA_LTE_FEC_VF_DEVICE_ID) + }, + {.device_id = 0}, +}; + +static struct rte_pci_driver fpga_lte_fec_pci_vf_driver = { + .probe = fpga_lte_fec_probe, + .remove = fpga_lte_fec_remove, + .id_table = pci_id_fpga_lte_fec_vf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING +}; + + +RTE_PMD_REGISTER_PCI(FPGA_LTE_FEC_PF_DRIVER_NAME, fpga_lte_fec_pci_pf_driver); +RTE_PMD_REGISTER_PCI_TABLE(FPGA_LTE_FEC_PF_DRIVER_NAME, + pci_id_fpga_lte_fec_pf_map); +RTE_PMD_REGISTER_PCI(FPGA_LTE_FEC_VF_DRIVER_NAME, fpga_lte_fec_pci_vf_driver); +RTE_PMD_REGISTER_PCI_TABLE(FPGA_LTE_FEC_VF_DRIVER_NAME, + pci_id_fpga_lte_fec_vf_map); + +RTE_INIT(fpga_lte_fec_init_log) +{ + fpga_lte_fec_logtype = rte_log_register("pmd.bb.fpga_lte_fec"); + if (fpga_lte_fec_logtype >= 0) +#ifdef RTE_LIBRTE_BBDEV_DEBUG + rte_log_set_level(fpga_lte_fec_logtype, RTE_LOG_DEBUG); +#else + rte_log_set_level(fpga_lte_fec_logtype, RTE_LOG_NOTICE); +#endif +} diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h new file mode 100644 index 0000000..2c6952e --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2018 Intel Corporation + */ + +#ifndef _FPGA_LTE_FEC_H_ +#define _FPGA_LTE_FEC_H_ + +#include <stdint.h> +#include <stdbool.h> + +/** + * @file fpga_lte_fec.h + * + * Interface for Intel(R) FGPA LTE FEC device configuration at the host level, + * directly accessible by the application. + * Configuration related to LTE Turbo coding functionality is done through + * librte_bbdev library. + * + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + */ + +#ifdef __cplusplus +extern "C" { +#endif + +/**< Number of Virtual Functions FGPA 4G FEC supports */ +#define FPGA_LTE_FEC_NUM_VFS 8 + +/** + * Structure to pass FPGA 4G FEC configuration. + */ +struct fpga_lte_fec_conf { + /**< 1 if PF is used for dataplane, 0 for VFs */ + bool pf_mode_en; + /**< Number of UL queues per VF */ + uint8_t vf_ul_queues_number[FPGA_LTE_FEC_NUM_VFS]; + /**< Number of DL queues per VF */ + uint8_t vf_dl_queues_number[FPGA_LTE_FEC_NUM_VFS]; + /**< UL bandwidth. Needed for schedule algorithm */ + uint8_t ul_bandwidth; + /**< DL bandwidth. Needed for schedule algorithm */ + uint8_t dl_bandwidth; + /**< UL Load Balance */ + uint8_t ul_load_balance; + /**< DL Load Balance */ + uint8_t dl_load_balance; + /**< FLR timeout value */ + uint16_t flr_time_out; +}; + +/** + * Configure Intel(R) FPGA LTE FEC device + * + * @param dev_name + * The name of the device. This is the short form of PCI BDF, e.g. 00:01.0. + * It can also be retrieved for a bbdev device from the dev_name field in the + * rte_bbdev_info structure returned by rte_bbdev_info_get(). + * @param conf + * Configuration to apply to FPGA 4G FEC. + * + * @return + * Zero on success, negative value on failure. + */ +int +fpga_lte_fec_configure(const char *dev_name, + const struct fpga_lte_fec_conf *conf); + +#ifdef __cplusplus +} +#endif + +#endif /* _FPGA_LTE_FEC_H_ */ diff --git a/drivers/baseband/fpga_lte_fec/meson.build b/drivers/baseband/fpga_lte_fec/meson.build new file mode 100644 index 0000000..bf44e6b --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +deps += ['bbdev', 'bus_vdev', 'ring', 'pci', 'bus_pci'] +name = 'bbdev_fpga_lte_fec' +allow_experimental_apis = true +sources = files('fpga_lte_fec.c') diff --git a/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map new file mode 100644 index 0000000..1a99f8d --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map @@ -0,0 +1,3 @@ +DPDK_18.08 { + local: *; +}; diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build index 52489df..a17d549 100644 --- a/drivers/baseband/meson.build +++ b/drivers/baseband/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <bluca@debian.org> -drivers = ['null'] +drivers = ['null', 'fpga_lte_fec'] config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' driver_name_fmt = 'rte_pmd_@0@' diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 7c9b4b5..21d32a2 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -218,6 +218,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_NETVSC_PMD) += -lrte_pmd_netvsc ifeq ($(CONFIG_RTE_LIBRTE_BBDEV),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += -lrte_pmd_fpga_lte_fec # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
[parent not found: <EEA9FF629BF25B47BD67ADE995041EE2496A888B@IRSMSX103.ger.corp.intel.com>]
* Re: [dpdk-dev] [PATCH v4] baseband/fpga_lte_fec: adding driver for FEC on FPGA [not found] ` <EEA9FF629BF25B47BD67ADE995041EE2496A888B@IRSMSX103.ger.corp.intel.com> @ 2019-06-13 17:05 ` Chautru, Nicolas 0 siblings, 0 replies; 62+ messages in thread From: Chautru, Nicolas @ 2019-06-13 17:05 UTC (permalink / raw) To: Chalupnik, KamilX, thomas, akhil.goyal, dev; +Cc: Yigit, Ferruh, Mokhtar, Amr >-----Original Message----- >From: Chalupnik, KamilX >Sent: Wednesday, June 12, 2019 9:03 AM >To: Chautru, Nicolas <nicolas.chautru@intel.com>; thomas@monjalon.net; akhil.goyal@nxp.com; dev@dpdk.org >Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; Mokhtar, Amr <amr.mokhtar@intel.com>; Chautru, Nicolas <nicolas.chautru@intel.com> >Subject: RE: [dpdk-dev] [PATCH v4] baseband/fpga_lte_fec: adding driver for FEC on FPGA > >Hi Nic, > > >> -----Original Message----- >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Nicolas Chautru >> Sent: Monday, June 10, 2019 7:01 PM >> To: thomas@monjalon.net; akhil.goyal@nxp.com; dev@dpdk.org >> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; Mokhtar, Amr >> <amr.mokhtar@intel.com>; Chautru, Nicolas <nicolas.chautru@intel.com> >> Subject: [dpdk-dev] [PATCH v4] baseband/fpga_lte_fec: adding driver >> for FEC on FPGA >> >> Supports for FEC 4G PMD Driver on FPGA card PAC N3000 >> >> Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> >> --- >> config/common_base | 6 + >> doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++ >> doc/guides/bbdevs/index.rst | 1 + >> drivers/baseband/Makefile | 2 + >> drivers/baseband/fpga_lte_fec/Makefile | 29 + >> drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 >> ++++++++++++++++++++ >> drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + >> drivers/baseband/fpga_lte_fec/meson.build | 7 + >> .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + >> drivers/baseband/meson.build | 2 +- >> mk/rte.app.mk | 1 + >> 11 files changed, 3115 insertions(+), 1 deletion(-) create mode >> 100644 doc/guides/bbdevs/fpga_lte_fec.rst >> create mode 100644 drivers/baseband/fpga_lte_fec/Makefile >> create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c >> create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h >> create mode 100644 drivers/baseband/fpga_lte_fec/meson.build >> create mode 100644 >> drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map > > >Generally code looks ok. Just few comments. >In all new files please change "Copyright(c) 2018" to "2019" > Thanks for the catch. Will update now. > > >> diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c >... >> +static inline int >> +enqueue_enc_one_op_cb(struct fpga_queue *q, struct >> rte_bbdev_enc_op *op, >> + uint16_t desc_offset) >> +{ >... >> + if (check_bit(op->turbo_enc.op_flags, >> RTE_BBDEV_TURBO_RATE_MATCH)) >> + out_length = (e + 7) >> 3; >> + else >> + out_length = (k >> 3) * 3 + 2; >> + >> + mbuf_append(output, output, out_length); >> + > >Value returned by mbuf_append should be check. I guess we can put that check under RTE_LIBRTE_BBDEV_DEBUG flag. >Please apply this to all places where mbuf_append is used. > We discussed in parallel that the driver is not meant to catch all user errors condition even in debug mode. We agreed that this was not required but thanks for the thorough review. > >> diff --git >> a/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.m >> ap >> b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.m >> ap >> +DPDK_18.08 { >> + local: *; >> +}; > >Shouldn't be DPDK_19.08? > You are right. Time flies since that driver started! > >Best regards, >Kamil > Take care, Thanks, Nic ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v5] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-10 17:01 ` Nicolas Chautru [not found] ` <EEA9FF629BF25B47BD67ADE995041EE2496A888B@IRSMSX103.ger.corp.intel.com> @ 2019-06-13 17:28 ` Nicolas Chautru 2019-06-13 17:28 ` Nicolas Chautru 2019-06-14 16:12 ` [dpdk-dev] [PATCH v6] " Nicolas Chautru 2019-06-14 16:17 ` [dpdk-dev] [PATCH v6] " Nicolas Chautru 3 siblings, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-13 17:28 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, Nicolas Chautru Update v5: Update date and version from dpdk review. Rebased to latest. Update v4: Fix warning for the DEBUG configuration. Update v3: Squashing 3 previous patches into one as recommended. This is adding a new PMD driver for BBDEV device based on FPGA implementation on PAC N3000 HW to provide FEC 4G acceleration. v1 was shared earlier on this patchwork : https://patches.dpdk.org/patch/53409/ The existing BBDEV test framework is still supported. Main updates based on feedback during the last v1 were: - correct the documentation which was arguably misleading in hinting that ICC was required. There is no dependency whatsoever. - add meson build support Note that a number of other BBDEV patches are in v1 here and will be resubmitted for 19.08 rc1 https://patches.dpdk.org/project/dpdk/list/?series=4657 Nicolas Chautru (1): baseband/fpga_lte_fec: adding driver for FEC on FPGA config/common_base | 6 + doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++ doc/guides/bbdevs/index.rst | 1 + drivers/baseband/Makefile | 2 + drivers/baseband/fpga_lte_fec/Makefile | 29 + drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 ++++++++++++++++++++ drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + drivers/baseband/fpga_lte_fec/meson.build | 7 + .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + drivers/baseband/meson.build | 2 +- mk/rte.app.mk | 1 + 11 files changed, 3115 insertions(+), 1 deletion(-) create mode 100644 doc/guides/bbdevs/fpga_lte_fec.rst create mode 100644 drivers/baseband/fpga_lte_fec/Makefile create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h create mode 100644 drivers/baseband/fpga_lte_fec/meson.build create mode 100644 drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v5] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-13 17:28 ` [dpdk-dev] [PATCH v5] " Nicolas Chautru @ 2019-06-13 17:28 ` Nicolas Chautru 0 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-13 17:28 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, Nicolas Chautru Supports for FEC 4G PMD Driver on FPGA card PAC N3000 Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 6 + doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++ doc/guides/bbdevs/index.rst | 1 + drivers/baseband/Makefile | 2 + drivers/baseband/fpga_lte_fec/Makefile | 29 + drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 ++++++++++++++++++++ drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + drivers/baseband/fpga_lte_fec/meson.build | 7 + .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + drivers/baseband/meson.build | 2 +- mk/rte.app.mk | 1 + 11 files changed, 3115 insertions(+), 1 deletion(-) create mode 100644 doc/guides/bbdevs/fpga_lte_fec.rst create mode 100644 drivers/baseband/fpga_lte_fec/Makefile create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h create mode 100644 drivers/baseband/fpga_lte_fec/meson.build create mode 100644 drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map diff --git a/config/common_base b/config/common_base index 6f19ad5..9632a07 100644 --- a/config/common_base +++ b/config/common_base @@ -521,6 +521,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y # EXPERIMENTAL: API may change without prior notice # CONFIG_RTE_LIBRTE_BBDEV=y +CONFIG_RTE_LIBRTE_BBDEV_DEBUG=n CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y @@ -535,6 +536,11 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n # +# Compile PMD for Intel FPGA LTE FEC bbdev device +# +CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC=y + +# # Compile generic crypto device library # CONFIG_RTE_LIBRTE_CRYPTODEV=y diff --git a/doc/guides/bbdevs/fpga_lte_fec.rst b/doc/guides/bbdevs/fpga_lte_fec.rst new file mode 100644 index 0000000..71b058c --- /dev/null +++ b/doc/guides/bbdevs/fpga_lte_fec.rst @@ -0,0 +1,318 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation + +Intel(R) FPGA LTE FEC Poll Mode Driver +====================================== + +The BBDEV FPGA LTE FEC poll mode driver (PMD) supports an FPGA implementation of a VRAN +Turbo Encode / Decode LTE wireless acceleration function, using Intel's PCI-e and FPGA +based Vista Creek device. + +Features +-------- + +FPGA LTE FEC PMD supports the following features: + +- Turbo Encode in the DL with total throughput of 4.5 Gbits/s +- Turbo Decode in the UL with total throughput of 1.5 Gbits/s assuming 8 decoder iterations +- 8 VFs per PF (physical device) +- Maximum of 32 UL queues per VF +- Maximum of 32 DL queues per VF +- PCIe Gen-3 x8 Interface +- MSI-X +- SR-IOV + + +FPGA LTE FEC PMD supports the following BBDEV capabilities: + +* For the turbo encode operation: + - ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` : set to attach CRC24B to CB(s) + - ``RTE_BBDEV_TURBO_RATE_MATCH`` : if set then do not do Rate Match bypass + - ``RTE_BBDEV_TURBO_ENC_INTERRUPTS`` : set for encoder dequeue interrupts + + +* For the turbo decode operation: + - ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` : check CRC24B from CB(s) + - ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` : perform subblock de-interleave + - ``RTE_BBDEV_TURBO_DEC_INTERRUPTS`` : set for decoder dequeue interrupts + - ``RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN`` : set if negative LLR encoder i/p is supported + - ``RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP`` : keep CRC24B bits appended while decoding + + +Limitations +----------- + +FPGA LTE FEC does not support the following: + +- Scatter-Gather function + + +Installation +-------------- + +Section 3 of the DPDK manual provides instuctions on installing and compiling DPDK. The +default set of bbdev compile flags may be found in config/common_base, where for example +the flag to build the FPGA LTE FEC device, ``CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC``, is already +set. It is assumed DPDK has been compiled using for instance: + +.. code-block:: console + + make install T=x86_64-native-linuxapp-gcc + + +DPDK requires hugepages to be configured as detailed in section 2 of the DPDK manual. +The bbdev test application has been tested with a configuration 40 x 1GB hugepages. The +hugepage configuration of a server may be examined using: + +.. code-block:: console + + grep Huge* /proc/meminfo + + +Initialization +-------------- + +When the device first powers up, its PCI Physical Functions (PF) can be listed through this command: + +.. code-block:: console + + sudo lspci -vd1172:5052 + +The physical and virtual functions are compatible with Linux UIO drivers: +``vfio`` and ``igb_uio``. However, in order to work the FPGA LTE FEC device firstly needs +to be bound to one of these linux drivers through DPDK. + + +Bind PF UIO driver(s) +~~~~~~~~~~~~~~~~~~~~~ + +Install the DPDK igb_uio driver, bind it with the PF PCI device ID and use +``lspci`` to confirm the PF device is under use by ``igb_uio`` DPDK UIO driver. + +The igb_uio driver may be bound to the PF PCI device using one of three methods: + + +1. PCI functions (physical or virtual, depending on the use case) can be bound to +the UIO driver by repeating this command for every function. + +.. code-block:: console + + cd <dpdk-top-level-directory> + insmod ./build/kmod/igb_uio.ko + echo "1172 5052" > /sys/bus/pci/drivers/igb_uio/new_id + lspci -vd1172: + + +2. Another way to bind PF with DPDK UIO driver is by using the ``dpdk-devbind.py`` tool + +.. code-block:: console + + cd <dpdk-top-level-directory> + ./usertools/dpdk-devbind.py -b igb_uio 0000:06:00.0 + +where the PCI device ID (example: 0000:06:00.0) is obtained using lspci -vd1172: + + +3. A third way to bind is to use ``dpdk-setup.sh`` tool + +.. code-block:: console + + cd <dpdk-top-level-directory> + ./usertools/dpdk-setup.sh + + select 'Bind Ethernet/Crypto/Baseband device to IGB UIO module' + or + select 'Bind Ethernet/Crypto/Baseband device to VFIO module' depending on driver required + enter PCI device ID + select 'Display current Ethernet/Crypto/Baseband device settings' to confirm binding + + +In the same way the FPGA LTE FEC PF can be bound with vfio, but vfio driver does not +support SR-IOV configuration right out of the box, so it will need to be patched. + + +Enable Virtual Functions +~~~~~~~~~~~~~~~~~~~~~~~~ + +Now, it should be visible in the printouts that PCI PF is under igb_uio control +"``Kernel driver in use: igb_uio``" + +To show the number of available VFs on the device, read ``sriov_totalvfs`` file.. + +.. code-block:: console + + cat /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_totalvfs + + where 0000\:<b>\:<d>.<f> is the PCI device ID + + +To enable VFs via igb_uio, echo the number of virtual functions intended to +enable to ``max_vfs`` file.. + +.. code-block:: console + + echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/max_vfs + + +Afterwards, all VFs must be bound to appropriate UIO drivers as required, same +way it was done with the physical function previously. + +Enabling SR-IOV via vfio driver is pretty much the same, except that the file +name is different: + +.. code-block:: console + + echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_numvfs + + +Configure the VFs through PF +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The PCI virtual functions must be configured before working or getting assigned +to VMs/Containers. The configuration involves allocating the number of hardware +queues, priorities, load balance, bandwidth and other settings necessary for the +device to perform FEC functions. + +This configuration needs to be executed at least once after reboot or PCI FLR and can +be achieved by using the function ``fpga_lte_fec_configure()``, which sets up the +parameters defined in ``fpga_lte_fec_conf`` structure: + +.. code-block:: c + + struct fpga_lte_fec_conf { + bool pf_mode_en; + uint8_t vf_ul_queues_number[FPGA_LTE_FEC_NUM_VFS]; + uint8_t vf_dl_queues_number[FPGA_LTE_FEC_NUM_VFS]; + uint8_t ul_bandwidth; + uint8_t dl_bandwidth; + uint8_t ul_load_balance; + uint8_t dl_load_balance; + uint16_t flr_time_out; + }; + +- ``pf_mode_en``: identifies whether only PF is to be used, or the VFs. PF and + VFs are mutually exclusive and cannot run simultaneously. + Set to 1 for PF mode enabled. + If PF mode is enabled all queues available in the device are assigned + exclusively to PF and 0 queues given to VFs. + +- ``vf_*l_queues_number``: defines the hardware queue mapping for every VF. + +- ``*l_bandwidth``: in case of congestion on PCIe interface. The device + allocates different bandwidth to UL and DL. The weight is configured by this + setting. The unit of weight is 3 code blocks. For example, if the code block + cbps (code block per second) ratio between UL and DL is 12:1, then the + configuration value should be set to 36:3. The schedule algorithm is based + on code block regardless the length of each block. + +- ``*l_load_balance``: hardware queues are load-balanced in a round-robin + fashion. Queues get filled first-in first-out until they reach a pre-defined + watermark level, if exceeded, they won't get assigned new code blocks.. + This watermark is defined by this setting. + + If all hardware queues exceeds the watermark, no code blocks will be + streamed in from UL/DL code block FIFO. + +- ``flr_time_out``: specifies how many 16.384us to be FLR time out. The + time_out = flr_time_out x 16.384us. For instance, if you want to set 10ms for + the FLR time out then set this setting to 0x262=610. + + +An example configuration code calling the function ``fpga_lte_fec_configure()`` is shown +below: + +.. code-block:: c + + struct fpga_lte_fec_conf conf; + unsigned int i; + + memset(&conf, 0, sizeof(struct fpga_lte_fec_conf)); + conf.pf_mode_en = 1; + + for (i = 0; i < FPGA_LTE_FEC_NUM_VFS; ++i) { + conf.vf_ul_queues_number[i] = 4; + conf.vf_dl_queues_number[i] = 4; + } + conf.ul_bandwidth = 12; + conf.dl_bandwidth = 5; + conf.dl_load_balance = 64; + conf.ul_load_balance = 64; + + /* setup FPGA PF */ + ret = fpga_lte_fec_configure(info->dev_name, &conf); + TEST_ASSERT_SUCCESS(ret, + "Failed to configure 4G FPGA PF for bbdev %s", + info->dev_name); + + +Test Application +---------------- + +BBDEV provides a test application, ``test-bbdev.py`` and range of test data for testing +the functionality of FPGA LTE FEC turbo encode and turbo decode, depending on the device's +capabilities. The test application is located under app->test-bbdev folder and has the +following options: + +.. code-block:: console + + "-p", "--testapp-path": specifies path to the bbdev test app. + "-e", "--eal-params" : EAL arguments which are passed to the test app. + "-t", "--timeout" : Timeout in seconds (default=300). + "-c", "--test-cases" : Defines test cases to run. Run all if not specified. + "-v", "--test-vector" : Test vector path (default=dpdk_path+/app/test-bbdev/test_vectors/bbdev_null.data). + "-n", "--num-ops" : Number of operations to process on device (default=32). + "-b", "--burst-size" : Operations enqueue/dequeue burst size (default=32). + "-l", "--num-lcores" : Number of lcores to run (default=16). + "-i", "--init-device" : Initialise PF device with default values. + + +To execute the test application tool using simple turbo decode or turbo encode data, +type one of the following: + +.. code-block:: console + + ./test-bbdev.py -c validation -n 64 -b 8 -v ./turbo_dec_default.data + ./test-bbdev.py -c validation -n 64 -b 8 -v ./turbo_enc_default.data + + +The test application ``test-bbdev.py``, supports the ability to configure the PF device with +a default set of values, if the "-i" or "- -init-device" option is included. The default values +are defined in test_bbdev_perf.c as: + +- VF_UL_QUEUE_VALUE 4 +- VF_DL_QUEUE_VALUE 4 +- UL_BANDWIDTH 3 +- DL_BANDWIDTH 3 +- UL_LOAD_BALANCE 128 +- DL_LOAD_BALANCE 128 +- FLR_TIMEOUT 610 + + +Test Vectors +~~~~~~~~~~~~ + +In addition to the simple turbo decoder and turbo encoder tests, bbdev also provides +a range of additional tests under the test_vectors folder, which may be useful. The results +of these tests will depend on the FPGA LTE FEC capabilities: + +* turbo decoder tests: + - ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_high_snr.data`` + - ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_low_snr.data`` + - ``turbo_dec_c1_k6144_r0_e34560_negllr.data`` + - ``turbo_dec_c1_k6144_r0_e34560_sbd_negllr.data`` + - ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr_crc24b.data`` + - ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr.data`` + + +* turbo encoder tests: + - ``turbo_enc_c1_k40_r0_e1190_rm.data`` + - ``turbo_enc_c1_k40_r0_e1194_rm.data`` + - ``turbo_enc_c1_k40_r0_e1196_rm.data`` + - ``turbo_enc_c1_k40_r0_e272_rm.data`` + - ``turbo_enc_c1_k6144_r0_e18444.data`` + - ``turbo_enc_c1_k6144_r0_e32256_crc24b_rm.data`` + - ``turbo_enc_c2_k5952_r0_e17868_crc24b.data`` + - ``turbo_enc_c3_k4800_r2_e14412_crc24b.data`` + - ``turbo_enc_c4_k4800_r2_e14412_crc24b.data`` + + diff --git a/doc/guides/bbdevs/index.rst b/doc/guides/bbdevs/index.rst index 93276ed..005b95e 100644 --- a/doc/guides/bbdevs/index.rst +++ b/doc/guides/bbdevs/index.rst @@ -10,3 +10,4 @@ Baseband Device Drivers null turbo_sw + fpga_lte_fec diff --git a/drivers/baseband/Makefile b/drivers/baseband/Makefile index 4ec83b0..ceffc7d 100644 --- a/drivers/baseband/Makefile +++ b/drivers/baseband/Makefile @@ -10,5 +10,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += null DEPDIRS-null = $(core-libs) DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += turbo_sw DEPDIRS-turbo_sw = $(core-libs) +DIRS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += fpga_lte_fec +DEPDIRS-fpga_lte_fec = $(core-libs) include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/drivers/baseband/fpga_lte_fec/Makefile b/drivers/baseband/fpga_lte_fec/Makefile new file mode 100644 index 0000000..a38a396 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/Makefile @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_pmd_fpga_lte_fec.a + +# build flags +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring +LDLIBS += -lrte_bbdev +LDLIBS += -lrte_pci -lrte_bus_pci + +# versioning export map +EXPORT_MAP := rte_pmd_bbdev_fpga_lte_fec_version.map + +# library version +LIBABIVER := 1 + +# library source files +SRCS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += fpga_lte_fec.c + +# export include files +SYMLINK-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC)-include += fpga_lte_fec.h + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c new file mode 100644 index 0000000..19e7689 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c @@ -0,0 +1,2674 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation + */ + +#include <unistd.h> + +#include <rte_common.h> +#include <rte_log.h> +#include <rte_dev.h> +#include <rte_malloc.h> +#include <rte_mempool.h> +#include <rte_errno.h> +#include <rte_pci.h> +#include <rte_bus_pci.h> +#include <rte_byteorder.h> +#ifdef RTE_BBDEV_OFFLOAD_COST +#include <rte_cycles.h> +#endif + +#include <rte_bbdev.h> +#include <rte_bbdev_pmd.h> + +#include "fpga_lte_fec.h" + +/* Turbo SW PMD logging ID */ +static int fpga_lte_fec_logtype; + +/* Helper macro for logging */ +#define rte_bbdev_log(level, fmt, ...) \ + rte_log(RTE_LOG_ ## level, fpga_lte_fec_logtype, fmt "\n", \ + ##__VA_ARGS__) + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +#define rte_bbdev_log_debug(fmt, ...) \ + rte_bbdev_log(DEBUG, "fpga_lte_fec: " fmt, \ + ##__VA_ARGS__) +#else +#define rte_bbdev_log_debug(fmt, ...) +#endif + +/* FPGA LTE FEC driver names */ +#define FPGA_LTE_FEC_PF_DRIVER_NAME intel_fpga_lte_fec_pf +#define FPGA_LTE_FEC_VF_DRIVER_NAME intel_fpga_lte_fec_vf + +/* FPGA LTE FEC PCI vendor & device IDs */ +#define FPGA_LTE_FEC_VENDOR_ID (0x1172) +#define FPGA_LTE_FEC_PF_DEVICE_ID (0x5052) +#define FPGA_LTE_FEC_VF_DEVICE_ID (0x5050) + +/* Align DMA descriptors to 256 bytes - cache-aligned */ +#define FPGA_RING_DESC_ENTRY_LENGTH (8) +/* Ring size is in 256 bits (32 bytes) units */ +#define FPGA_RING_DESC_LEN_UNIT_BYTES (32) +/* Maximum size of queue */ +#define FPGA_RING_MAX_SIZE (1024) +#define FPGA_FLR_TIMEOUT_UNIT (16.384) + +#define FPGA_NUM_UL_QUEUES (32) +#define FPGA_NUM_DL_QUEUES (32) +#define FPGA_TOTAL_NUM_QUEUES (FPGA_NUM_UL_QUEUES + FPGA_NUM_DL_QUEUES) +#define FPGA_NUM_INTR_VEC (FPGA_TOTAL_NUM_QUEUES - RTE_INTR_VEC_RXTX_OFFSET) + +#define FPGA_INVALID_HW_QUEUE_ID (0xFFFFFFFF) + +#define FPGA_QUEUE_FLUSH_TIMEOUT_US (1000) +#define FPGA_TIMEOUT_CHECK_INTERVAL (5) + +/* FPGA LTE FEC Register mapping on BAR0 */ +enum { + FPGA_LTE_FEC_VERSION_ID = 0x00000000, /* len: 4B */ + FPGA_LTE_FEC_CONFIGURATION = 0x00000004, /* len: 2B */ + FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE = 0x00000008, /* len: 1B */ + FPGA_LTE_FEC_LOAD_BALANCE_FACTOR = 0x0000000a, /* len: 2B */ + FPGA_LTE_FEC_RING_DESC_LEN = 0x0000000c, /* len: 2B */ + FPGA_LTE_FEC_FLR_TIME_OUT = 0x0000000e, /* len: 2B */ + FPGA_LTE_FEC_VFQ_FLUSH_STATUS_LW = 0x00000018, /* len: 4B */ + FPGA_LTE_FEC_VFQ_FLUSH_STATUS_HI = 0x0000001c, /* len: 4B */ + FPGA_LTE_FEC_VF0_DEBUG = 0x00000020, /* len: 4B */ + FPGA_LTE_FEC_VF1_DEBUG = 0x00000024, /* len: 4B */ + FPGA_LTE_FEC_VF2_DEBUG = 0x00000028, /* len: 4B */ + FPGA_LTE_FEC_VF3_DEBUG = 0x0000002c, /* len: 4B */ + FPGA_LTE_FEC_VF4_DEBUG = 0x00000030, /* len: 4B */ + FPGA_LTE_FEC_VF5_DEBUG = 0x00000034, /* len: 4B */ + FPGA_LTE_FEC_VF6_DEBUG = 0x00000038, /* len: 4B */ + FPGA_LTE_FEC_VF7_DEBUG = 0x0000003c, /* len: 4B */ + FPGA_LTE_FEC_QUEUE_MAP = 0x00000040, /* len: 256B */ + FPGA_LTE_FEC_RING_CTRL_REGS = 0x00000200 /* len: 2048B */ +}; + +/* FPGA LTE FEC Ring Control Registers */ +enum { + FPGA_LTE_FEC_RING_HEAD_ADDR = 0x00000008, + FPGA_LTE_FEC_RING_SIZE = 0x00000010, + FPGA_LTE_FEC_RING_MISC = 0x00000014, + FPGA_LTE_FEC_RING_ENABLE = 0x00000015, + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN = 0x00000016, + FPGA_LTE_FEC_RING_SHADOW_TAIL = 0x00000018, + FPGA_LTE_FEC_RING_HEAD_POINT = 0x0000001C +}; + +/* FPGA LTE FEC DESCRIPTOR ERROR */ +enum { + DESC_ERR_NO_ERR = 0x0, + DESC_ERR_K_OUT_OF_RANGE = 0x1, + DESC_ERR_K_NOT_NORMAL = 0x2, + DESC_ERR_KPAI_NOT_NORMAL = 0x3, + DESC_ERR_DESC_OFFSET_ERR = 0x4, + DESC_ERR_DESC_READ_FAIL = 0x8, + DESC_ERR_DESC_READ_TIMEOUT = 0x9, + DESC_ERR_DESC_READ_TLP_POISONED = 0xA, + DESC_ERR_CB_READ_FAIL = 0xC, + DESC_ERR_CB_READ_TIMEOUT = 0xD, + DESC_ERR_CB_READ_TLP_POISONED = 0xE +}; + +/* FPGA LTE FEC DMA Encoding Request Descriptor */ +struct __attribute__((__packed__)) fpga_dma_enc_desc { + uint32_t done:1, + rsrvd0:11, + error:4, + rsrvd1:16; + uint32_t ncb:16, + rsrvd2:14, + rv:2; + uint32_t bypass_rm:1, + irq_en:1, + crc_en:1, + rsrvd3:13, + offset:10, + rsrvd4:6; + uint16_t e; + uint16_t k; + uint32_t out_addr_lw; + uint32_t out_addr_hi; + uint32_t in_addr_lw; + uint32_t in_addr_hi; + + union { + struct { + /* Virtual addresses used to retrieve SW context info */ + void *op_addr; + /* Stores information about total number of Code Blocks + * in currently processed Transport Block + */ + uint64_t cbs_in_op; + }; + + uint8_t sw_ctxt[FPGA_RING_DESC_LEN_UNIT_BYTES * + (FPGA_RING_DESC_ENTRY_LENGTH - 1)]; + }; +}; + +/* FPGA LTE FEC DMA Decoding Request Descriptor */ +struct __attribute__((__packed__)) fpga_dma_dec_desc { + uint32_t done:1, + iter:5, + rsrvd0:2, + crc_pass:1, + rsrvd1:3, + error:4, + crc_type:1, + rsrvd2:7, + max_iter:5, + rsrvd3:3; + uint32_t rsrvd4; + uint32_t bypass_rm:1, + irq_en:1, + drop_crc:1, + rsrvd5:13, + offset:10, + rsrvd6:6; + uint16_t k; + uint16_t in_len; + uint32_t out_addr_lw; + uint32_t out_addr_hi; + uint32_t in_addr_lw; + uint32_t in_addr_hi; + + union { + struct { + /* Virtual addresses used to retrieve SW context info */ + void *op_addr; + /* Stores information about total number of Code Blocks + * in currently processed Transport Block + */ + uint8_t cbs_in_op; + }; + + uint32_t sw_ctxt[8 * (FPGA_RING_DESC_ENTRY_LENGTH - 1)]; + }; +}; + +/* FPGA LTE DMA Descriptor */ +union fpga_dma_desc { + struct fpga_dma_enc_desc enc_req; + struct fpga_dma_dec_desc dec_req; +}; + +/* FPGA LTE FEC Ring Control Register */ +struct __attribute__((__packed__)) fpga_ring_ctrl_reg { + uint64_t ring_base_addr; + uint64_t ring_head_addr; + uint16_t ring_size:11; + uint16_t rsrvd0; + union { /* Miscellaneous register */ + uint8_t misc; + uint8_t max_ul_dec:5, + max_ul_dec_en:1, + rsrvd1:2; + }; + uint8_t enable; + uint8_t flush_queue_en; + uint8_t rsrvd2; + uint16_t shadow_tail; + uint16_t rsrvd3; + uint16_t head_point; + uint16_t rsrvd4; + +}; + +/* Private data structure for each FPGA FEC device */ +struct fpga_lte_fec_device { + /** Base address of MMIO registers (BAR0) */ + void *mmio_base; + /** Base address of memory for sw rings */ + void *sw_rings; + /** Physical address of sw_rings */ + rte_iova_t sw_rings_phys; + /** Number of bytes available for each queue in device. */ + uint32_t sw_ring_size; + /** Max number of entries available for each queue in device */ + uint32_t sw_ring_max_depth; + /** Base address of response tail pointer buffer */ + uint32_t *tail_ptrs; + /** Physical address of tail pointers */ + rte_iova_t tail_ptr_phys; + /** Queues flush completion flag */ + uint64_t *flush_queue_status; + /* Bitmap capturing which Queues are bound to the PF/VF */ + uint64_t q_bound_bit_map; + /* Bitmap capturing which Queues have already been assigned */ + uint64_t q_assigned_bit_map; + /** True if this is a PF FPGA FEC device */ + bool pf_device; +}; + +/* Structure associated with each queue. */ +struct __rte_cache_aligned fpga_queue { + struct fpga_ring_ctrl_reg ring_ctrl_reg; /* Ring Control Register */ + union fpga_dma_desc *ring_addr; /* Virtual address of software ring */ + uint64_t *ring_head_addr; /* Virtual address of completion_head */ + uint64_t shadow_completion_head; /* Shadow completion head value */ + uint16_t head_free_desc; /* Ring head */ + uint16_t tail; /* Ring tail */ + /* Mask used to wrap enqueued descriptors on the sw ring */ + uint32_t sw_ring_wrap_mask; + uint32_t irq_enable; /* Enable ops dequeue interrupts if set to 1 */ + uint8_t q_idx; /* Queue index */ + struct fpga_lte_fec_device *d; + /* MMIO register of shadow_tail used to enqueue descriptors */ + void *shadow_tail_addr; +}; + +/* Write to 16 bit MMIO register address */ +static inline void +mmio_write_16(void *addr, uint16_t value) +{ + *((volatile uint16_t *)(addr)) = rte_cpu_to_le_16(value); +} + +/* Write to 32 bit MMIO register address */ +static inline void +mmio_write_32(void *addr, uint32_t value) +{ + *((volatile uint32_t *)(addr)) = rte_cpu_to_le_32(value); +} + +/* Write to 64 bit MMIO register address */ +static inline void +mmio_write_64(void *addr, uint64_t value) +{ + *((volatile uint64_t *)(addr)) = rte_cpu_to_le_64(value); +} + +/* Write a 8 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_8(void *mmio_base, uint32_t offset, uint8_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + *((volatile uint8_t *)(reg_addr)) = payload; +} + +/* Write a 16 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_16(void *mmio_base, uint32_t offset, uint16_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_16(reg_addr, payload); +} + +/* Write a 32 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_32(void *mmio_base, uint32_t offset, uint32_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_32(reg_addr, payload); +} + +/* Write a 64 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_64(void *mmio_base, uint32_t offset, uint64_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_64(reg_addr, payload); +} + +/* Write a ring control register of a FPGA LTE FEC device */ +static inline void +fpga_ring_reg_write(void *mmio_base, uint32_t offset, + struct fpga_ring_ctrl_reg payload) +{ + fpga_reg_write_64(mmio_base, offset, payload.ring_base_addr); + fpga_reg_write_64(mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_ADDR, + payload.ring_head_addr); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_SIZE, + payload.ring_size); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_POINT, + payload.head_point); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN, + payload.flush_queue_en); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_SHADOW_TAIL, + payload.shadow_tail); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_MISC, + payload.misc); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + payload.enable); +} + +/* Read a register of FPGA LTE FEC device */ +static uint32_t +fpga_reg_read_32(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint32_t ret = *((volatile uint32_t *)(reg_addr)); + return rte_le_to_cpu_32(ret); +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Read a register of FPGA LTE FEC device */ +static uint8_t +fpga_reg_read_8(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + return *((volatile uint8_t *)(reg_addr)); +} + +/* Read a register of FPGA LTE FEC device */ +static uint16_t +fpga_reg_read_16(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint16_t ret = *((volatile uint16_t *)(reg_addr)); + return rte_le_to_cpu_16(ret); +} + +/* Read a register of FPGA LTE FEC device */ +static uint64_t +fpga_reg_read_64(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint64_t ret = *((volatile uint64_t *)(reg_addr)); + return rte_le_to_cpu_64(ret); +} + +/* Read Ring Control Register of FPGA LTE FEC device */ +static inline void +print_ring_reg_debug_info(void *mmio_base, uint32_t offset) +{ + rte_bbdev_log_debug( + "FPGA MMIO base address @ %p | Ring Control Register @ offset = 0x%08" + PRIx32, mmio_base, offset); + rte_bbdev_log_debug( + "RING_BASE_ADDR = 0x%016"PRIx64, + fpga_reg_read_64(mmio_base, offset)); + rte_bbdev_log_debug( + "RING_HEAD_ADDR = 0x%016"PRIx64, + fpga_reg_read_64(mmio_base, offset + + FPGA_LTE_FEC_RING_HEAD_ADDR)); + rte_bbdev_log_debug( + "RING_SIZE = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_SIZE)); + rte_bbdev_log_debug( + "RING_MISC = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_MISC)); + rte_bbdev_log_debug( + "RING_ENABLE = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_ENABLE)); + rte_bbdev_log_debug( + "RING_FLUSH_QUEUE_EN = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN)); + rte_bbdev_log_debug( + "RING_SHADOW_TAIL = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_SHADOW_TAIL)); + rte_bbdev_log_debug( + "RING_HEAD_POINT = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_HEAD_POINT)); +} + +/* Read Static Register of FPGA LTE FEC device */ +static inline void +print_static_reg_debug_info(void *mmio_base) +{ + uint16_t config = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_CONFIGURATION); + uint8_t qmap_done = fpga_reg_read_8(mmio_base, + FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE); + uint16_t lb_factor = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_LOAD_BALANCE_FACTOR); + uint16_t ring_desc_len = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_RING_DESC_LEN); + uint16_t flr_time_out = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_FLR_TIME_OUT); + + rte_bbdev_log_debug("UL.DL Weights = %u.%u", + ((uint8_t)config), ((uint8_t)(config >> 8))); + rte_bbdev_log_debug("UL.DL Load Balance = %u.%u", + ((uint8_t)lb_factor), ((uint8_t)(lb_factor >> 8))); + rte_bbdev_log_debug("Queue-PF/VF Mapping Table = %s", + (qmap_done > 0) ? "READY" : "NOT-READY"); + rte_bbdev_log_debug("Ring Descriptor Size = %u bytes", + ring_desc_len*FPGA_RING_DESC_LEN_UNIT_BYTES); + rte_bbdev_log_debug("FLR Timeout = %f usec", + (float)flr_time_out*FPGA_FLR_TIMEOUT_UNIT); +} + +/* Print decode DMA Descriptor of FPGA LTE FEC device */ +static void +print_dma_dec_desc_debug_info(union fpga_dma_desc *desc) +{ + rte_bbdev_log_debug("DMA response desc %p\n" + "\t-- done(%"PRIu32") | iter(%"PRIu32") | crc_pass(%"PRIu32")" + " | error (%"PRIu32") | crc_type(%"PRIu32")\n" + "\t-- max_iter(%"PRIu32") | bypass_rm(%"PRIu32") | " + "irq_en (%"PRIu32") | drop_crc(%"PRIu32") | offset(%"PRIu32")\n" + "\t-- k(%"PRIu32") | in_len (%"PRIu16") | op_add(%p)\n" + "\t-- cbs_in_op(%"PRIu32") | in_add (0x%08"PRIx32"%08"PRIx32") | " + "out_add (0x%08"PRIx32"%08"PRIx32")", + desc, + (uint32_t)desc->dec_req.done, + (uint32_t)desc->dec_req.iter, + (uint32_t)desc->dec_req.crc_pass, + (uint32_t)desc->dec_req.error, + (uint32_t)desc->dec_req.crc_type, + (uint32_t)desc->dec_req.max_iter, + (uint32_t)desc->dec_req.bypass_rm, + (uint32_t)desc->dec_req.irq_en, + (uint32_t)desc->dec_req.drop_crc, + (uint32_t)desc->dec_req.offset, + (uint32_t)desc->dec_req.k, + (uint16_t)desc->dec_req.in_len, + desc->dec_req.op_addr, + (uint32_t)desc->dec_req.cbs_in_op, + (uint32_t)desc->dec_req.in_addr_hi, + (uint32_t)desc->dec_req.in_addr_lw, + (uint32_t)desc->dec_req.out_addr_hi, + (uint32_t)desc->dec_req.out_addr_lw); +} +#endif + +static int +fpga_setup_queues(struct rte_bbdev *dev, uint16_t num_queues, int socket_id) +{ + /* Number of queues bound to a PF/VF */ + uint32_t hw_q_num = 0; + uint32_t ring_size, payload, address, q_id, offset; + rte_iova_t phys_addr; + struct fpga_ring_ctrl_reg ring_reg; + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + + address = FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE; + if (!(fpga_reg_read_32(fpga_dev->mmio_base, address) & 0x1)) { + rte_bbdev_log(ERR, + "Queue-PF/VF mapping is not set! Was PF configured for device (%s) ?", + dev->data->name); + return -EPERM; + } + + /* Clear queue registers structure */ + memset(&ring_reg, 0, sizeof(struct fpga_ring_ctrl_reg)); + + /* Scan queue map. + * If a queue is valid and mapped to a calling PF/VF the read value is + * replaced with a queue ID and if it's not then + * FPGA_INVALID_HW_QUEUE_ID is returned. + */ + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + uint32_t hw_q_id = fpga_reg_read_32(fpga_dev->mmio_base, + FPGA_LTE_FEC_QUEUE_MAP + (q_id << 2)); + + rte_bbdev_log_debug("%s: queue ID: %u, registry queue ID: %u", + dev->device->name, q_id, hw_q_id); + + if (hw_q_id != FPGA_INVALID_HW_QUEUE_ID) { + fpga_dev->q_bound_bit_map |= (1ULL << q_id); + /* Clear queue register of found queue */ + offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q_id); + fpga_ring_reg_write(fpga_dev->mmio_base, + offset, ring_reg); + ++hw_q_num; + } + } + if (hw_q_num == 0) { + rte_bbdev_log(ERR, + "No HW queues assigned to this device. Probably this is a VF configured for PF mode. Check device configuration!"); + return -ENODEV; + } + + if (num_queues > hw_q_num) { + rte_bbdev_log(ERR, + "Not enough queues for device %s! Requested: %u, available: %u", + dev->device->name, num_queues, hw_q_num); + return -EINVAL; + } + + ring_size = FPGA_RING_MAX_SIZE * sizeof(struct fpga_dma_dec_desc); + + /* Enforce 32 byte alignment */ + RTE_BUILD_BUG_ON((RTE_CACHE_LINE_SIZE % 32) != 0); + + /* Allocate memory for SW descriptor rings */ + fpga_dev->sw_rings = rte_zmalloc_socket(dev->device->driver->name, + num_queues * ring_size, RTE_CACHE_LINE_SIZE, + socket_id); + if (fpga_dev->sw_rings == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u sw_rings", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + + fpga_dev->sw_rings_phys = rte_malloc_virt2iova(fpga_dev->sw_rings); + fpga_dev->sw_ring_size = ring_size; + fpga_dev->sw_ring_max_depth = FPGA_RING_MAX_SIZE; + + /* Allocate memory for ring flush status */ + fpga_dev->flush_queue_status = rte_zmalloc_socket(NULL, + sizeof(uint64_t), RTE_CACHE_LINE_SIZE, socket_id); + if (fpga_dev->flush_queue_status == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u flush_queue_status", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + + /* Set the flush status address registers */ + phys_addr = rte_malloc_virt2iova(fpga_dev->flush_queue_status); + + address = FPGA_LTE_FEC_VFQ_FLUSH_STATUS_LW; + payload = (uint32_t)(phys_addr); + fpga_reg_write_32(fpga_dev->mmio_base, address, payload); + + address = FPGA_LTE_FEC_VFQ_FLUSH_STATUS_HI; + payload = (uint32_t)(phys_addr >> 32); + fpga_reg_write_32(fpga_dev->mmio_base, address, payload); + + return 0; +} + +static int +fpga_dev_close(struct rte_bbdev *dev) +{ + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + + rte_free(fpga_dev->sw_rings); + rte_free(fpga_dev->flush_queue_status); + + return 0; +} + +static void +fpga_dev_info_get(struct rte_bbdev *dev, + struct rte_bbdev_driver_info *dev_info) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + uint32_t q_id = 0; + + /* TODO RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN and numbers of buffers are set + * to temporary values as they are required by test application while + * validation phase. + */ + static const struct rte_bbdev_op_cap bbdev_capabilities[] = { + { + .type = RTE_BBDEV_OP_TURBO_DEC, + .cap.turbo_dec = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_TYPE_24B | + RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE | + RTE_BBDEV_TURBO_DEC_INTERRUPTS | + RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN | + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP, + .max_llr_modulus = INT8_MAX, + .num_buffers_src = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_hard_out = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_soft_out = 0 + } + }, + { + .type = RTE_BBDEV_OP_TURBO_ENC, + .cap.turbo_enc = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_24B_ATTACH | + RTE_BBDEV_TURBO_RATE_MATCH | + RTE_BBDEV_TURBO_ENC_INTERRUPTS, + .num_buffers_src = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_dst = + RTE_BBDEV_MAX_CODE_BLOCKS + } + }, + RTE_BBDEV_END_OF_CAPABILITIES_LIST() + }; + + static struct rte_bbdev_queue_conf default_queue_conf; + default_queue_conf.socket = dev->data->socket_id; + default_queue_conf.queue_size = FPGA_RING_MAX_SIZE; + + + dev_info->driver_name = dev->device->driver->name; + dev_info->queue_size_lim = FPGA_RING_MAX_SIZE; + dev_info->hardware_accelerated = true; + dev_info->min_alignment = 64; + dev_info->default_queue_conf = default_queue_conf; + dev_info->capabilities = bbdev_capabilities; + dev_info->cpu_flag_reqs = NULL; + + /* Calculates number of queues assigned to device */ + dev_info->max_num_queues = 0; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + uint32_t hw_q_id = fpga_reg_read_32(d->mmio_base, + FPGA_LTE_FEC_QUEUE_MAP + (q_id << 2)); + if (hw_q_id != FPGA_INVALID_HW_QUEUE_ID) + dev_info->max_num_queues++; + } +} + +/** + * Find index of queue bound to current PF/VF which is unassigned. Return -1 + * when there is no available queue + */ +static int +fpga_find_free_queue_idx(struct rte_bbdev *dev, + const struct rte_bbdev_queue_conf *conf) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + uint64_t q_idx; + uint8_t i = 0; + uint8_t range = FPGA_TOTAL_NUM_QUEUES >> 1; + + if (conf->op_type == RTE_BBDEV_OP_TURBO_ENC) { + i = FPGA_NUM_DL_QUEUES; + range = FPGA_TOTAL_NUM_QUEUES; + } + + for (; i < range; ++i) { + q_idx = 1ULL << i; + /* Check if index of queue is bound to current PF/VF */ + if (d->q_bound_bit_map & q_idx) + /* Check if found queue was not already assigned */ + if (!(d->q_assigned_bit_map & q_idx)) { + d->q_assigned_bit_map |= q_idx; + return i; + } + } + + rte_bbdev_log(INFO, "Failed to find free queue on %s", dev->data->name); + + return -1; +} + +static int +fpga_queue_setup(struct rte_bbdev *dev, uint16_t queue_id, + const struct rte_bbdev_queue_conf *conf) +{ + uint32_t address, ring_offset; + struct fpga_lte_fec_device *d = dev->data->dev_private; + struct fpga_queue *q; + int8_t q_idx; + + /* Check if there is a free queue to assign */ + q_idx = fpga_find_free_queue_idx(dev, conf); + if (q_idx == -1) + return -1; + + /* Allocate the queue data structure. */ + q = rte_zmalloc_socket(dev->device->driver->name, sizeof(*q), + RTE_CACHE_LINE_SIZE, conf->socket); + if (q == NULL) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_bbdev_log(ERR, "Failed to allocate queue memory"); + return -ENOMEM; + } + + q->d = d; + q->q_idx = q_idx; + + /* Set ring_base_addr */ + q->ring_addr = RTE_PTR_ADD(d->sw_rings, (d->sw_ring_size * queue_id)); + q->ring_ctrl_reg.ring_base_addr = d->sw_rings_phys + + (d->sw_ring_size * queue_id); + + /* Allocate memory for Completion Head variable*/ + q->ring_head_addr = rte_zmalloc_socket(dev->device->driver->name, + sizeof(uint64_t), RTE_CACHE_LINE_SIZE, conf->socket); + if (q->ring_head_addr == NULL) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_free(q); + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u completion_head", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + /* Set ring_head_addr */ + q->ring_ctrl_reg.ring_head_addr = + rte_malloc_virt2iova(q->ring_head_addr); + + /* Clear shadow_completion_head */ + q->shadow_completion_head = 0; + + /* Set ring_size */ + if (conf->queue_size > FPGA_RING_MAX_SIZE) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_free(q->ring_head_addr); + rte_free(q); + rte_bbdev_log(ERR, + "Size of queue is too big %d (MAX: %d ) for %s:%u", + conf->queue_size, FPGA_RING_MAX_SIZE, + dev->device->driver->name, dev->data->dev_id); + return -EINVAL; + } + q->ring_ctrl_reg.ring_size = conf->queue_size; + + /* Set Miscellaneous FPGA register*/ + /* Max iteration number for TTI mitigation - todo */ + q->ring_ctrl_reg.max_ul_dec = 0; + /* Enable max iteration number for TTI - todo */ + q->ring_ctrl_reg.max_ul_dec_en = 0; + + /* Enable the ring */ + q->ring_ctrl_reg.enable = 1; + + /* Set FPGA head_point and tail registers */ + q->ring_ctrl_reg.head_point = q->tail = 0; + + /* Set FPGA shadow_tail register */ + q->ring_ctrl_reg.shadow_tail = q->tail; + + /* Calculates the ring offset for found queue */ + ring_offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q_idx); + + /* Set FPGA Ring Control Registers */ + fpga_ring_reg_write(d->mmio_base, ring_offset, q->ring_ctrl_reg); + + /* Store MMIO register of shadow_tail */ + address = ring_offset + FPGA_LTE_FEC_RING_SHADOW_TAIL; + q->shadow_tail_addr = RTE_PTR_ADD(d->mmio_base, address); + + q->head_free_desc = q->tail; + + /* Set wrap mask */ + q->sw_ring_wrap_mask = conf->queue_size - 1; + + rte_bbdev_log_debug("Setup dev%u q%u: queue_idx=%u", + dev->data->dev_id, queue_id, q->q_idx); + + dev->data->queues[queue_id].queue_private = q; + + rte_bbdev_log_debug("BBDEV queue[%d] set up for FPGA queue[%d]", + queue_id, q_idx); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Read FPGA Ring Control Registers after configuration*/ + print_ring_reg_debug_info(d->mmio_base, ring_offset); +#endif + return 0; +} + +static int +fpga_queue_release(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + struct fpga_ring_ctrl_reg ring_reg; + uint32_t offset; + + rte_bbdev_log_debug("FPGA Queue[%d] released", queue_id); + + if (q != NULL) { + memset(&ring_reg, 0, sizeof(struct fpga_ring_ctrl_reg)); + offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + /* Disable queue */ + fpga_reg_write_8(d->mmio_base, + offset + FPGA_LTE_FEC_RING_ENABLE, 0x00); + /* Clear queue registers */ + fpga_ring_reg_write(d->mmio_base, offset, ring_reg); + + /* Mark the Queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q->q_idx)); + rte_free(q->ring_head_addr); + rte_free(q); + dev->data->queues[queue_id].queue_private = NULL; + } + + return 0; +} + +/* Function starts a device queue. */ +static int +fpga_queue_start(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (d == NULL) { + rte_bbdev_log(ERR, "Invalid device pointer"); + return -1; + } +#endif + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + uint32_t offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + uint8_t enable = 0x01; + uint16_t zero = 0x0000; + + /* Clear queue head and tail variables */ + q->tail = q->head_free_desc = 0; + + /* Clear FPGA head_point and tail registers */ + fpga_reg_write_16(d->mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_POINT, + zero); + fpga_reg_write_16(d->mmio_base, offset + FPGA_LTE_FEC_RING_SHADOW_TAIL, + zero); + + /* Enable queue */ + fpga_reg_write_8(d->mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + enable); + + rte_bbdev_log_debug("FPGA Queue[%d] started", queue_id); + return 0; +} + +/* Function stops a device queue. */ +static int +fpga_queue_stop(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (d == NULL) { + rte_bbdev_log(ERR, "Invalid device pointer"); + return -1; + } +#endif + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + uint32_t offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + uint8_t payload = 0x01; + uint8_t counter = 0; + uint8_t timeout = FPGA_QUEUE_FLUSH_TIMEOUT_US / + FPGA_TIMEOUT_CHECK_INTERVAL; + + /* Set flush_queue_en bit to trigger queue flushing */ + fpga_reg_write_8(d->mmio_base, + offset + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN, payload); + + /** Check if queue flush is completed. + * FPGA will update the completion flag after queue flushing is + * completed. If completion flag is not updated within 1ms it is + * considered as a failure. + */ + while (!(*((uint8_t *)d->flush_queue_status + q->q_idx) & payload)) { + if (counter > timeout) { + rte_bbdev_log(ERR, "FPGA Queue Flush failed for queue %d", + queue_id); + return -1; + } + usleep(FPGA_TIMEOUT_CHECK_INTERVAL); + counter++; + } + + /* Disable queue */ + payload = 0x00; + fpga_reg_write_8(d->mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + payload); + + rte_bbdev_log_debug("FPGA Queue[%d] stopped", queue_id); + return 0; +} + +static inline uint16_t +get_queue_id(struct rte_bbdev_data *data, uint8_t q_idx) +{ + uint16_t queue_id; + + for (queue_id = 0; queue_id < data->num_queues; ++queue_id) { + struct fpga_queue *q = data->queues[queue_id].queue_private; + if (q != NULL && q->q_idx == q_idx) + return queue_id; + } + + return -1; +} + +/* Interrupt handler triggered by FPGA dev for handling specific interrupt */ +static void +fpga_dev_interrupt_handler(void *cb_arg) +{ + struct rte_bbdev *dev = cb_arg; + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + struct fpga_queue *q; + uint64_t ring_head; + uint64_t q_idx; + uint16_t queue_id; + uint8_t i; + + /* Scan queue assigned to this device */ + for (i = 0; i < FPGA_TOTAL_NUM_QUEUES; ++i) { + q_idx = 1ULL << i; + if (fpga_dev->q_bound_bit_map & q_idx) { + queue_id = get_queue_id(dev->data, i); + if (queue_id == (uint16_t) -1) + continue; + + /* Check if completion head was changed */ + q = dev->data->queues[queue_id].queue_private; + ring_head = *q->ring_head_addr; + if (q->shadow_completion_head != ring_head && + q->irq_enable == 1) { + q->shadow_completion_head = ring_head; + rte_bbdev_pmd_callback_process( + dev, + RTE_BBDEV_EVENT_DEQUEUE, + &queue_id); + } + } + } +} + +static int +fpga_queue_intr_enable(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + + if (!rte_intr_cap_multiple(dev->intr_handle)) + return -ENOTSUP; + + q->irq_enable = 1; + + return 0; +} + +static int +fpga_queue_intr_disable(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + q->irq_enable = 0; + + return 0; +} + +static int +fpga_intr_enable(struct rte_bbdev *dev) +{ + int ret; + uint8_t i; + + if (!rte_intr_cap_multiple(dev->intr_handle)) { + rte_bbdev_log(ERR, "Multiple intr vector is not supported by FPGA (%s)", + dev->data->name); + return -ENOTSUP; + } + + /* Create event file descriptors for each of 64 queue. Event fds will be + * mapped to FPGA IRQs in rte_intr_enable(). This is a 1:1 mapping where + * the IRQ number is a direct translation to the queue number. + * + * 63 (FPGA_NUM_INTR_VEC) event fds are created as rte_intr_enable() + * mapped the first IRQ to already created interrupt event file + * descriptor (intr_handle->fd). + */ + if (rte_intr_efd_enable(dev->intr_handle, FPGA_NUM_INTR_VEC)) { + rte_bbdev_log(ERR, "Failed to create fds for %u queues", + dev->data->num_queues); + return -1; + } + + /* TODO Each event file descriptor is overwritten by interrupt event + * file descriptor. That descriptor is added to epoll observed list. + * It ensures that callback function assigned to that descriptor will + * invoked when any FPGA queue issues interrupt. + */ + for (i = 0; i < FPGA_NUM_INTR_VEC; ++i) + dev->intr_handle->efds[i] = dev->intr_handle->fd; + + if (!dev->intr_handle->intr_vec) { + dev->intr_handle->intr_vec = rte_zmalloc("intr_vec", + dev->data->num_queues * sizeof(int), 0); + if (!dev->intr_handle->intr_vec) { + rte_bbdev_log(ERR, "Failed to allocate %u vectors", + dev->data->num_queues); + return -ENOMEM; + } + } + + ret = rte_intr_enable(dev->intr_handle); + if (ret < 0) { + rte_bbdev_log(ERR, + "Couldn't enable interrupts for device: %s", + dev->data->name); + return ret; + } + + ret = rte_intr_callback_register(dev->intr_handle, + fpga_dev_interrupt_handler, dev); + if (ret < 0) { + rte_bbdev_log(ERR, + "Couldn't register interrupt callback for device: %s", + dev->data->name); + return ret; + } + + return 0; +} + +static const struct rte_bbdev_ops fpga_ops = { + .setup_queues = fpga_setup_queues, + .intr_enable = fpga_intr_enable, + .close = fpga_dev_close, + .info_get = fpga_dev_info_get, + .queue_setup = fpga_queue_setup, + .queue_stop = fpga_queue_stop, + .queue_start = fpga_queue_start, + .queue_release = fpga_queue_release, + .queue_intr_enable = fpga_queue_intr_enable, + .queue_intr_disable = fpga_queue_intr_disable +}; + +static inline void +fpga_dma_enqueue(struct fpga_queue *q, uint16_t num_desc, + struct rte_bbdev_stats *queue_stats) +{ +#ifdef RTE_BBDEV_OFFLOAD_COST + uint64_t start_time = 0; + queue_stats->acc_offload_cycles = 0; +#else + RTE_SET_USED(queue_stats); +#endif + + /* Update tail and shadow_tail register */ + q->tail = (q->tail + num_desc) & q->sw_ring_wrap_mask; + + rte_wmb(); + +#ifdef RTE_BBDEV_OFFLOAD_COST + /* Start time measurement for enqueue function offload. */ + start_time = rte_rdtsc_precise(); +#endif + mmio_write_16(q->shadow_tail_addr, q->tail); + +#ifdef RTE_BBDEV_OFFLOAD_COST + rte_wmb(); + queue_stats->acc_offload_cycles += rte_rdtsc_precise() - start_time; +#endif +} + +/* Calculates number of CBs in processed encoder TB based on 'r' and input + * length. + */ +static inline uint8_t +get_num_cbs_in_op_enc(struct rte_bbdev_op_turbo_enc *turbo_enc) +{ + uint8_t c, c_neg, r, crc24_bits = 0; + uint16_t k, k_neg, k_pos; + uint8_t cbs_in_op = 0; + int32_t length; + + length = turbo_enc->input.length; + r = turbo_enc->tb_params.r; + c = turbo_enc->tb_params.c; + c_neg = turbo_enc->tb_params.c_neg; + k_neg = turbo_enc->tb_params.k_neg; + k_pos = turbo_enc->tb_params.k_pos; + crc24_bits = 24; + while (length > 0 && r < c) { + k = (r < c_neg) ? k_neg : k_pos; + length -= (k - crc24_bits) >> 3; + r++; + cbs_in_op++; + } + + return cbs_in_op; +} + +/* Calculates number of CBs in processed decoder TB based on 'r' and input + * length. + */ +static inline uint16_t +get_num_cbs_in_op_dec(struct rte_bbdev_op_turbo_dec *turbo_dec) +{ + uint8_t c, c_neg, r = 0; + uint16_t kw, k, k_neg, k_pos, cbs_in_op = 0; + int32_t length; + + length = turbo_dec->input.length; + r = turbo_dec->tb_params.r; + c = turbo_dec->tb_params.c; + c_neg = turbo_dec->tb_params.c_neg; + k_neg = turbo_dec->tb_params.k_neg; + k_pos = turbo_dec->tb_params.k_pos; + while (length > 0 && r < c) { + k = (r < c_neg) ? k_neg : k_pos; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + length -= kw; + r++; + cbs_in_op++; + } + + return cbs_in_op; +} + +/* Read flag value 0/1/ from bitmap */ +static inline bool +check_bit(uint32_t bitmap, uint32_t bitmask) +{ + return bitmap & bitmask; +} + +/* Print an error if a descriptor error has occurred. + * Return 0 on success, 1 on failure + */ +static inline int +check_desc_error(uint32_t error_code) { + switch (error_code) { + case DESC_ERR_NO_ERR: + return 0; + case DESC_ERR_K_OUT_OF_RANGE: + rte_bbdev_log(ERR, "Block_size_k is out of range (k<40 or k>6144)"); + break; + case DESC_ERR_K_NOT_NORMAL: + rte_bbdev_log(ERR, "Block_size_k is not a normal value within normal range"); + break; + case DESC_ERR_KPAI_NOT_NORMAL: + rte_bbdev_log(ERR, "Three_kpai is not a normal value for UL only"); + break; + case DESC_ERR_DESC_OFFSET_ERR: + rte_bbdev_log(ERR, "Queue offset does not meet the expectation in the FPGA"); + break; + case (DESC_ERR_K_OUT_OF_RANGE | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Block_size_k is out of range (k<40 or k>6144) and queue offset error"); + break; + case (DESC_ERR_K_NOT_NORMAL | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Block_size_k is not a normal value within normal range and queue offset error"); + break; + case (DESC_ERR_KPAI_NOT_NORMAL | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Three_kpai is not a normal value for UL only and queue offset error"); + break; + case DESC_ERR_DESC_READ_FAIL: + rte_bbdev_log(ERR, "Unsuccessful completion for descriptor read"); + break; + case DESC_ERR_DESC_READ_TIMEOUT: + rte_bbdev_log(ERR, "Descriptor read time-out"); + break; + case DESC_ERR_DESC_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Descriptor read TLP poisoned"); + break; + case DESC_ERR_CB_READ_FAIL: + rte_bbdev_log(ERR, "Unsuccessful completion for code block"); + break; + case DESC_ERR_CB_READ_TIMEOUT: + rte_bbdev_log(ERR, "Code block read time-out"); + break; + case DESC_ERR_CB_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Code block read TLP poisoned"); + break; + default: + rte_bbdev_log(ERR, "Descriptor error unknown error code %u", + error_code); + break; + } + return 1; +} + +/** + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param k + * K value (length of input in bits). + * @param e + * E value (length of output in bits). + * @param ncb + * Ncb value (size of the soft buffer). + * @param out_length + * Length of output buffer + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +fpga_dma_desc_te_fill(struct rte_bbdev_enc_op *op, + struct fpga_dma_enc_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t k, uint16_t e, uint16_t ncb, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) + +{ + /* reset */ + desc->done = 0; + desc->crc_en = check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_CRC_24B_ATTACH); + desc->bypass_rm = !check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_RATE_MATCH); + desc->k = k; + desc->e = e; + desc->ncb = ncb; + desc->rv = op->turbo_enc.rv_index; + desc->offset = desc_offset; + /* Set inbound data buffer address */ + desc->in_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset) >> 32); + desc->in_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset)); + + desc->out_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset) >> 32); + desc->out_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset)); + + /* Save software context needed for dequeue */ + desc->op_addr = op; + + /* Set total number of CBs in an op */ + desc->cbs_in_op = cbs_in_op; + + return 0; +} + +/** + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param in_length + * Length of an input. + * @param k + * K value (length of an output in bits). + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +fpga_dma_desc_td_fill(struct rte_bbdev_dec_op *op, + struct fpga_dma_dec_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t in_length, uint16_t k, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) +{ + /* reset */ + desc->done = 0; + /* Set inbound data buffer address */ + desc->in_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset) >> 32); + desc->in_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset)); + desc->in_len = in_length; + desc->k = k; + desc->crc_type = !check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_CRC_TYPE_24B); + if ((op->turbo_dec.code_block_mode == 0) + && !check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP)) + desc->drop_crc = 1; + desc->max_iter = op->turbo_dec.iter_max * 2; + desc->offset = desc_offset; + desc->out_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset) >> 32); + desc->out_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset)); + + /* Save software context needed for dequeue */ + desc->op_addr = op; + + /* Set total number of CBs in an op */ + desc->cbs_in_op = cbs_in_op; + + return 0; +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Validates turbo encoder parameters */ +static int +validate_enc_op(struct rte_bbdev_enc_op *op) +{ + struct rte_bbdev_op_turbo_enc *turbo_enc = &op->turbo_enc; + struct rte_bbdev_op_enc_cb_params *cb = NULL; + struct rte_bbdev_op_enc_tb_params *tb = NULL; + uint16_t kw, kw_neg, kw_pos; + + if (turbo_enc->input.length > + RTE_BBDEV_MAX_TB_SIZE >> 3) { + rte_bbdev_log(ERR, "TB size (%u) is too big, max: %d", + turbo_enc->input.length, RTE_BBDEV_MAX_TB_SIZE); + op->status = 1 << RTE_BBDEV_DATA_ERROR; + return -1; + } + + if (op->mempool == NULL) { + rte_bbdev_log(ERR, "Invalid mempool pointer"); + return -1; + } + if (turbo_enc->input.data == NULL) { + rte_bbdev_log(ERR, "Invalid input pointer"); + return -1; + } + if (turbo_enc->output.data == NULL) { + rte_bbdev_log(ERR, "Invalid output pointer"); + return -1; + } + if (turbo_enc->rv_index > 3) { + rte_bbdev_log(ERR, + "rv_index (%u) is out of range 0 <= value <= 3", + turbo_enc->rv_index); + return -1; + } + if (turbo_enc->code_block_mode != 0 && + turbo_enc->code_block_mode != 1) { + rte_bbdev_log(ERR, + "code_block_mode (%u) is out of range 0 <= value <= 1", + turbo_enc->code_block_mode); + return -1; + } + + if (turbo_enc->code_block_mode == 0) { + tb = &turbo_enc->tb_params; + if ((tb->k_neg < RTE_BBDEV_MIN_CB_SIZE + || tb->k_neg > RTE_BBDEV_MAX_CB_SIZE) + && tb->c_neg > 0) { + rte_bbdev_log(ERR, + "k_neg (%u) is out of range %u <= value <= %u", + tb->k_neg, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->k_pos < RTE_BBDEV_MIN_CB_SIZE + || tb->k_pos > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k_pos (%u) is out of range %u <= value <= %u", + tb->k_pos, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->c_neg > (RTE_BBDEV_MAX_CODE_BLOCKS - 1)) + rte_bbdev_log(ERR, + "c_neg (%u) is out of range 0 <= value <= %u", + tb->c_neg, + RTE_BBDEV_MAX_CODE_BLOCKS - 1); + if (tb->c < 1 || tb->c > RTE_BBDEV_MAX_CODE_BLOCKS) { + rte_bbdev_log(ERR, + "c (%u) is out of range 1 <= value <= %u", + tb->c, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + if (tb->cab > tb->c) { + rte_bbdev_log(ERR, + "cab (%u) is greater than c (%u)", + tb->cab, tb->c); + return -1; + } + if ((tb->ea < RTE_BBDEV_MIN_CB_SIZE || (tb->ea % 2)) + && tb->r < tb->cab) { + rte_bbdev_log(ERR, + "ea (%u) is less than %u or it is not even", + tb->ea, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + if ((tb->eb < RTE_BBDEV_MIN_CB_SIZE || (tb->eb % 2)) + && tb->c > tb->cab) { + rte_bbdev_log(ERR, + "eb (%u) is less than %u or it is not even", + tb->eb, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + + kw_neg = 3 * RTE_ALIGN_CEIL(tb->k_neg + 4, + RTE_BBDEV_C_SUBBLOCK); + if (tb->ncb_neg < tb->k_neg || tb->ncb_neg > kw_neg) { + rte_bbdev_log(ERR, + "ncb_neg (%u) is out of range (%u) k_neg <= value <= (%u) kw_neg", + tb->ncb_neg, tb->k_neg, kw_neg); + return -1; + } + + kw_pos = 3 * RTE_ALIGN_CEIL(tb->k_pos + 4, + RTE_BBDEV_C_SUBBLOCK); + if (tb->ncb_pos < tb->k_pos || tb->ncb_pos > kw_pos) { + rte_bbdev_log(ERR, + "ncb_pos (%u) is out of range (%u) k_pos <= value <= (%u) kw_pos", + tb->ncb_pos, tb->k_pos, kw_pos); + return -1; + } + if (tb->r > (tb->c - 1)) { + rte_bbdev_log(ERR, + "r (%u) is greater than c - 1 (%u)", + tb->r, tb->c - 1); + return -1; + } + } else { + cb = &turbo_enc->cb_params; + if (cb->k < RTE_BBDEV_MIN_CB_SIZE + || cb->k > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k (%u) is out of range %u <= value <= %u", + cb->k, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + + if (cb->e < RTE_BBDEV_MIN_CB_SIZE || (cb->e % 2)) { + rte_bbdev_log(ERR, + "e (%u) is less than %u or it is not even", + cb->e, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + + kw = RTE_ALIGN_CEIL(cb->k + 4, RTE_BBDEV_C_SUBBLOCK) * 3; + if (cb->ncb < cb->k || cb->ncb > kw) { + rte_bbdev_log(ERR, + "ncb (%u) is out of range (%u) k <= value <= (%u) kw", + cb->ncb, cb->k, kw); + return -1; + } + } + + return 0; +} +#endif + +static inline char * +mbuf_append(struct rte_mbuf *m_head, struct rte_mbuf *m, uint16_t len) +{ + if (unlikely(len > rte_pktmbuf_tailroom(m))) + return NULL; + + char *tail = (char *)m->buf_addr + m->data_off + m->data_len; + m->data_len = (uint16_t)(m->data_len + len); + m_head->pkt_len = (m_head->pkt_len + len); + return tail; +} + +static inline int +enqueue_enc_one_op_cb(struct fpga_queue *q, struct rte_bbdev_enc_op *op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input; + struct rte_mbuf *output; + int ret; + uint16_t k, e, ncb, ring_offset; + uint32_t total_left, in_length, out_length, in_offset, out_offset; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_enc_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo encoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_enc.input.data; + output = op->turbo_enc.output.data; + in_offset = op->turbo_enc.input.offset; + out_offset = op->turbo_enc.output.offset; + total_left = op->turbo_enc.input.length; + k = op->turbo_enc.cb_params.k; + e = op->turbo_enc.cb_params.e; + ncb = op->turbo_enc.cb_params.ncb; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_CRC_24B_ATTACH)) + in_length = ((k - 24) >> 3); + else + in_length = k >> 3; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_RATE_MATCH)) + out_length = (e + 7) >> 3; + else + out_length = (k >> 3) * 3 + 2; + + mbuf_append(output, output, out_length); + + /* Offset into the ring */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + /* Setup DMA Descriptor */ + desc = q->ring_addr + ring_offset; + + ret = fpga_dma_desc_te_fill(op, &desc->enc_req, input, output, k, e, + ncb, in_offset, out_offset, ring_offset, 1); + if (unlikely(ret < 0)) + return ret; + + /* Update lengths */ + total_left -= in_length; + op->turbo_enc.output.length += out_length; + + if (total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + total_left, in_length); + return -1; + } + + return 1; +} + +static inline int +enqueue_enc_one_op_tb(struct fpga_queue *q, struct rte_bbdev_enc_op *op, + uint16_t desc_offset, uint8_t cbs_in_op) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input, *output_head, *output; + int ret; + uint8_t r, c, crc24_bits = 0; + uint16_t k, e, ncb, ring_offset; + uint32_t mbuf_total_left, in_length, out_length, in_offset, out_offset; + uint32_t seg_total_left; + uint16_t current_enqueued_cbs = 0; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_enc_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo encoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_enc.input.data; + output_head = output = op->turbo_enc.output.data; + in_offset = op->turbo_enc.input.offset; + out_offset = op->turbo_enc.output.offset; + mbuf_total_left = op->turbo_enc.input.length; + + c = op->turbo_enc.tb_params.c; + r = op->turbo_enc.tb_params.r; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_CRC_24B_ATTACH)) + crc24_bits = 24; + + while (mbuf_total_left > 0 && r < c && input != NULL) { + seg_total_left = rte_pktmbuf_data_len(input) - in_offset; + + e = (r < op->turbo_enc.tb_params.cab) ? + op->turbo_enc.tb_params.ea : + op->turbo_enc.tb_params.eb; + k = (r < op->turbo_enc.tb_params.c_neg) ? + op->turbo_enc.tb_params.k_neg : + op->turbo_enc.tb_params.k_pos; + ncb = (r < op->turbo_enc.tb_params.c_neg) ? + op->turbo_enc.tb_params.ncb_neg : + op->turbo_enc.tb_params.ncb_pos; + + in_length = ((k - crc24_bits) >> 3); + + if (check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_RATE_MATCH)) + out_length = (e + 7) >> 3; + else + out_length = (k >> 3) * 3 + 2; + + mbuf_append(output_head, output, out_length); + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_te_fill(op, &desc->enc_req, input, output, + k, e, ncb, in_offset, out_offset, ring_offset, + cbs_in_op); + if (unlikely(ret < 0)) + return ret; + + rte_bbdev_log_debug("DMA request desc %p", desc); + + /* Update lengths */ + op->turbo_enc.output.length += out_length; + mbuf_total_left -= in_length; + + /* Update offsets */ + if (seg_total_left == in_length) { + /* Go to the next mbuf */ + input = input->next; + output = output->next; + in_offset = 0; + out_offset = 0; + } else { + in_offset += in_length; + out_offset += out_length; + } + + r++; + desc_offset++; + current_enqueued_cbs++; + } + + if (mbuf_total_left > 0) { + rte_bbdev_log(ERR, + "Some date still left for processing: mbuf_total_left = %u", + mbuf_total_left); + return -1; + } + + return current_enqueued_cbs; +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Validates turbo decoder parameters */ +static int +validate_dec_op(struct rte_bbdev_dec_op *op) +{ + struct rte_bbdev_op_turbo_dec *turbo_dec = &op->turbo_dec; + struct rte_bbdev_op_dec_cb_params *cb = NULL; + struct rte_bbdev_op_dec_tb_params *tb = NULL; + + if (op->mempool == NULL) { + rte_bbdev_log(ERR, "Invalid mempool pointer"); + return -1; + } + if (turbo_dec->input.data == NULL) { + rte_bbdev_log(ERR, "Invalid input pointer"); + return -1; + } + if (turbo_dec->hard_output.data == NULL) { + rte_bbdev_log(ERR, "Invalid hard_output pointer"); + return -1; + } + if (turbo_dec->rv_index > 3) { + rte_bbdev_log(ERR, + "rv_index (%u) is out of range 0 <= value <= 3", + turbo_dec->rv_index); + return -1; + } + if (turbo_dec->iter_min < 1) { + rte_bbdev_log(ERR, + "iter_min (%u) is less than 1", + turbo_dec->iter_min); + return -1; + } + if (turbo_dec->iter_max <= 2) { + rte_bbdev_log(ERR, + "iter_max (%u) is less than or equal to 2", + turbo_dec->iter_max); + return -1; + } + if (turbo_dec->iter_min > turbo_dec->iter_max) { + rte_bbdev_log(ERR, + "iter_min (%u) is greater than iter_max (%u)", + turbo_dec->iter_min, turbo_dec->iter_max); + return -1; + } + if (turbo_dec->code_block_mode != 0 && + turbo_dec->code_block_mode != 1) { + rte_bbdev_log(ERR, + "code_block_mode (%u) is out of range 0 <= value <= 1", + turbo_dec->code_block_mode); + return -1; + } + + if (turbo_dec->code_block_mode == 0) { + + if ((turbo_dec->op_flags & + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP) && + !(turbo_dec->op_flags & RTE_BBDEV_TURBO_CRC_TYPE_24B)) { + rte_bbdev_log(ERR, + "RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP should accompany RTE_BBDEV_TURBO_CRC_TYPE_24B"); + return -1; + } + + tb = &turbo_dec->tb_params; + if ((tb->k_neg < RTE_BBDEV_MIN_CB_SIZE + || tb->k_neg > RTE_BBDEV_MAX_CB_SIZE) + && tb->c_neg > 0) { + rte_bbdev_log(ERR, + "k_neg (%u) is out of range %u <= value <= %u", + tb->k_neg, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if ((tb->k_pos < RTE_BBDEV_MIN_CB_SIZE + || tb->k_pos > RTE_BBDEV_MAX_CB_SIZE) + && tb->c > tb->c_neg) { + rte_bbdev_log(ERR, + "k_pos (%u) is out of range %u <= value <= %u", + tb->k_pos, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->c_neg > (RTE_BBDEV_MAX_CODE_BLOCKS - 1)) + rte_bbdev_log(ERR, + "c_neg (%u) is out of range 0 <= value <= %u", + tb->c_neg, + RTE_BBDEV_MAX_CODE_BLOCKS - 1); + if (tb->c < 1 || tb->c > RTE_BBDEV_MAX_CODE_BLOCKS) { + rte_bbdev_log(ERR, + "c (%u) is out of range 1 <= value <= %u", + tb->c, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + if (tb->cab > tb->c) { + rte_bbdev_log(ERR, + "cab (%u) is greater than c (%u)", + tb->cab, tb->c); + return -1; + } + } else { + + if (turbo_dec->op_flags & RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP) { + rte_bbdev_log(ERR, + "RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP is invalid in CB-mode"); + return -1; + } + + cb = &turbo_dec->cb_params; + if (cb->k < RTE_BBDEV_MIN_CB_SIZE + || cb->k > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k (%u) is out of range %u <= value <= %u", + cb->k, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + } + + return 0; +} +#endif + +static inline int +enqueue_dec_one_op_cb(struct fpga_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input; + struct rte_mbuf *output; + int ret; + uint16_t k, kw, ring_offset; + uint32_t total_left, in_length, out_length, in_offset, out_offset; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_dec_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo decoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_dec.input.data; + output = op->turbo_dec.hard_output.data; + total_left = op->turbo_dec.input.length; + in_offset = op->turbo_dec.input.offset; + out_offset = op->turbo_dec.hard_output.offset; + + k = op->turbo_dec.cb_params.k; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + in_length = kw; + out_length = k >> 3; + + mbuf_append(output, output, out_length); + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_td_fill(op, &desc->dec_req, input, output, + in_length, k, in_offset, out_offset, ring_offset, 1); + if (unlikely(ret < 0)) + return ret; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_dma_dec_desc_debug_info(desc); +#endif + + /* Update lengths */ + total_left -= in_length; + op->turbo_dec.hard_output.length += out_length; + + if (total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + total_left, in_length); + return -1; + } + + return 1; +} + + +static inline int +enqueue_dec_one_op_tb(struct fpga_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset, uint8_t cbs_in_op) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input, *output_head, *output; + int ret; + uint8_t r, c; + uint16_t k, kw, in_length, out_length, ring_offset; + uint32_t mbuf_total_left, seg_total_left, in_offset, out_offset; + uint16_t current_enqueued_cbs = 0; + uint16_t crc24_overlap = 0; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_dec_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo decoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_dec.input.data; + output_head = output = op->turbo_dec.hard_output.data; + mbuf_total_left = op->turbo_dec.input.length; + in_offset = op->turbo_dec.input.offset; + out_offset = op->turbo_dec.hard_output.offset; + + if (!check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP)) + crc24_overlap = 24; + + c = op->turbo_dec.tb_params.c; + r = op->turbo_dec.tb_params.r; + + while (mbuf_total_left > 0 && r < c && input != NULL) { + seg_total_left = rte_pktmbuf_data_len(input) - in_offset; + k = (r < op->turbo_dec.tb_params.c_neg) ? + op->turbo_dec.tb_params.k_neg : + op->turbo_dec.tb_params.k_pos; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + + in_length = kw; + out_length = (k - crc24_overlap) >> 3; + + mbuf_append(output_head, output, out_length); + + if (seg_total_left < in_length) { + rte_bbdev_log(ERR, + "Partial CB found in a TB. FPGA Driver doesn't support scatter-gather operations!"); + return -1; + } + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_td_fill(op, &desc->dec_req, input, output, + in_length, k, in_offset, out_offset, + ring_offset, cbs_in_op); + if (unlikely(ret < 0)) + return ret; + + /* Update lengths */ + ret = rte_pktmbuf_trim(op->turbo_dec.hard_output.data, + (crc24_overlap >> 3)); +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (ret < 0) { + rte_bbdev_log(ERR, + "The length to remove is greater than the length of the last segment"); + return -EINVAL; + } +#endif + op->turbo_dec.hard_output.length += out_length; + mbuf_total_left -= in_length; + + /* Update offsets */ + if (seg_total_left == in_length) { + /* Go to the next mbuf */ + input = input->next; + output = output->next; + in_offset = 0; + out_offset = 0; + } else { + in_offset += in_length; + out_offset += out_length; + } + + r++; + desc_offset++; + current_enqueued_cbs++; + } + + if (mbuf_total_left > 0) { + rte_bbdev_log(ERR, + "Some date still left for processing: mbuf_total_left = %u", + mbuf_total_left); + return -1; + } + + return current_enqueued_cbs; +} + +static uint16_t +fpga_enqueue_enc(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num) +{ + uint8_t cbs_in_op; + uint16_t i, total_enqueued_cbs = 0; + int32_t avail; + int enqueued_cbs; + struct fpga_queue *q = q_data->queue_private; + union fpga_dma_desc *desc; + + /* Check if queue is not full */ + if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == + q->head_free_desc)) + return 0; + + /* Calculates available space */ + avail = (q->head_free_desc > q->tail) ? + q->head_free_desc - q->tail - 1 : + q->ring_ctrl_reg.ring_size + q->head_free_desc - q->tail - 1; + + for (i = 0; i < num; ++i) { + if (ops[i]->turbo_enc.code_block_mode == 0) { + cbs_in_op = get_num_cbs_in_op_enc(&ops[i]->turbo_enc); + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - cbs_in_op < 0)) + break; + avail -= cbs_in_op; + enqueued_cbs = enqueue_enc_one_op_tb(q, ops[i], + total_enqueued_cbs, cbs_in_op); + } else { + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - 1 < 0)) + break; + avail -= 1; + enqueued_cbs = enqueue_enc_one_op_cb(q, ops[i], + total_enqueued_cbs); + } + + if (enqueued_cbs < 0) + break; + + total_enqueued_cbs += enqueued_cbs; + + rte_bbdev_log_debug("enqueuing enc ops [%d/%d] | head %d | tail %d", + total_enqueued_cbs, num, + q->head_free_desc, q->tail); + } + + /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt + * only when all previous CBs were already processed. + */ + desc = q->ring_addr + ((q->tail + total_enqueued_cbs - 1) + & q->sw_ring_wrap_mask); + desc->enc_req.irq_en = q->irq_enable; + + fpga_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); + + /* Update stats */ + q_data->queue_stats.enqueued_count += i; + q_data->queue_stats.enqueue_err_count += num - i; + + return i; +} + +static uint16_t +fpga_enqueue_dec(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num) +{ + uint8_t cbs_in_op; + uint16_t i, total_enqueued_cbs = 0; + int32_t avail; + int enqueued_cbs; + struct fpga_queue *q = q_data->queue_private; + union fpga_dma_desc *desc; + + /* Check if queue is not full */ + if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == + q->head_free_desc)) + return 0; + + /* Calculates available space */ + avail = (q->head_free_desc > q->tail) ? + q->head_free_desc - q->tail - 1 : + q->ring_ctrl_reg.ring_size + q->head_free_desc - q->tail - 1; + + for (i = 0; i < num; ++i) { + if (ops[i]->turbo_dec.code_block_mode == 0) { + cbs_in_op = get_num_cbs_in_op_dec(&ops[i]->turbo_dec); + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - cbs_in_op < 0)) + break; + avail -= cbs_in_op; + enqueued_cbs = enqueue_dec_one_op_tb(q, ops[i], + total_enqueued_cbs, cbs_in_op); + } else { + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - 1 < 0)) + break; + avail -= 1; + enqueued_cbs = enqueue_dec_one_op_cb(q, ops[i], + total_enqueued_cbs); + } + + if (enqueued_cbs < 0) + break; + + total_enqueued_cbs += enqueued_cbs; + + rte_bbdev_log_debug("enqueuing dec ops [%d/%d] | head %d | tail %d", + total_enqueued_cbs, num, + q->head_free_desc, q->tail); + } + + /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt + * only when all previous CBs were already processed. + */ + desc = q->ring_addr + ((q->tail + total_enqueued_cbs - 1) + & q->sw_ring_wrap_mask); + desc->dec_req.irq_en = q->irq_enable; + + fpga_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); + + /* Update stats */ + q_data->queue_stats.enqueued_count += i; + q_data->queue_stats.enqueue_err_count += num - i; + + return i; +} + +static inline int +dequeue_enc_one_op_cb(struct fpga_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + int desc_error = 0; + + /* Set current desc */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /*check if done */ + if (desc->enc_req.done == 0) + return -1; + + /* make sure the response is read atomically */ + rte_smp_rmb(); + + rte_bbdev_log_debug("DMA response desc %p", desc); + + *op = desc->enc_req.op_addr; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + (*op)->status = desc_error << RTE_BBDEV_DATA_ERROR; + + return 1; +} + +static inline int +dequeue_enc_one_op_tb(struct fpga_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + uint8_t cbs_in_op, cb_idx; + int desc_error = 0; + int status = 0; + + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify if done bit is set */ + if (desc->enc_req.done == 0) + return -1; + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + /* Verify if done bit in all CBs is set */ + cbs_in_op = desc->enc_req.cbs_in_op; + for (cb_idx = 1; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + if (desc->enc_req.done == 0) + return -1; + } + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + for (cb_idx = 0; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + status |= desc_error << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log_debug("DMA response desc %p", desc); + } + + *op = desc->enc_req.op_addr; + (*op)->status = status; + return cbs_in_op; +} + +static inline int +dequeue_dec_one_op_cb(struct fpga_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + int desc_error = 0; + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify done bit is set */ + if (desc->dec_req.done == 0) + return -1; + + /* make sure the response is read atomically */ + rte_smp_rmb(); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_dma_dec_desc_debug_info(desc); + +#endif + + *op = desc->dec_req.op_addr; + /* FPGA reports in half-iterations, from 0 to 31. get ceiling */ + (*op)->turbo_dec.iter_count = (desc->dec_req.iter + 2) >> 1; + /* crc_pass = 0 when decoder fails */ + (*op)->status = !(desc->dec_req.crc_pass) << RTE_BBDEV_CRC_ERROR; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + (*op)->status |= desc_error << RTE_BBDEV_DATA_ERROR; + return 1; +} + +static inline int +dequeue_dec_one_op_tb(struct fpga_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + uint8_t cbs_in_op, cb_idx, iter_count = 0; + int status = 0; + int desc_error = 0; + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify if done bit is set */ + if (desc->dec_req.done == 0) + return -1; + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + /* Verify if done bit in all CBs is set */ + cbs_in_op = desc->dec_req.cbs_in_op; + for (cb_idx = 1; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + if (desc->dec_req.done == 0) + return -1; + } + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + for (cb_idx = 0; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + /* get max iter_count for all CBs in op */ + iter_count = RTE_MAX(iter_count, (uint8_t) desc->dec_req.iter); + /* crc_pass = 0 when decoder fails, one fails all */ + status |= !(desc->dec_req.crc_pass) << RTE_BBDEV_CRC_ERROR; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + status |= desc_error << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log_debug("DMA response desc %p", desc); + } + + *op = desc->dec_req.op_addr; + + /* FPGA reports in half-iterations, get ceiling */ + (*op)->turbo_dec.iter_count = (iter_count + 2) >> 1; + (*op)->status = status; + return cbs_in_op; +} + +static uint16_t +fpga_dequeue_enc(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num) +{ + struct fpga_queue *q = q_data->queue_private; + uint32_t avail = (q->tail - q->head_free_desc) & q->sw_ring_wrap_mask; + uint16_t i; + uint16_t dequeued_cbs = 0; + struct rte_bbdev_enc_op *op; + int ret; + + for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { + op = (q->ring_addr + ((q->head_free_desc + dequeued_cbs) + & q->sw_ring_wrap_mask))->enc_req.op_addr; + if (op->turbo_enc.code_block_mode == 0) + ret = dequeue_enc_one_op_tb(q, &ops[i], dequeued_cbs); + else + ret = dequeue_enc_one_op_cb(q, &ops[i], dequeued_cbs); + + if (ret < 0) + break; + + dequeued_cbs += ret; + + rte_bbdev_log_debug("dequeuing enc ops [%d/%d] | head %d | tail %d", + dequeued_cbs, num, q->head_free_desc, q->tail); + } + + /* Update head */ + q->head_free_desc = (q->head_free_desc + dequeued_cbs) & + q->sw_ring_wrap_mask; + + /* Update stats */ + q_data->queue_stats.dequeued_count += i; + + return i; +} + +static uint16_t +fpga_dequeue_dec(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num) +{ + struct fpga_queue *q = q_data->queue_private; + uint32_t avail = (q->tail - q->head_free_desc) & q->sw_ring_wrap_mask; + uint16_t i; + uint16_t dequeued_cbs = 0; + struct rte_bbdev_dec_op *op; + int ret; + + for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { + op = (q->ring_addr + ((q->head_free_desc + dequeued_cbs) + & q->sw_ring_wrap_mask))->dec_req.op_addr; + if (op->turbo_dec.code_block_mode == 0) + ret = dequeue_dec_one_op_tb(q, &ops[i], dequeued_cbs); + else + ret = dequeue_dec_one_op_cb(q, &ops[i], dequeued_cbs); + + if (ret < 0) + break; + + dequeued_cbs += ret; + + rte_bbdev_log_debug("dequeuing dec ops [%d/%d] | head %d | tail %d", + dequeued_cbs, num, q->head_free_desc, q->tail); + } + + /* Update head */ + q->head_free_desc = (q->head_free_desc + dequeued_cbs) & + q->sw_ring_wrap_mask; + + /* Update stats */ + q_data->queue_stats.dequeued_count += i; + + return i; +} + +/* Initialization Function */ +static void +fpga_lte_fec_init(struct rte_bbdev *dev) +{ + struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev->device); + + dev->dev_ops = &fpga_ops; + dev->enqueue_enc_ops = fpga_enqueue_enc; + dev->enqueue_dec_ops = fpga_enqueue_dec; + dev->dequeue_enc_ops = fpga_dequeue_enc; + dev->dequeue_dec_ops = fpga_dequeue_dec; + + ((struct fpga_lte_fec_device *) dev->data->dev_private)->pf_device = + !strcmp(dev->device->driver->name, + RTE_STR(FPGA_LTE_FEC_PF_DRIVER_NAME)); + ((struct fpga_lte_fec_device *) dev->data->dev_private)->mmio_base = + pci_dev->mem_resource[0].addr; + + rte_bbdev_log_debug( + "Init device %s [%s] @ virtaddr %p phyaddr %#"PRIx64, + dev->device->driver->name, dev->data->name, + (void *)pci_dev->mem_resource[0].addr, + pci_dev->mem_resource[0].phys_addr); +} + +static int +fpga_lte_fec_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct rte_bbdev *bbdev = NULL; + char dev_name[RTE_BBDEV_NAME_MAX_LEN]; + + if (pci_dev == NULL) { + rte_bbdev_log(ERR, "NULL PCI device"); + return -EINVAL; + } + + rte_pci_device_name(&pci_dev->addr, dev_name, sizeof(dev_name)); + + /* Allocate memory to be used privately by drivers */ + bbdev = rte_bbdev_allocate(pci_dev->device.name); + if (bbdev == NULL) + return -ENODEV; + + /* allocate device private memory */ + bbdev->data->dev_private = rte_zmalloc_socket(dev_name, + sizeof(struct fpga_lte_fec_device), RTE_CACHE_LINE_SIZE, + pci_dev->device.numa_node); + + if (bbdev->data->dev_private == NULL) { + rte_bbdev_log(CRIT, + "Allocate of %zu bytes for device \"%s\" failed", + sizeof(struct fpga_lte_fec_device), dev_name); + rte_bbdev_release(bbdev); + return -ENOMEM; + } + + /* Fill HW specific part of device structure */ + bbdev->device = &pci_dev->device; + bbdev->intr_handle = &pci_dev->intr_handle; + bbdev->data->socket_id = pci_dev->device.numa_node; + + /* Invoke FEC FPGA device initialization function */ + fpga_lte_fec_init(bbdev); + + rte_bbdev_log_debug("bbdev id = %u [%s]", + bbdev->data->dev_id, dev_name); + + struct fpga_lte_fec_device *d = bbdev->data->dev_private; + uint32_t version_id = fpga_reg_read_32(d->mmio_base, + FPGA_LTE_FEC_VERSION_ID); + rte_bbdev_log(INFO, "FEC FPGA RTL v%u.%u", + ((uint16_t)(version_id >> 16)), ((uint16_t)version_id)); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (!strcmp(bbdev->device->driver->name, + RTE_STR(FPGA_LTE_FEC_PF_DRIVER_NAME))) + print_static_reg_debug_info(d->mmio_base); +#endif + return 0; +} + +static int +fpga_lte_fec_remove(struct rte_pci_device *pci_dev) +{ + struct rte_bbdev *bbdev; + int ret; + uint8_t dev_id; + + if (pci_dev == NULL) + return -EINVAL; + + /* Find device */ + bbdev = rte_bbdev_get_named_dev(pci_dev->device.name); + if (bbdev == NULL) { + rte_bbdev_log(CRIT, + "Couldn't find HW dev \"%s\" to uninitialise it", + pci_dev->device.name); + return -ENODEV; + } + dev_id = bbdev->data->dev_id; + + /* free device private memory before close */ + rte_free(bbdev->data->dev_private); + + /* Close device */ + ret = rte_bbdev_close(dev_id); + if (ret < 0) + rte_bbdev_log(ERR, + "Device %i failed to close during uninit: %i", + dev_id, ret); + + /* release bbdev from library */ + ret = rte_bbdev_release(bbdev); + if (ret) + rte_bbdev_log(ERR, "Device %i failed to uninit: %i", dev_id, + ret); + + rte_bbdev_log_debug("Destroyed bbdev = %u", dev_id); + + return 0; +} + +static inline void +set_default_fpga_conf(struct fpga_lte_fec_conf *def_conf) +{ + /* clear default configuration before initialization */ + memset(def_conf, 0, sizeof(struct fpga_lte_fec_conf)); + /* Set pf mode to true */ + def_conf->pf_mode_en = true; + + /* Set ratio between UL and DL to 1:1 (unit of weight is 3 CBs) */ + def_conf->ul_bandwidth = 3; + def_conf->dl_bandwidth = 3; + + /* Set Load Balance Factor to 64 */ + def_conf->dl_load_balance = 64; + def_conf->ul_load_balance = 64; +} + +/* Initial configuration of FPGA LTE FEC device */ +int +fpga_lte_fec_configure(const char *dev_name, + const struct fpga_lte_fec_conf *conf) +{ + uint32_t payload_32, address; + uint16_t payload_16; + uint8_t payload_8; + uint16_t q_id, vf_id, total_q_id, total_ul_q_id, total_dl_q_id; + struct rte_bbdev *bbdev = rte_bbdev_get_named_dev(dev_name); + struct fpga_lte_fec_conf def_conf; + + if (bbdev == NULL) { + rte_bbdev_log(ERR, + "Invalid dev_name (%s), or device is not yet initialised", + dev_name); + return -ENODEV; + } + + struct fpga_lte_fec_device *d = bbdev->data->dev_private; + + if (conf == NULL) { + rte_bbdev_log(ERR, + "FPGA Configuration was not provided. Default configuration will be loaded."); + set_default_fpga_conf(&def_conf); + conf = &def_conf; + } + + /* + * Configure UL:DL ratio. + * [7:0]: UL weight + * [15:8]: DL weight + */ + payload_16 = (conf->dl_bandwidth << 8) | conf->ul_bandwidth; + address = FPGA_LTE_FEC_CONFIGURATION; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Clear all queues registers */ + payload_32 = FPGA_INVALID_HW_QUEUE_ID; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + address = (q_id << 2) + FPGA_LTE_FEC_QUEUE_MAP; + fpga_reg_write_32(d->mmio_base, address, payload_32); + } + + /* + * If PF mode is enabled allocate all queues for PF only. + * + * For VF mode each VF can have different number of UL and DL queues. + * Total number of queues to configure cannot exceed FPGA + * capabilities - 64 queues - 32 queues for UL and 32 queues for DL. + * Queues mapping is done according to configuration: + * + * UL queues: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | ... | 0 | + * | conf->vf_dl_queues_number[0] - 1 | 0 | + * | conf->vf_dl_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_dl_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_dl_queues_number[7] - 1 | 7 | + * + * DL queues: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | ... | 0 | + * | conf->vf_ul_queues_number[0] - 1 | 0 | + * | conf->vf_ul_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_ul_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_ul_queues_number[7] - 1 | 7 | + * + * Example of configuration: + * conf->vf_ul_queues_number[0] = 4; -> 4 UL queues for VF0 + * conf->vf_dl_queues_number[0] = 4; -> 4 DL queues for VF0 + * conf->vf_ul_queues_number[1] = 2; -> 2 UL queues for VF1 + * conf->vf_dl_queues_number[1] = 2; -> 2 DL queues for VF1 + * + * UL: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | 1 | 0 | + * | 2 | 0 | + * | 3 | 0 | + * | 4 | 1 | + * | 5 | 1 | + * + * DL: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | 33 | 0 | + * | 34 | 0 | + * | 35 | 0 | + * | 36 | 1 | + * | 37 | 1 | + */ + if (conf->pf_mode_en) { + payload_32 = 0x1; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + address = (q_id << 2) + FPGA_LTE_FEC_QUEUE_MAP; + fpga_reg_write_32(d->mmio_base, address, payload_32); + } + } else { + /* Calculate total number of UL and DL queues to configure */ + total_ul_q_id = total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + total_ul_q_id += conf->vf_ul_queues_number[vf_id]; + total_dl_q_id += conf->vf_dl_queues_number[vf_id]; + } + total_q_id = total_dl_q_id + total_ul_q_id; + /* + * Check if total number of queues to configure does not exceed + * FPGA capabilities (64 queues - 32 UL and 32 DL queues) + */ + if ((total_ul_q_id > FPGA_NUM_UL_QUEUES) || + (total_dl_q_id > FPGA_NUM_DL_QUEUES) || + (total_q_id > FPGA_TOTAL_NUM_QUEUES)) { + rte_bbdev_log(ERR, + "FPGA Configuration failed. Too many queues to configure: UL_Q %u, DL_Q %u, FPGA_Q %u", + total_ul_q_id, total_dl_q_id, + FPGA_TOTAL_NUM_QUEUES); + return -EINVAL; + } + total_ul_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_ul_queues_number[vf_id]; + ++q_id, ++total_ul_q_id) { + address = (total_ul_q_id << 2) + + FPGA_LTE_FEC_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_reg_write_32(d->mmio_base, address, + payload_32); + } + } + total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_dl_queues_number[vf_id]; + ++q_id, ++total_dl_q_id) { + address = ((total_dl_q_id + FPGA_NUM_UL_QUEUES) + << 2) + FPGA_LTE_FEC_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_reg_write_32(d->mmio_base, address, + payload_32); + } + } + } + + /* Setting Load Balance Factor */ + payload_16 = (conf->dl_load_balance << 8) | (conf->ul_load_balance); + address = FPGA_LTE_FEC_LOAD_BALANCE_FACTOR; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting length of ring descriptor entry */ + payload_16 = FPGA_RING_DESC_ENTRY_LENGTH; + address = FPGA_LTE_FEC_RING_DESC_LEN; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting FLR timeout value */ + payload_16 = conf->flr_time_out; + address = FPGA_LTE_FEC_FLR_TIME_OUT; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Queue PF/VF mapping table is ready */ + payload_8 = 0x1; + address = FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE; + fpga_reg_write_8(d->mmio_base, address, payload_8); + + rte_bbdev_log_debug("PF FPGA LTE FEC configuration complete for %s", + dev_name); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_static_reg_debug_info(d->mmio_base); +#endif + return 0; +} + +/* FPGA LTE FEC PCI PF address map */ +static struct rte_pci_id pci_id_fpga_lte_fec_pf_map[] = { + { + RTE_PCI_DEVICE(FPGA_LTE_FEC_VENDOR_ID, + FPGA_LTE_FEC_PF_DEVICE_ID) + }, + {.device_id = 0}, +}; + +static struct rte_pci_driver fpga_lte_fec_pci_pf_driver = { + .probe = fpga_lte_fec_probe, + .remove = fpga_lte_fec_remove, + .id_table = pci_id_fpga_lte_fec_pf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING +}; + +/* FPGA LTE FEC PCI VF address map */ +static struct rte_pci_id pci_id_fpga_lte_fec_vf_map[] = { + { + RTE_PCI_DEVICE(FPGA_LTE_FEC_VENDOR_ID, + FPGA_LTE_FEC_VF_DEVICE_ID) + }, + {.device_id = 0}, +}; + +static struct rte_pci_driver fpga_lte_fec_pci_vf_driver = { + .probe = fpga_lte_fec_probe, + .remove = fpga_lte_fec_remove, + .id_table = pci_id_fpga_lte_fec_vf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING +}; + + +RTE_PMD_REGISTER_PCI(FPGA_LTE_FEC_PF_DRIVER_NAME, fpga_lte_fec_pci_pf_driver); +RTE_PMD_REGISTER_PCI_TABLE(FPGA_LTE_FEC_PF_DRIVER_NAME, + pci_id_fpga_lte_fec_pf_map); +RTE_PMD_REGISTER_PCI(FPGA_LTE_FEC_VF_DRIVER_NAME, fpga_lte_fec_pci_vf_driver); +RTE_PMD_REGISTER_PCI_TABLE(FPGA_LTE_FEC_VF_DRIVER_NAME, + pci_id_fpga_lte_fec_vf_map); + +RTE_INIT(fpga_lte_fec_init_log) +{ + fpga_lte_fec_logtype = rte_log_register("pmd.bb.fpga_lte_fec"); + if (fpga_lte_fec_logtype >= 0) +#ifdef RTE_LIBRTE_BBDEV_DEBUG + rte_log_set_level(fpga_lte_fec_logtype, RTE_LOG_DEBUG); +#else + rte_log_set_level(fpga_lte_fec_logtype, RTE_LOG_NOTICE); +#endif +} diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h new file mode 100644 index 0000000..9ae8b12 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation + */ + +#ifndef _FPGA_LTE_FEC_H_ +#define _FPGA_LTE_FEC_H_ + +#include <stdint.h> +#include <stdbool.h> + +/** + * @file fpga_lte_fec.h + * + * Interface for Intel(R) FGPA LTE FEC device configuration at the host level, + * directly accessible by the application. + * Configuration related to LTE Turbo coding functionality is done through + * librte_bbdev library. + * + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + */ + +#ifdef __cplusplus +extern "C" { +#endif + +/**< Number of Virtual Functions FGPA 4G FEC supports */ +#define FPGA_LTE_FEC_NUM_VFS 8 + +/** + * Structure to pass FPGA 4G FEC configuration. + */ +struct fpga_lte_fec_conf { + /**< 1 if PF is used for dataplane, 0 for VFs */ + bool pf_mode_en; + /**< Number of UL queues per VF */ + uint8_t vf_ul_queues_number[FPGA_LTE_FEC_NUM_VFS]; + /**< Number of DL queues per VF */ + uint8_t vf_dl_queues_number[FPGA_LTE_FEC_NUM_VFS]; + /**< UL bandwidth. Needed for schedule algorithm */ + uint8_t ul_bandwidth; + /**< DL bandwidth. Needed for schedule algorithm */ + uint8_t dl_bandwidth; + /**< UL Load Balance */ + uint8_t ul_load_balance; + /**< DL Load Balance */ + uint8_t dl_load_balance; + /**< FLR timeout value */ + uint16_t flr_time_out; +}; + +/** + * Configure Intel(R) FPGA LTE FEC device + * + * @param dev_name + * The name of the device. This is the short form of PCI BDF, e.g. 00:01.0. + * It can also be retrieved for a bbdev device from the dev_name field in the + * rte_bbdev_info structure returned by rte_bbdev_info_get(). + * @param conf + * Configuration to apply to FPGA 4G FEC. + * + * @return + * Zero on success, negative value on failure. + */ +int +fpga_lte_fec_configure(const char *dev_name, + const struct fpga_lte_fec_conf *conf); + +#ifdef __cplusplus +} +#endif + +#endif /* _FPGA_LTE_FEC_H_ */ diff --git a/drivers/baseband/fpga_lte_fec/meson.build b/drivers/baseband/fpga_lte_fec/meson.build new file mode 100644 index 0000000..bf44e6b --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +deps += ['bbdev', 'bus_vdev', 'ring', 'pci', 'bus_pci'] +name = 'bbdev_fpga_lte_fec' +allow_experimental_apis = true +sources = files('fpga_lte_fec.c') diff --git a/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map new file mode 100644 index 0000000..e923270 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map @@ -0,0 +1,3 @@ +DPDK_19.08 { + local: *; +}; diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build index 52489df..a17d549 100644 --- a/drivers/baseband/meson.build +++ b/drivers/baseband/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <bluca@debian.org> -drivers = ['null'] +drivers = ['null', 'fpga_lte_fec'] config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' driver_name_fmt = 'rte_pmd_@0@' diff --git a/mk/rte.app.mk b/mk/rte.app.mk index b6106e1..73bba4a 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -215,6 +215,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_NETVSC_PMD) += -lrte_pmd_netvsc ifeq ($(CONFIG_RTE_LIBRTE_BBDEV),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += -lrte_pmd_fpga_lte_fec # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v6] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-10 17:01 ` Nicolas Chautru [not found] ` <EEA9FF629BF25B47BD67ADE995041EE2496A888B@IRSMSX103.ger.corp.intel.com> 2019-06-13 17:28 ` [dpdk-dev] [PATCH v5] " Nicolas Chautru @ 2019-06-14 16:12 ` Nicolas Chautru 2019-06-14 16:12 ` [dpdk-dev] [PATCH v5] " Nicolas Chautru 2019-06-14 16:17 ` [dpdk-dev] [PATCH v6] " Nicolas Chautru 3 siblings, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-14 16:12 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, Nicolas Chautru Update v6: Update of one copyright date in documentation file. Update v5: Update date and version from dpdk review. Rebased to latest. Update v4: Fix warning for the DEBUG configuration. Update v3: Squashing 3 previous patches into one as recommended. This is adding a new PMD driver for BBDEV device based on FPGA implementation on PAC N3000 HW to provide FEC 4G acceleration. v1 was shared earlier on this patchwork : https://patches.dpdk.org/patch/53409/ The existing BBDEV test framework is still supported. Main updates based on feedback during the last v1 were: - correct the documentation which was arguably misleading in hinting that ICC was required. There is no dependency whatsoever. - add meson build support Note that a number of other BBDEV patches are in v1 here and will be resubmitted for 19.08 rc1 https://patches.dpdk.org/project/dpdk/list/?series=4657 Nicolas Chautru (1): baseband/fpga_lte_fec: adding driver for FEC on FPGA config/common_base | 6 + doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++ doc/guides/bbdevs/index.rst | 1 + drivers/baseband/Makefile | 2 + drivers/baseband/fpga_lte_fec/Makefile | 29 + drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 ++++++++++++++++++++ drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + drivers/baseband/fpga_lte_fec/meson.build | 7 + .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + drivers/baseband/meson.build | 2 +- mk/rte.app.mk | 1 + 11 files changed, 3115 insertions(+), 1 deletion(-) create mode 100644 doc/guides/bbdevs/fpga_lte_fec.rst create mode 100644 drivers/baseband/fpga_lte_fec/Makefile create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h create mode 100644 drivers/baseband/fpga_lte_fec/meson.build create mode 100644 drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v5] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-14 16:12 ` [dpdk-dev] [PATCH v6] " Nicolas Chautru @ 2019-06-14 16:12 ` Nicolas Chautru 0 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-14 16:12 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, Nicolas Chautru Supports for FEC 4G PMD Driver on FPGA card PAC N3000 Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 6 + doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++ doc/guides/bbdevs/index.rst | 1 + drivers/baseband/Makefile | 2 + drivers/baseband/fpga_lte_fec/Makefile | 29 + drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 ++++++++++++++++++++ drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + drivers/baseband/fpga_lte_fec/meson.build | 7 + .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + drivers/baseband/meson.build | 2 +- mk/rte.app.mk | 1 + 11 files changed, 3115 insertions(+), 1 deletion(-) create mode 100644 doc/guides/bbdevs/fpga_lte_fec.rst create mode 100644 drivers/baseband/fpga_lte_fec/Makefile create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h create mode 100644 drivers/baseband/fpga_lte_fec/meson.build create mode 100644 drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map diff --git a/config/common_base b/config/common_base index 6f19ad5..9632a07 100644 --- a/config/common_base +++ b/config/common_base @@ -521,6 +521,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y # EXPERIMENTAL: API may change without prior notice # CONFIG_RTE_LIBRTE_BBDEV=y +CONFIG_RTE_LIBRTE_BBDEV_DEBUG=n CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y @@ -535,6 +536,11 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n # +# Compile PMD for Intel FPGA LTE FEC bbdev device +# +CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC=y + +# # Compile generic crypto device library # CONFIG_RTE_LIBRTE_CRYPTODEV=y diff --git a/doc/guides/bbdevs/fpga_lte_fec.rst b/doc/guides/bbdevs/fpga_lte_fec.rst new file mode 100644 index 0000000..71b058c --- /dev/null +++ b/doc/guides/bbdevs/fpga_lte_fec.rst @@ -0,0 +1,318 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation + +Intel(R) FPGA LTE FEC Poll Mode Driver +====================================== + +The BBDEV FPGA LTE FEC poll mode driver (PMD) supports an FPGA implementation of a VRAN +Turbo Encode / Decode LTE wireless acceleration function, using Intel's PCI-e and FPGA +based Vista Creek device. + +Features +-------- + +FPGA LTE FEC PMD supports the following features: + +- Turbo Encode in the DL with total throughput of 4.5 Gbits/s +- Turbo Decode in the UL with total throughput of 1.5 Gbits/s assuming 8 decoder iterations +- 8 VFs per PF (physical device) +- Maximum of 32 UL queues per VF +- Maximum of 32 DL queues per VF +- PCIe Gen-3 x8 Interface +- MSI-X +- SR-IOV + + +FPGA LTE FEC PMD supports the following BBDEV capabilities: + +* For the turbo encode operation: + - ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` : set to attach CRC24B to CB(s) + - ``RTE_BBDEV_TURBO_RATE_MATCH`` : if set then do not do Rate Match bypass + - ``RTE_BBDEV_TURBO_ENC_INTERRUPTS`` : set for encoder dequeue interrupts + + +* For the turbo decode operation: + - ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` : check CRC24B from CB(s) + - ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` : perform subblock de-interleave + - ``RTE_BBDEV_TURBO_DEC_INTERRUPTS`` : set for decoder dequeue interrupts + - ``RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN`` : set if negative LLR encoder i/p is supported + - ``RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP`` : keep CRC24B bits appended while decoding + + +Limitations +----------- + +FPGA LTE FEC does not support the following: + +- Scatter-Gather function + + +Installation +-------------- + +Section 3 of the DPDK manual provides instuctions on installing and compiling DPDK. The +default set of bbdev compile flags may be found in config/common_base, where for example +the flag to build the FPGA LTE FEC device, ``CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC``, is already +set. It is assumed DPDK has been compiled using for instance: + +.. code-block:: console + + make install T=x86_64-native-linuxapp-gcc + + +DPDK requires hugepages to be configured as detailed in section 2 of the DPDK manual. +The bbdev test application has been tested with a configuration 40 x 1GB hugepages. The +hugepage configuration of a server may be examined using: + +.. code-block:: console + + grep Huge* /proc/meminfo + + +Initialization +-------------- + +When the device first powers up, its PCI Physical Functions (PF) can be listed through this command: + +.. code-block:: console + + sudo lspci -vd1172:5052 + +The physical and virtual functions are compatible with Linux UIO drivers: +``vfio`` and ``igb_uio``. However, in order to work the FPGA LTE FEC device firstly needs +to be bound to one of these linux drivers through DPDK. + + +Bind PF UIO driver(s) +~~~~~~~~~~~~~~~~~~~~~ + +Install the DPDK igb_uio driver, bind it with the PF PCI device ID and use +``lspci`` to confirm the PF device is under use by ``igb_uio`` DPDK UIO driver. + +The igb_uio driver may be bound to the PF PCI device using one of three methods: + + +1. PCI functions (physical or virtual, depending on the use case) can be bound to +the UIO driver by repeating this command for every function. + +.. code-block:: console + + cd <dpdk-top-level-directory> + insmod ./build/kmod/igb_uio.ko + echo "1172 5052" > /sys/bus/pci/drivers/igb_uio/new_id + lspci -vd1172: + + +2. Another way to bind PF with DPDK UIO driver is by using the ``dpdk-devbind.py`` tool + +.. code-block:: console + + cd <dpdk-top-level-directory> + ./usertools/dpdk-devbind.py -b igb_uio 0000:06:00.0 + +where the PCI device ID (example: 0000:06:00.0) is obtained using lspci -vd1172: + + +3. A third way to bind is to use ``dpdk-setup.sh`` tool + +.. code-block:: console + + cd <dpdk-top-level-directory> + ./usertools/dpdk-setup.sh + + select 'Bind Ethernet/Crypto/Baseband device to IGB UIO module' + or + select 'Bind Ethernet/Crypto/Baseband device to VFIO module' depending on driver required + enter PCI device ID + select 'Display current Ethernet/Crypto/Baseband device settings' to confirm binding + + +In the same way the FPGA LTE FEC PF can be bound with vfio, but vfio driver does not +support SR-IOV configuration right out of the box, so it will need to be patched. + + +Enable Virtual Functions +~~~~~~~~~~~~~~~~~~~~~~~~ + +Now, it should be visible in the printouts that PCI PF is under igb_uio control +"``Kernel driver in use: igb_uio``" + +To show the number of available VFs on the device, read ``sriov_totalvfs`` file.. + +.. code-block:: console + + cat /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_totalvfs + + where 0000\:<b>\:<d>.<f> is the PCI device ID + + +To enable VFs via igb_uio, echo the number of virtual functions intended to +enable to ``max_vfs`` file.. + +.. code-block:: console + + echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/max_vfs + + +Afterwards, all VFs must be bound to appropriate UIO drivers as required, same +way it was done with the physical function previously. + +Enabling SR-IOV via vfio driver is pretty much the same, except that the file +name is different: + +.. code-block:: console + + echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_numvfs + + +Configure the VFs through PF +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The PCI virtual functions must be configured before working or getting assigned +to VMs/Containers. The configuration involves allocating the number of hardware +queues, priorities, load balance, bandwidth and other settings necessary for the +device to perform FEC functions. + +This configuration needs to be executed at least once after reboot or PCI FLR and can +be achieved by using the function ``fpga_lte_fec_configure()``, which sets up the +parameters defined in ``fpga_lte_fec_conf`` structure: + +.. code-block:: c + + struct fpga_lte_fec_conf { + bool pf_mode_en; + uint8_t vf_ul_queues_number[FPGA_LTE_FEC_NUM_VFS]; + uint8_t vf_dl_queues_number[FPGA_LTE_FEC_NUM_VFS]; + uint8_t ul_bandwidth; + uint8_t dl_bandwidth; + uint8_t ul_load_balance; + uint8_t dl_load_balance; + uint16_t flr_time_out; + }; + +- ``pf_mode_en``: identifies whether only PF is to be used, or the VFs. PF and + VFs are mutually exclusive and cannot run simultaneously. + Set to 1 for PF mode enabled. + If PF mode is enabled all queues available in the device are assigned + exclusively to PF and 0 queues given to VFs. + +- ``vf_*l_queues_number``: defines the hardware queue mapping for every VF. + +- ``*l_bandwidth``: in case of congestion on PCIe interface. The device + allocates different bandwidth to UL and DL. The weight is configured by this + setting. The unit of weight is 3 code blocks. For example, if the code block + cbps (code block per second) ratio between UL and DL is 12:1, then the + configuration value should be set to 36:3. The schedule algorithm is based + on code block regardless the length of each block. + +- ``*l_load_balance``: hardware queues are load-balanced in a round-robin + fashion. Queues get filled first-in first-out until they reach a pre-defined + watermark level, if exceeded, they won't get assigned new code blocks.. + This watermark is defined by this setting. + + If all hardware queues exceeds the watermark, no code blocks will be + streamed in from UL/DL code block FIFO. + +- ``flr_time_out``: specifies how many 16.384us to be FLR time out. The + time_out = flr_time_out x 16.384us. For instance, if you want to set 10ms for + the FLR time out then set this setting to 0x262=610. + + +An example configuration code calling the function ``fpga_lte_fec_configure()`` is shown +below: + +.. code-block:: c + + struct fpga_lte_fec_conf conf; + unsigned int i; + + memset(&conf, 0, sizeof(struct fpga_lte_fec_conf)); + conf.pf_mode_en = 1; + + for (i = 0; i < FPGA_LTE_FEC_NUM_VFS; ++i) { + conf.vf_ul_queues_number[i] = 4; + conf.vf_dl_queues_number[i] = 4; + } + conf.ul_bandwidth = 12; + conf.dl_bandwidth = 5; + conf.dl_load_balance = 64; + conf.ul_load_balance = 64; + + /* setup FPGA PF */ + ret = fpga_lte_fec_configure(info->dev_name, &conf); + TEST_ASSERT_SUCCESS(ret, + "Failed to configure 4G FPGA PF for bbdev %s", + info->dev_name); + + +Test Application +---------------- + +BBDEV provides a test application, ``test-bbdev.py`` and range of test data for testing +the functionality of FPGA LTE FEC turbo encode and turbo decode, depending on the device's +capabilities. The test application is located under app->test-bbdev folder and has the +following options: + +.. code-block:: console + + "-p", "--testapp-path": specifies path to the bbdev test app. + "-e", "--eal-params" : EAL arguments which are passed to the test app. + "-t", "--timeout" : Timeout in seconds (default=300). + "-c", "--test-cases" : Defines test cases to run. Run all if not specified. + "-v", "--test-vector" : Test vector path (default=dpdk_path+/app/test-bbdev/test_vectors/bbdev_null.data). + "-n", "--num-ops" : Number of operations to process on device (default=32). + "-b", "--burst-size" : Operations enqueue/dequeue burst size (default=32). + "-l", "--num-lcores" : Number of lcores to run (default=16). + "-i", "--init-device" : Initialise PF device with default values. + + +To execute the test application tool using simple turbo decode or turbo encode data, +type one of the following: + +.. code-block:: console + + ./test-bbdev.py -c validation -n 64 -b 8 -v ./turbo_dec_default.data + ./test-bbdev.py -c validation -n 64 -b 8 -v ./turbo_enc_default.data + + +The test application ``test-bbdev.py``, supports the ability to configure the PF device with +a default set of values, if the "-i" or "- -init-device" option is included. The default values +are defined in test_bbdev_perf.c as: + +- VF_UL_QUEUE_VALUE 4 +- VF_DL_QUEUE_VALUE 4 +- UL_BANDWIDTH 3 +- DL_BANDWIDTH 3 +- UL_LOAD_BALANCE 128 +- DL_LOAD_BALANCE 128 +- FLR_TIMEOUT 610 + + +Test Vectors +~~~~~~~~~~~~ + +In addition to the simple turbo decoder and turbo encoder tests, bbdev also provides +a range of additional tests under the test_vectors folder, which may be useful. The results +of these tests will depend on the FPGA LTE FEC capabilities: + +* turbo decoder tests: + - ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_high_snr.data`` + - ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_low_snr.data`` + - ``turbo_dec_c1_k6144_r0_e34560_negllr.data`` + - ``turbo_dec_c1_k6144_r0_e34560_sbd_negllr.data`` + - ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr_crc24b.data`` + - ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr.data`` + + +* turbo encoder tests: + - ``turbo_enc_c1_k40_r0_e1190_rm.data`` + - ``turbo_enc_c1_k40_r0_e1194_rm.data`` + - ``turbo_enc_c1_k40_r0_e1196_rm.data`` + - ``turbo_enc_c1_k40_r0_e272_rm.data`` + - ``turbo_enc_c1_k6144_r0_e18444.data`` + - ``turbo_enc_c1_k6144_r0_e32256_crc24b_rm.data`` + - ``turbo_enc_c2_k5952_r0_e17868_crc24b.data`` + - ``turbo_enc_c3_k4800_r2_e14412_crc24b.data`` + - ``turbo_enc_c4_k4800_r2_e14412_crc24b.data`` + + diff --git a/doc/guides/bbdevs/index.rst b/doc/guides/bbdevs/index.rst index 93276ed..005b95e 100644 --- a/doc/guides/bbdevs/index.rst +++ b/doc/guides/bbdevs/index.rst @@ -10,3 +10,4 @@ Baseband Device Drivers null turbo_sw + fpga_lte_fec diff --git a/drivers/baseband/Makefile b/drivers/baseband/Makefile index 4ec83b0..ceffc7d 100644 --- a/drivers/baseband/Makefile +++ b/drivers/baseband/Makefile @@ -10,5 +10,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += null DEPDIRS-null = $(core-libs) DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += turbo_sw DEPDIRS-turbo_sw = $(core-libs) +DIRS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += fpga_lte_fec +DEPDIRS-fpga_lte_fec = $(core-libs) include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/drivers/baseband/fpga_lte_fec/Makefile b/drivers/baseband/fpga_lte_fec/Makefile new file mode 100644 index 0000000..a38a396 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/Makefile @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_pmd_fpga_lte_fec.a + +# build flags +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring +LDLIBS += -lrte_bbdev +LDLIBS += -lrte_pci -lrte_bus_pci + +# versioning export map +EXPORT_MAP := rte_pmd_bbdev_fpga_lte_fec_version.map + +# library version +LIBABIVER := 1 + +# library source files +SRCS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += fpga_lte_fec.c + +# export include files +SYMLINK-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC)-include += fpga_lte_fec.h + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c new file mode 100644 index 0000000..19e7689 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c @@ -0,0 +1,2674 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation + */ + +#include <unistd.h> + +#include <rte_common.h> +#include <rte_log.h> +#include <rte_dev.h> +#include <rte_malloc.h> +#include <rte_mempool.h> +#include <rte_errno.h> +#include <rte_pci.h> +#include <rte_bus_pci.h> +#include <rte_byteorder.h> +#ifdef RTE_BBDEV_OFFLOAD_COST +#include <rte_cycles.h> +#endif + +#include <rte_bbdev.h> +#include <rte_bbdev_pmd.h> + +#include "fpga_lte_fec.h" + +/* Turbo SW PMD logging ID */ +static int fpga_lte_fec_logtype; + +/* Helper macro for logging */ +#define rte_bbdev_log(level, fmt, ...) \ + rte_log(RTE_LOG_ ## level, fpga_lte_fec_logtype, fmt "\n", \ + ##__VA_ARGS__) + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +#define rte_bbdev_log_debug(fmt, ...) \ + rte_bbdev_log(DEBUG, "fpga_lte_fec: " fmt, \ + ##__VA_ARGS__) +#else +#define rte_bbdev_log_debug(fmt, ...) +#endif + +/* FPGA LTE FEC driver names */ +#define FPGA_LTE_FEC_PF_DRIVER_NAME intel_fpga_lte_fec_pf +#define FPGA_LTE_FEC_VF_DRIVER_NAME intel_fpga_lte_fec_vf + +/* FPGA LTE FEC PCI vendor & device IDs */ +#define FPGA_LTE_FEC_VENDOR_ID (0x1172) +#define FPGA_LTE_FEC_PF_DEVICE_ID (0x5052) +#define FPGA_LTE_FEC_VF_DEVICE_ID (0x5050) + +/* Align DMA descriptors to 256 bytes - cache-aligned */ +#define FPGA_RING_DESC_ENTRY_LENGTH (8) +/* Ring size is in 256 bits (32 bytes) units */ +#define FPGA_RING_DESC_LEN_UNIT_BYTES (32) +/* Maximum size of queue */ +#define FPGA_RING_MAX_SIZE (1024) +#define FPGA_FLR_TIMEOUT_UNIT (16.384) + +#define FPGA_NUM_UL_QUEUES (32) +#define FPGA_NUM_DL_QUEUES (32) +#define FPGA_TOTAL_NUM_QUEUES (FPGA_NUM_UL_QUEUES + FPGA_NUM_DL_QUEUES) +#define FPGA_NUM_INTR_VEC (FPGA_TOTAL_NUM_QUEUES - RTE_INTR_VEC_RXTX_OFFSET) + +#define FPGA_INVALID_HW_QUEUE_ID (0xFFFFFFFF) + +#define FPGA_QUEUE_FLUSH_TIMEOUT_US (1000) +#define FPGA_TIMEOUT_CHECK_INTERVAL (5) + +/* FPGA LTE FEC Register mapping on BAR0 */ +enum { + FPGA_LTE_FEC_VERSION_ID = 0x00000000, /* len: 4B */ + FPGA_LTE_FEC_CONFIGURATION = 0x00000004, /* len: 2B */ + FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE = 0x00000008, /* len: 1B */ + FPGA_LTE_FEC_LOAD_BALANCE_FACTOR = 0x0000000a, /* len: 2B */ + FPGA_LTE_FEC_RING_DESC_LEN = 0x0000000c, /* len: 2B */ + FPGA_LTE_FEC_FLR_TIME_OUT = 0x0000000e, /* len: 2B */ + FPGA_LTE_FEC_VFQ_FLUSH_STATUS_LW = 0x00000018, /* len: 4B */ + FPGA_LTE_FEC_VFQ_FLUSH_STATUS_HI = 0x0000001c, /* len: 4B */ + FPGA_LTE_FEC_VF0_DEBUG = 0x00000020, /* len: 4B */ + FPGA_LTE_FEC_VF1_DEBUG = 0x00000024, /* len: 4B */ + FPGA_LTE_FEC_VF2_DEBUG = 0x00000028, /* len: 4B */ + FPGA_LTE_FEC_VF3_DEBUG = 0x0000002c, /* len: 4B */ + FPGA_LTE_FEC_VF4_DEBUG = 0x00000030, /* len: 4B */ + FPGA_LTE_FEC_VF5_DEBUG = 0x00000034, /* len: 4B */ + FPGA_LTE_FEC_VF6_DEBUG = 0x00000038, /* len: 4B */ + FPGA_LTE_FEC_VF7_DEBUG = 0x0000003c, /* len: 4B */ + FPGA_LTE_FEC_QUEUE_MAP = 0x00000040, /* len: 256B */ + FPGA_LTE_FEC_RING_CTRL_REGS = 0x00000200 /* len: 2048B */ +}; + +/* FPGA LTE FEC Ring Control Registers */ +enum { + FPGA_LTE_FEC_RING_HEAD_ADDR = 0x00000008, + FPGA_LTE_FEC_RING_SIZE = 0x00000010, + FPGA_LTE_FEC_RING_MISC = 0x00000014, + FPGA_LTE_FEC_RING_ENABLE = 0x00000015, + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN = 0x00000016, + FPGA_LTE_FEC_RING_SHADOW_TAIL = 0x00000018, + FPGA_LTE_FEC_RING_HEAD_POINT = 0x0000001C +}; + +/* FPGA LTE FEC DESCRIPTOR ERROR */ +enum { + DESC_ERR_NO_ERR = 0x0, + DESC_ERR_K_OUT_OF_RANGE = 0x1, + DESC_ERR_K_NOT_NORMAL = 0x2, + DESC_ERR_KPAI_NOT_NORMAL = 0x3, + DESC_ERR_DESC_OFFSET_ERR = 0x4, + DESC_ERR_DESC_READ_FAIL = 0x8, + DESC_ERR_DESC_READ_TIMEOUT = 0x9, + DESC_ERR_DESC_READ_TLP_POISONED = 0xA, + DESC_ERR_CB_READ_FAIL = 0xC, + DESC_ERR_CB_READ_TIMEOUT = 0xD, + DESC_ERR_CB_READ_TLP_POISONED = 0xE +}; + +/* FPGA LTE FEC DMA Encoding Request Descriptor */ +struct __attribute__((__packed__)) fpga_dma_enc_desc { + uint32_t done:1, + rsrvd0:11, + error:4, + rsrvd1:16; + uint32_t ncb:16, + rsrvd2:14, + rv:2; + uint32_t bypass_rm:1, + irq_en:1, + crc_en:1, + rsrvd3:13, + offset:10, + rsrvd4:6; + uint16_t e; + uint16_t k; + uint32_t out_addr_lw; + uint32_t out_addr_hi; + uint32_t in_addr_lw; + uint32_t in_addr_hi; + + union { + struct { + /* Virtual addresses used to retrieve SW context info */ + void *op_addr; + /* Stores information about total number of Code Blocks + * in currently processed Transport Block + */ + uint64_t cbs_in_op; + }; + + uint8_t sw_ctxt[FPGA_RING_DESC_LEN_UNIT_BYTES * + (FPGA_RING_DESC_ENTRY_LENGTH - 1)]; + }; +}; + +/* FPGA LTE FEC DMA Decoding Request Descriptor */ +struct __attribute__((__packed__)) fpga_dma_dec_desc { + uint32_t done:1, + iter:5, + rsrvd0:2, + crc_pass:1, + rsrvd1:3, + error:4, + crc_type:1, + rsrvd2:7, + max_iter:5, + rsrvd3:3; + uint32_t rsrvd4; + uint32_t bypass_rm:1, + irq_en:1, + drop_crc:1, + rsrvd5:13, + offset:10, + rsrvd6:6; + uint16_t k; + uint16_t in_len; + uint32_t out_addr_lw; + uint32_t out_addr_hi; + uint32_t in_addr_lw; + uint32_t in_addr_hi; + + union { + struct { + /* Virtual addresses used to retrieve SW context info */ + void *op_addr; + /* Stores information about total number of Code Blocks + * in currently processed Transport Block + */ + uint8_t cbs_in_op; + }; + + uint32_t sw_ctxt[8 * (FPGA_RING_DESC_ENTRY_LENGTH - 1)]; + }; +}; + +/* FPGA LTE DMA Descriptor */ +union fpga_dma_desc { + struct fpga_dma_enc_desc enc_req; + struct fpga_dma_dec_desc dec_req; +}; + +/* FPGA LTE FEC Ring Control Register */ +struct __attribute__((__packed__)) fpga_ring_ctrl_reg { + uint64_t ring_base_addr; + uint64_t ring_head_addr; + uint16_t ring_size:11; + uint16_t rsrvd0; + union { /* Miscellaneous register */ + uint8_t misc; + uint8_t max_ul_dec:5, + max_ul_dec_en:1, + rsrvd1:2; + }; + uint8_t enable; + uint8_t flush_queue_en; + uint8_t rsrvd2; + uint16_t shadow_tail; + uint16_t rsrvd3; + uint16_t head_point; + uint16_t rsrvd4; + +}; + +/* Private data structure for each FPGA FEC device */ +struct fpga_lte_fec_device { + /** Base address of MMIO registers (BAR0) */ + void *mmio_base; + /** Base address of memory for sw rings */ + void *sw_rings; + /** Physical address of sw_rings */ + rte_iova_t sw_rings_phys; + /** Number of bytes available for each queue in device. */ + uint32_t sw_ring_size; + /** Max number of entries available for each queue in device */ + uint32_t sw_ring_max_depth; + /** Base address of response tail pointer buffer */ + uint32_t *tail_ptrs; + /** Physical address of tail pointers */ + rte_iova_t tail_ptr_phys; + /** Queues flush completion flag */ + uint64_t *flush_queue_status; + /* Bitmap capturing which Queues are bound to the PF/VF */ + uint64_t q_bound_bit_map; + /* Bitmap capturing which Queues have already been assigned */ + uint64_t q_assigned_bit_map; + /** True if this is a PF FPGA FEC device */ + bool pf_device; +}; + +/* Structure associated with each queue. */ +struct __rte_cache_aligned fpga_queue { + struct fpga_ring_ctrl_reg ring_ctrl_reg; /* Ring Control Register */ + union fpga_dma_desc *ring_addr; /* Virtual address of software ring */ + uint64_t *ring_head_addr; /* Virtual address of completion_head */ + uint64_t shadow_completion_head; /* Shadow completion head value */ + uint16_t head_free_desc; /* Ring head */ + uint16_t tail; /* Ring tail */ + /* Mask used to wrap enqueued descriptors on the sw ring */ + uint32_t sw_ring_wrap_mask; + uint32_t irq_enable; /* Enable ops dequeue interrupts if set to 1 */ + uint8_t q_idx; /* Queue index */ + struct fpga_lte_fec_device *d; + /* MMIO register of shadow_tail used to enqueue descriptors */ + void *shadow_tail_addr; +}; + +/* Write to 16 bit MMIO register address */ +static inline void +mmio_write_16(void *addr, uint16_t value) +{ + *((volatile uint16_t *)(addr)) = rte_cpu_to_le_16(value); +} + +/* Write to 32 bit MMIO register address */ +static inline void +mmio_write_32(void *addr, uint32_t value) +{ + *((volatile uint32_t *)(addr)) = rte_cpu_to_le_32(value); +} + +/* Write to 64 bit MMIO register address */ +static inline void +mmio_write_64(void *addr, uint64_t value) +{ + *((volatile uint64_t *)(addr)) = rte_cpu_to_le_64(value); +} + +/* Write a 8 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_8(void *mmio_base, uint32_t offset, uint8_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + *((volatile uint8_t *)(reg_addr)) = payload; +} + +/* Write a 16 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_16(void *mmio_base, uint32_t offset, uint16_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_16(reg_addr, payload); +} + +/* Write a 32 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_32(void *mmio_base, uint32_t offset, uint32_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_32(reg_addr, payload); +} + +/* Write a 64 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_64(void *mmio_base, uint32_t offset, uint64_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_64(reg_addr, payload); +} + +/* Write a ring control register of a FPGA LTE FEC device */ +static inline void +fpga_ring_reg_write(void *mmio_base, uint32_t offset, + struct fpga_ring_ctrl_reg payload) +{ + fpga_reg_write_64(mmio_base, offset, payload.ring_base_addr); + fpga_reg_write_64(mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_ADDR, + payload.ring_head_addr); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_SIZE, + payload.ring_size); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_POINT, + payload.head_point); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN, + payload.flush_queue_en); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_SHADOW_TAIL, + payload.shadow_tail); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_MISC, + payload.misc); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + payload.enable); +} + +/* Read a register of FPGA LTE FEC device */ +static uint32_t +fpga_reg_read_32(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint32_t ret = *((volatile uint32_t *)(reg_addr)); + return rte_le_to_cpu_32(ret); +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Read a register of FPGA LTE FEC device */ +static uint8_t +fpga_reg_read_8(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + return *((volatile uint8_t *)(reg_addr)); +} + +/* Read a register of FPGA LTE FEC device */ +static uint16_t +fpga_reg_read_16(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint16_t ret = *((volatile uint16_t *)(reg_addr)); + return rte_le_to_cpu_16(ret); +} + +/* Read a register of FPGA LTE FEC device */ +static uint64_t +fpga_reg_read_64(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint64_t ret = *((volatile uint64_t *)(reg_addr)); + return rte_le_to_cpu_64(ret); +} + +/* Read Ring Control Register of FPGA LTE FEC device */ +static inline void +print_ring_reg_debug_info(void *mmio_base, uint32_t offset) +{ + rte_bbdev_log_debug( + "FPGA MMIO base address @ %p | Ring Control Register @ offset = 0x%08" + PRIx32, mmio_base, offset); + rte_bbdev_log_debug( + "RING_BASE_ADDR = 0x%016"PRIx64, + fpga_reg_read_64(mmio_base, offset)); + rte_bbdev_log_debug( + "RING_HEAD_ADDR = 0x%016"PRIx64, + fpga_reg_read_64(mmio_base, offset + + FPGA_LTE_FEC_RING_HEAD_ADDR)); + rte_bbdev_log_debug( + "RING_SIZE = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_SIZE)); + rte_bbdev_log_debug( + "RING_MISC = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_MISC)); + rte_bbdev_log_debug( + "RING_ENABLE = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_ENABLE)); + rte_bbdev_log_debug( + "RING_FLUSH_QUEUE_EN = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN)); + rte_bbdev_log_debug( + "RING_SHADOW_TAIL = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_SHADOW_TAIL)); + rte_bbdev_log_debug( + "RING_HEAD_POINT = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_HEAD_POINT)); +} + +/* Read Static Register of FPGA LTE FEC device */ +static inline void +print_static_reg_debug_info(void *mmio_base) +{ + uint16_t config = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_CONFIGURATION); + uint8_t qmap_done = fpga_reg_read_8(mmio_base, + FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE); + uint16_t lb_factor = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_LOAD_BALANCE_FACTOR); + uint16_t ring_desc_len = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_RING_DESC_LEN); + uint16_t flr_time_out = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_FLR_TIME_OUT); + + rte_bbdev_log_debug("UL.DL Weights = %u.%u", + ((uint8_t)config), ((uint8_t)(config >> 8))); + rte_bbdev_log_debug("UL.DL Load Balance = %u.%u", + ((uint8_t)lb_factor), ((uint8_t)(lb_factor >> 8))); + rte_bbdev_log_debug("Queue-PF/VF Mapping Table = %s", + (qmap_done > 0) ? "READY" : "NOT-READY"); + rte_bbdev_log_debug("Ring Descriptor Size = %u bytes", + ring_desc_len*FPGA_RING_DESC_LEN_UNIT_BYTES); + rte_bbdev_log_debug("FLR Timeout = %f usec", + (float)flr_time_out*FPGA_FLR_TIMEOUT_UNIT); +} + +/* Print decode DMA Descriptor of FPGA LTE FEC device */ +static void +print_dma_dec_desc_debug_info(union fpga_dma_desc *desc) +{ + rte_bbdev_log_debug("DMA response desc %p\n" + "\t-- done(%"PRIu32") | iter(%"PRIu32") | crc_pass(%"PRIu32")" + " | error (%"PRIu32") | crc_type(%"PRIu32")\n" + "\t-- max_iter(%"PRIu32") | bypass_rm(%"PRIu32") | " + "irq_en (%"PRIu32") | drop_crc(%"PRIu32") | offset(%"PRIu32")\n" + "\t-- k(%"PRIu32") | in_len (%"PRIu16") | op_add(%p)\n" + "\t-- cbs_in_op(%"PRIu32") | in_add (0x%08"PRIx32"%08"PRIx32") | " + "out_add (0x%08"PRIx32"%08"PRIx32")", + desc, + (uint32_t)desc->dec_req.done, + (uint32_t)desc->dec_req.iter, + (uint32_t)desc->dec_req.crc_pass, + (uint32_t)desc->dec_req.error, + (uint32_t)desc->dec_req.crc_type, + (uint32_t)desc->dec_req.max_iter, + (uint32_t)desc->dec_req.bypass_rm, + (uint32_t)desc->dec_req.irq_en, + (uint32_t)desc->dec_req.drop_crc, + (uint32_t)desc->dec_req.offset, + (uint32_t)desc->dec_req.k, + (uint16_t)desc->dec_req.in_len, + desc->dec_req.op_addr, + (uint32_t)desc->dec_req.cbs_in_op, + (uint32_t)desc->dec_req.in_addr_hi, + (uint32_t)desc->dec_req.in_addr_lw, + (uint32_t)desc->dec_req.out_addr_hi, + (uint32_t)desc->dec_req.out_addr_lw); +} +#endif + +static int +fpga_setup_queues(struct rte_bbdev *dev, uint16_t num_queues, int socket_id) +{ + /* Number of queues bound to a PF/VF */ + uint32_t hw_q_num = 0; + uint32_t ring_size, payload, address, q_id, offset; + rte_iova_t phys_addr; + struct fpga_ring_ctrl_reg ring_reg; + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + + address = FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE; + if (!(fpga_reg_read_32(fpga_dev->mmio_base, address) & 0x1)) { + rte_bbdev_log(ERR, + "Queue-PF/VF mapping is not set! Was PF configured for device (%s) ?", + dev->data->name); + return -EPERM; + } + + /* Clear queue registers structure */ + memset(&ring_reg, 0, sizeof(struct fpga_ring_ctrl_reg)); + + /* Scan queue map. + * If a queue is valid and mapped to a calling PF/VF the read value is + * replaced with a queue ID and if it's not then + * FPGA_INVALID_HW_QUEUE_ID is returned. + */ + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + uint32_t hw_q_id = fpga_reg_read_32(fpga_dev->mmio_base, + FPGA_LTE_FEC_QUEUE_MAP + (q_id << 2)); + + rte_bbdev_log_debug("%s: queue ID: %u, registry queue ID: %u", + dev->device->name, q_id, hw_q_id); + + if (hw_q_id != FPGA_INVALID_HW_QUEUE_ID) { + fpga_dev->q_bound_bit_map |= (1ULL << q_id); + /* Clear queue register of found queue */ + offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q_id); + fpga_ring_reg_write(fpga_dev->mmio_base, + offset, ring_reg); + ++hw_q_num; + } + } + if (hw_q_num == 0) { + rte_bbdev_log(ERR, + "No HW queues assigned to this device. Probably this is a VF configured for PF mode. Check device configuration!"); + return -ENODEV; + } + + if (num_queues > hw_q_num) { + rte_bbdev_log(ERR, + "Not enough queues for device %s! Requested: %u, available: %u", + dev->device->name, num_queues, hw_q_num); + return -EINVAL; + } + + ring_size = FPGA_RING_MAX_SIZE * sizeof(struct fpga_dma_dec_desc); + + /* Enforce 32 byte alignment */ + RTE_BUILD_BUG_ON((RTE_CACHE_LINE_SIZE % 32) != 0); + + /* Allocate memory for SW descriptor rings */ + fpga_dev->sw_rings = rte_zmalloc_socket(dev->device->driver->name, + num_queues * ring_size, RTE_CACHE_LINE_SIZE, + socket_id); + if (fpga_dev->sw_rings == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u sw_rings", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + + fpga_dev->sw_rings_phys = rte_malloc_virt2iova(fpga_dev->sw_rings); + fpga_dev->sw_ring_size = ring_size; + fpga_dev->sw_ring_max_depth = FPGA_RING_MAX_SIZE; + + /* Allocate memory for ring flush status */ + fpga_dev->flush_queue_status = rte_zmalloc_socket(NULL, + sizeof(uint64_t), RTE_CACHE_LINE_SIZE, socket_id); + if (fpga_dev->flush_queue_status == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u flush_queue_status", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + + /* Set the flush status address registers */ + phys_addr = rte_malloc_virt2iova(fpga_dev->flush_queue_status); + + address = FPGA_LTE_FEC_VFQ_FLUSH_STATUS_LW; + payload = (uint32_t)(phys_addr); + fpga_reg_write_32(fpga_dev->mmio_base, address, payload); + + address = FPGA_LTE_FEC_VFQ_FLUSH_STATUS_HI; + payload = (uint32_t)(phys_addr >> 32); + fpga_reg_write_32(fpga_dev->mmio_base, address, payload); + + return 0; +} + +static int +fpga_dev_close(struct rte_bbdev *dev) +{ + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + + rte_free(fpga_dev->sw_rings); + rte_free(fpga_dev->flush_queue_status); + + return 0; +} + +static void +fpga_dev_info_get(struct rte_bbdev *dev, + struct rte_bbdev_driver_info *dev_info) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + uint32_t q_id = 0; + + /* TODO RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN and numbers of buffers are set + * to temporary values as they are required by test application while + * validation phase. + */ + static const struct rte_bbdev_op_cap bbdev_capabilities[] = { + { + .type = RTE_BBDEV_OP_TURBO_DEC, + .cap.turbo_dec = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_TYPE_24B | + RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE | + RTE_BBDEV_TURBO_DEC_INTERRUPTS | + RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN | + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP, + .max_llr_modulus = INT8_MAX, + .num_buffers_src = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_hard_out = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_soft_out = 0 + } + }, + { + .type = RTE_BBDEV_OP_TURBO_ENC, + .cap.turbo_enc = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_24B_ATTACH | + RTE_BBDEV_TURBO_RATE_MATCH | + RTE_BBDEV_TURBO_ENC_INTERRUPTS, + .num_buffers_src = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_dst = + RTE_BBDEV_MAX_CODE_BLOCKS + } + }, + RTE_BBDEV_END_OF_CAPABILITIES_LIST() + }; + + static struct rte_bbdev_queue_conf default_queue_conf; + default_queue_conf.socket = dev->data->socket_id; + default_queue_conf.queue_size = FPGA_RING_MAX_SIZE; + + + dev_info->driver_name = dev->device->driver->name; + dev_info->queue_size_lim = FPGA_RING_MAX_SIZE; + dev_info->hardware_accelerated = true; + dev_info->min_alignment = 64; + dev_info->default_queue_conf = default_queue_conf; + dev_info->capabilities = bbdev_capabilities; + dev_info->cpu_flag_reqs = NULL; + + /* Calculates number of queues assigned to device */ + dev_info->max_num_queues = 0; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + uint32_t hw_q_id = fpga_reg_read_32(d->mmio_base, + FPGA_LTE_FEC_QUEUE_MAP + (q_id << 2)); + if (hw_q_id != FPGA_INVALID_HW_QUEUE_ID) + dev_info->max_num_queues++; + } +} + +/** + * Find index of queue bound to current PF/VF which is unassigned. Return -1 + * when there is no available queue + */ +static int +fpga_find_free_queue_idx(struct rte_bbdev *dev, + const struct rte_bbdev_queue_conf *conf) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + uint64_t q_idx; + uint8_t i = 0; + uint8_t range = FPGA_TOTAL_NUM_QUEUES >> 1; + + if (conf->op_type == RTE_BBDEV_OP_TURBO_ENC) { + i = FPGA_NUM_DL_QUEUES; + range = FPGA_TOTAL_NUM_QUEUES; + } + + for (; i < range; ++i) { + q_idx = 1ULL << i; + /* Check if index of queue is bound to current PF/VF */ + if (d->q_bound_bit_map & q_idx) + /* Check if found queue was not already assigned */ + if (!(d->q_assigned_bit_map & q_idx)) { + d->q_assigned_bit_map |= q_idx; + return i; + } + } + + rte_bbdev_log(INFO, "Failed to find free queue on %s", dev->data->name); + + return -1; +} + +static int +fpga_queue_setup(struct rte_bbdev *dev, uint16_t queue_id, + const struct rte_bbdev_queue_conf *conf) +{ + uint32_t address, ring_offset; + struct fpga_lte_fec_device *d = dev->data->dev_private; + struct fpga_queue *q; + int8_t q_idx; + + /* Check if there is a free queue to assign */ + q_idx = fpga_find_free_queue_idx(dev, conf); + if (q_idx == -1) + return -1; + + /* Allocate the queue data structure. */ + q = rte_zmalloc_socket(dev->device->driver->name, sizeof(*q), + RTE_CACHE_LINE_SIZE, conf->socket); + if (q == NULL) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_bbdev_log(ERR, "Failed to allocate queue memory"); + return -ENOMEM; + } + + q->d = d; + q->q_idx = q_idx; + + /* Set ring_base_addr */ + q->ring_addr = RTE_PTR_ADD(d->sw_rings, (d->sw_ring_size * queue_id)); + q->ring_ctrl_reg.ring_base_addr = d->sw_rings_phys + + (d->sw_ring_size * queue_id); + + /* Allocate memory for Completion Head variable*/ + q->ring_head_addr = rte_zmalloc_socket(dev->device->driver->name, + sizeof(uint64_t), RTE_CACHE_LINE_SIZE, conf->socket); + if (q->ring_head_addr == NULL) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_free(q); + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u completion_head", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + /* Set ring_head_addr */ + q->ring_ctrl_reg.ring_head_addr = + rte_malloc_virt2iova(q->ring_head_addr); + + /* Clear shadow_completion_head */ + q->shadow_completion_head = 0; + + /* Set ring_size */ + if (conf->queue_size > FPGA_RING_MAX_SIZE) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_free(q->ring_head_addr); + rte_free(q); + rte_bbdev_log(ERR, + "Size of queue is too big %d (MAX: %d ) for %s:%u", + conf->queue_size, FPGA_RING_MAX_SIZE, + dev->device->driver->name, dev->data->dev_id); + return -EINVAL; + } + q->ring_ctrl_reg.ring_size = conf->queue_size; + + /* Set Miscellaneous FPGA register*/ + /* Max iteration number for TTI mitigation - todo */ + q->ring_ctrl_reg.max_ul_dec = 0; + /* Enable max iteration number for TTI - todo */ + q->ring_ctrl_reg.max_ul_dec_en = 0; + + /* Enable the ring */ + q->ring_ctrl_reg.enable = 1; + + /* Set FPGA head_point and tail registers */ + q->ring_ctrl_reg.head_point = q->tail = 0; + + /* Set FPGA shadow_tail register */ + q->ring_ctrl_reg.shadow_tail = q->tail; + + /* Calculates the ring offset for found queue */ + ring_offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q_idx); + + /* Set FPGA Ring Control Registers */ + fpga_ring_reg_write(d->mmio_base, ring_offset, q->ring_ctrl_reg); + + /* Store MMIO register of shadow_tail */ + address = ring_offset + FPGA_LTE_FEC_RING_SHADOW_TAIL; + q->shadow_tail_addr = RTE_PTR_ADD(d->mmio_base, address); + + q->head_free_desc = q->tail; + + /* Set wrap mask */ + q->sw_ring_wrap_mask = conf->queue_size - 1; + + rte_bbdev_log_debug("Setup dev%u q%u: queue_idx=%u", + dev->data->dev_id, queue_id, q->q_idx); + + dev->data->queues[queue_id].queue_private = q; + + rte_bbdev_log_debug("BBDEV queue[%d] set up for FPGA queue[%d]", + queue_id, q_idx); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Read FPGA Ring Control Registers after configuration*/ + print_ring_reg_debug_info(d->mmio_base, ring_offset); +#endif + return 0; +} + +static int +fpga_queue_release(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + struct fpga_ring_ctrl_reg ring_reg; + uint32_t offset; + + rte_bbdev_log_debug("FPGA Queue[%d] released", queue_id); + + if (q != NULL) { + memset(&ring_reg, 0, sizeof(struct fpga_ring_ctrl_reg)); + offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + /* Disable queue */ + fpga_reg_write_8(d->mmio_base, + offset + FPGA_LTE_FEC_RING_ENABLE, 0x00); + /* Clear queue registers */ + fpga_ring_reg_write(d->mmio_base, offset, ring_reg); + + /* Mark the Queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q->q_idx)); + rte_free(q->ring_head_addr); + rte_free(q); + dev->data->queues[queue_id].queue_private = NULL; + } + + return 0; +} + +/* Function starts a device queue. */ +static int +fpga_queue_start(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (d == NULL) { + rte_bbdev_log(ERR, "Invalid device pointer"); + return -1; + } +#endif + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + uint32_t offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + uint8_t enable = 0x01; + uint16_t zero = 0x0000; + + /* Clear queue head and tail variables */ + q->tail = q->head_free_desc = 0; + + /* Clear FPGA head_point and tail registers */ + fpga_reg_write_16(d->mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_POINT, + zero); + fpga_reg_write_16(d->mmio_base, offset + FPGA_LTE_FEC_RING_SHADOW_TAIL, + zero); + + /* Enable queue */ + fpga_reg_write_8(d->mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + enable); + + rte_bbdev_log_debug("FPGA Queue[%d] started", queue_id); + return 0; +} + +/* Function stops a device queue. */ +static int +fpga_queue_stop(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (d == NULL) { + rte_bbdev_log(ERR, "Invalid device pointer"); + return -1; + } +#endif + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + uint32_t offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + uint8_t payload = 0x01; + uint8_t counter = 0; + uint8_t timeout = FPGA_QUEUE_FLUSH_TIMEOUT_US / + FPGA_TIMEOUT_CHECK_INTERVAL; + + /* Set flush_queue_en bit to trigger queue flushing */ + fpga_reg_write_8(d->mmio_base, + offset + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN, payload); + + /** Check if queue flush is completed. + * FPGA will update the completion flag after queue flushing is + * completed. If completion flag is not updated within 1ms it is + * considered as a failure. + */ + while (!(*((uint8_t *)d->flush_queue_status + q->q_idx) & payload)) { + if (counter > timeout) { + rte_bbdev_log(ERR, "FPGA Queue Flush failed for queue %d", + queue_id); + return -1; + } + usleep(FPGA_TIMEOUT_CHECK_INTERVAL); + counter++; + } + + /* Disable queue */ + payload = 0x00; + fpga_reg_write_8(d->mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + payload); + + rte_bbdev_log_debug("FPGA Queue[%d] stopped", queue_id); + return 0; +} + +static inline uint16_t +get_queue_id(struct rte_bbdev_data *data, uint8_t q_idx) +{ + uint16_t queue_id; + + for (queue_id = 0; queue_id < data->num_queues; ++queue_id) { + struct fpga_queue *q = data->queues[queue_id].queue_private; + if (q != NULL && q->q_idx == q_idx) + return queue_id; + } + + return -1; +} + +/* Interrupt handler triggered by FPGA dev for handling specific interrupt */ +static void +fpga_dev_interrupt_handler(void *cb_arg) +{ + struct rte_bbdev *dev = cb_arg; + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + struct fpga_queue *q; + uint64_t ring_head; + uint64_t q_idx; + uint16_t queue_id; + uint8_t i; + + /* Scan queue assigned to this device */ + for (i = 0; i < FPGA_TOTAL_NUM_QUEUES; ++i) { + q_idx = 1ULL << i; + if (fpga_dev->q_bound_bit_map & q_idx) { + queue_id = get_queue_id(dev->data, i); + if (queue_id == (uint16_t) -1) + continue; + + /* Check if completion head was changed */ + q = dev->data->queues[queue_id].queue_private; + ring_head = *q->ring_head_addr; + if (q->shadow_completion_head != ring_head && + q->irq_enable == 1) { + q->shadow_completion_head = ring_head; + rte_bbdev_pmd_callback_process( + dev, + RTE_BBDEV_EVENT_DEQUEUE, + &queue_id); + } + } + } +} + +static int +fpga_queue_intr_enable(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + + if (!rte_intr_cap_multiple(dev->intr_handle)) + return -ENOTSUP; + + q->irq_enable = 1; + + return 0; +} + +static int +fpga_queue_intr_disable(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + q->irq_enable = 0; + + return 0; +} + +static int +fpga_intr_enable(struct rte_bbdev *dev) +{ + int ret; + uint8_t i; + + if (!rte_intr_cap_multiple(dev->intr_handle)) { + rte_bbdev_log(ERR, "Multiple intr vector is not supported by FPGA (%s)", + dev->data->name); + return -ENOTSUP; + } + + /* Create event file descriptors for each of 64 queue. Event fds will be + * mapped to FPGA IRQs in rte_intr_enable(). This is a 1:1 mapping where + * the IRQ number is a direct translation to the queue number. + * + * 63 (FPGA_NUM_INTR_VEC) event fds are created as rte_intr_enable() + * mapped the first IRQ to already created interrupt event file + * descriptor (intr_handle->fd). + */ + if (rte_intr_efd_enable(dev->intr_handle, FPGA_NUM_INTR_VEC)) { + rte_bbdev_log(ERR, "Failed to create fds for %u queues", + dev->data->num_queues); + return -1; + } + + /* TODO Each event file descriptor is overwritten by interrupt event + * file descriptor. That descriptor is added to epoll observed list. + * It ensures that callback function assigned to that descriptor will + * invoked when any FPGA queue issues interrupt. + */ + for (i = 0; i < FPGA_NUM_INTR_VEC; ++i) + dev->intr_handle->efds[i] = dev->intr_handle->fd; + + if (!dev->intr_handle->intr_vec) { + dev->intr_handle->intr_vec = rte_zmalloc("intr_vec", + dev->data->num_queues * sizeof(int), 0); + if (!dev->intr_handle->intr_vec) { + rte_bbdev_log(ERR, "Failed to allocate %u vectors", + dev->data->num_queues); + return -ENOMEM; + } + } + + ret = rte_intr_enable(dev->intr_handle); + if (ret < 0) { + rte_bbdev_log(ERR, + "Couldn't enable interrupts for device: %s", + dev->data->name); + return ret; + } + + ret = rte_intr_callback_register(dev->intr_handle, + fpga_dev_interrupt_handler, dev); + if (ret < 0) { + rte_bbdev_log(ERR, + "Couldn't register interrupt callback for device: %s", + dev->data->name); + return ret; + } + + return 0; +} + +static const struct rte_bbdev_ops fpga_ops = { + .setup_queues = fpga_setup_queues, + .intr_enable = fpga_intr_enable, + .close = fpga_dev_close, + .info_get = fpga_dev_info_get, + .queue_setup = fpga_queue_setup, + .queue_stop = fpga_queue_stop, + .queue_start = fpga_queue_start, + .queue_release = fpga_queue_release, + .queue_intr_enable = fpga_queue_intr_enable, + .queue_intr_disable = fpga_queue_intr_disable +}; + +static inline void +fpga_dma_enqueue(struct fpga_queue *q, uint16_t num_desc, + struct rte_bbdev_stats *queue_stats) +{ +#ifdef RTE_BBDEV_OFFLOAD_COST + uint64_t start_time = 0; + queue_stats->acc_offload_cycles = 0; +#else + RTE_SET_USED(queue_stats); +#endif + + /* Update tail and shadow_tail register */ + q->tail = (q->tail + num_desc) & q->sw_ring_wrap_mask; + + rte_wmb(); + +#ifdef RTE_BBDEV_OFFLOAD_COST + /* Start time measurement for enqueue function offload. */ + start_time = rte_rdtsc_precise(); +#endif + mmio_write_16(q->shadow_tail_addr, q->tail); + +#ifdef RTE_BBDEV_OFFLOAD_COST + rte_wmb(); + queue_stats->acc_offload_cycles += rte_rdtsc_precise() - start_time; +#endif +} + +/* Calculates number of CBs in processed encoder TB based on 'r' and input + * length. + */ +static inline uint8_t +get_num_cbs_in_op_enc(struct rte_bbdev_op_turbo_enc *turbo_enc) +{ + uint8_t c, c_neg, r, crc24_bits = 0; + uint16_t k, k_neg, k_pos; + uint8_t cbs_in_op = 0; + int32_t length; + + length = turbo_enc->input.length; + r = turbo_enc->tb_params.r; + c = turbo_enc->tb_params.c; + c_neg = turbo_enc->tb_params.c_neg; + k_neg = turbo_enc->tb_params.k_neg; + k_pos = turbo_enc->tb_params.k_pos; + crc24_bits = 24; + while (length > 0 && r < c) { + k = (r < c_neg) ? k_neg : k_pos; + length -= (k - crc24_bits) >> 3; + r++; + cbs_in_op++; + } + + return cbs_in_op; +} + +/* Calculates number of CBs in processed decoder TB based on 'r' and input + * length. + */ +static inline uint16_t +get_num_cbs_in_op_dec(struct rte_bbdev_op_turbo_dec *turbo_dec) +{ + uint8_t c, c_neg, r = 0; + uint16_t kw, k, k_neg, k_pos, cbs_in_op = 0; + int32_t length; + + length = turbo_dec->input.length; + r = turbo_dec->tb_params.r; + c = turbo_dec->tb_params.c; + c_neg = turbo_dec->tb_params.c_neg; + k_neg = turbo_dec->tb_params.k_neg; + k_pos = turbo_dec->tb_params.k_pos; + while (length > 0 && r < c) { + k = (r < c_neg) ? k_neg : k_pos; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + length -= kw; + r++; + cbs_in_op++; + } + + return cbs_in_op; +} + +/* Read flag value 0/1/ from bitmap */ +static inline bool +check_bit(uint32_t bitmap, uint32_t bitmask) +{ + return bitmap & bitmask; +} + +/* Print an error if a descriptor error has occurred. + * Return 0 on success, 1 on failure + */ +static inline int +check_desc_error(uint32_t error_code) { + switch (error_code) { + case DESC_ERR_NO_ERR: + return 0; + case DESC_ERR_K_OUT_OF_RANGE: + rte_bbdev_log(ERR, "Block_size_k is out of range (k<40 or k>6144)"); + break; + case DESC_ERR_K_NOT_NORMAL: + rte_bbdev_log(ERR, "Block_size_k is not a normal value within normal range"); + break; + case DESC_ERR_KPAI_NOT_NORMAL: + rte_bbdev_log(ERR, "Three_kpai is not a normal value for UL only"); + break; + case DESC_ERR_DESC_OFFSET_ERR: + rte_bbdev_log(ERR, "Queue offset does not meet the expectation in the FPGA"); + break; + case (DESC_ERR_K_OUT_OF_RANGE | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Block_size_k is out of range (k<40 or k>6144) and queue offset error"); + break; + case (DESC_ERR_K_NOT_NORMAL | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Block_size_k is not a normal value within normal range and queue offset error"); + break; + case (DESC_ERR_KPAI_NOT_NORMAL | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Three_kpai is not a normal value for UL only and queue offset error"); + break; + case DESC_ERR_DESC_READ_FAIL: + rte_bbdev_log(ERR, "Unsuccessful completion for descriptor read"); + break; + case DESC_ERR_DESC_READ_TIMEOUT: + rte_bbdev_log(ERR, "Descriptor read time-out"); + break; + case DESC_ERR_DESC_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Descriptor read TLP poisoned"); + break; + case DESC_ERR_CB_READ_FAIL: + rte_bbdev_log(ERR, "Unsuccessful completion for code block"); + break; + case DESC_ERR_CB_READ_TIMEOUT: + rte_bbdev_log(ERR, "Code block read time-out"); + break; + case DESC_ERR_CB_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Code block read TLP poisoned"); + break; + default: + rte_bbdev_log(ERR, "Descriptor error unknown error code %u", + error_code); + break; + } + return 1; +} + +/** + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param k + * K value (length of input in bits). + * @param e + * E value (length of output in bits). + * @param ncb + * Ncb value (size of the soft buffer). + * @param out_length + * Length of output buffer + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +fpga_dma_desc_te_fill(struct rte_bbdev_enc_op *op, + struct fpga_dma_enc_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t k, uint16_t e, uint16_t ncb, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) + +{ + /* reset */ + desc->done = 0; + desc->crc_en = check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_CRC_24B_ATTACH); + desc->bypass_rm = !check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_RATE_MATCH); + desc->k = k; + desc->e = e; + desc->ncb = ncb; + desc->rv = op->turbo_enc.rv_index; + desc->offset = desc_offset; + /* Set inbound data buffer address */ + desc->in_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset) >> 32); + desc->in_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset)); + + desc->out_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset) >> 32); + desc->out_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset)); + + /* Save software context needed for dequeue */ + desc->op_addr = op; + + /* Set total number of CBs in an op */ + desc->cbs_in_op = cbs_in_op; + + return 0; +} + +/** + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param in_length + * Length of an input. + * @param k + * K value (length of an output in bits). + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +fpga_dma_desc_td_fill(struct rte_bbdev_dec_op *op, + struct fpga_dma_dec_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t in_length, uint16_t k, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) +{ + /* reset */ + desc->done = 0; + /* Set inbound data buffer address */ + desc->in_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset) >> 32); + desc->in_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset)); + desc->in_len = in_length; + desc->k = k; + desc->crc_type = !check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_CRC_TYPE_24B); + if ((op->turbo_dec.code_block_mode == 0) + && !check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP)) + desc->drop_crc = 1; + desc->max_iter = op->turbo_dec.iter_max * 2; + desc->offset = desc_offset; + desc->out_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset) >> 32); + desc->out_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset)); + + /* Save software context needed for dequeue */ + desc->op_addr = op; + + /* Set total number of CBs in an op */ + desc->cbs_in_op = cbs_in_op; + + return 0; +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Validates turbo encoder parameters */ +static int +validate_enc_op(struct rte_bbdev_enc_op *op) +{ + struct rte_bbdev_op_turbo_enc *turbo_enc = &op->turbo_enc; + struct rte_bbdev_op_enc_cb_params *cb = NULL; + struct rte_bbdev_op_enc_tb_params *tb = NULL; + uint16_t kw, kw_neg, kw_pos; + + if (turbo_enc->input.length > + RTE_BBDEV_MAX_TB_SIZE >> 3) { + rte_bbdev_log(ERR, "TB size (%u) is too big, max: %d", + turbo_enc->input.length, RTE_BBDEV_MAX_TB_SIZE); + op->status = 1 << RTE_BBDEV_DATA_ERROR; + return -1; + } + + if (op->mempool == NULL) { + rte_bbdev_log(ERR, "Invalid mempool pointer"); + return -1; + } + if (turbo_enc->input.data == NULL) { + rte_bbdev_log(ERR, "Invalid input pointer"); + return -1; + } + if (turbo_enc->output.data == NULL) { + rte_bbdev_log(ERR, "Invalid output pointer"); + return -1; + } + if (turbo_enc->rv_index > 3) { + rte_bbdev_log(ERR, + "rv_index (%u) is out of range 0 <= value <= 3", + turbo_enc->rv_index); + return -1; + } + if (turbo_enc->code_block_mode != 0 && + turbo_enc->code_block_mode != 1) { + rte_bbdev_log(ERR, + "code_block_mode (%u) is out of range 0 <= value <= 1", + turbo_enc->code_block_mode); + return -1; + } + + if (turbo_enc->code_block_mode == 0) { + tb = &turbo_enc->tb_params; + if ((tb->k_neg < RTE_BBDEV_MIN_CB_SIZE + || tb->k_neg > RTE_BBDEV_MAX_CB_SIZE) + && tb->c_neg > 0) { + rte_bbdev_log(ERR, + "k_neg (%u) is out of range %u <= value <= %u", + tb->k_neg, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->k_pos < RTE_BBDEV_MIN_CB_SIZE + || tb->k_pos > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k_pos (%u) is out of range %u <= value <= %u", + tb->k_pos, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->c_neg > (RTE_BBDEV_MAX_CODE_BLOCKS - 1)) + rte_bbdev_log(ERR, + "c_neg (%u) is out of range 0 <= value <= %u", + tb->c_neg, + RTE_BBDEV_MAX_CODE_BLOCKS - 1); + if (tb->c < 1 || tb->c > RTE_BBDEV_MAX_CODE_BLOCKS) { + rte_bbdev_log(ERR, + "c (%u) is out of range 1 <= value <= %u", + tb->c, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + if (tb->cab > tb->c) { + rte_bbdev_log(ERR, + "cab (%u) is greater than c (%u)", + tb->cab, tb->c); + return -1; + } + if ((tb->ea < RTE_BBDEV_MIN_CB_SIZE || (tb->ea % 2)) + && tb->r < tb->cab) { + rte_bbdev_log(ERR, + "ea (%u) is less than %u or it is not even", + tb->ea, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + if ((tb->eb < RTE_BBDEV_MIN_CB_SIZE || (tb->eb % 2)) + && tb->c > tb->cab) { + rte_bbdev_log(ERR, + "eb (%u) is less than %u or it is not even", + tb->eb, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + + kw_neg = 3 * RTE_ALIGN_CEIL(tb->k_neg + 4, + RTE_BBDEV_C_SUBBLOCK); + if (tb->ncb_neg < tb->k_neg || tb->ncb_neg > kw_neg) { + rte_bbdev_log(ERR, + "ncb_neg (%u) is out of range (%u) k_neg <= value <= (%u) kw_neg", + tb->ncb_neg, tb->k_neg, kw_neg); + return -1; + } + + kw_pos = 3 * RTE_ALIGN_CEIL(tb->k_pos + 4, + RTE_BBDEV_C_SUBBLOCK); + if (tb->ncb_pos < tb->k_pos || tb->ncb_pos > kw_pos) { + rte_bbdev_log(ERR, + "ncb_pos (%u) is out of range (%u) k_pos <= value <= (%u) kw_pos", + tb->ncb_pos, tb->k_pos, kw_pos); + return -1; + } + if (tb->r > (tb->c - 1)) { + rte_bbdev_log(ERR, + "r (%u) is greater than c - 1 (%u)", + tb->r, tb->c - 1); + return -1; + } + } else { + cb = &turbo_enc->cb_params; + if (cb->k < RTE_BBDEV_MIN_CB_SIZE + || cb->k > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k (%u) is out of range %u <= value <= %u", + cb->k, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + + if (cb->e < RTE_BBDEV_MIN_CB_SIZE || (cb->e % 2)) { + rte_bbdev_log(ERR, + "e (%u) is less than %u or it is not even", + cb->e, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + + kw = RTE_ALIGN_CEIL(cb->k + 4, RTE_BBDEV_C_SUBBLOCK) * 3; + if (cb->ncb < cb->k || cb->ncb > kw) { + rte_bbdev_log(ERR, + "ncb (%u) is out of range (%u) k <= value <= (%u) kw", + cb->ncb, cb->k, kw); + return -1; + } + } + + return 0; +} +#endif + +static inline char * +mbuf_append(struct rte_mbuf *m_head, struct rte_mbuf *m, uint16_t len) +{ + if (unlikely(len > rte_pktmbuf_tailroom(m))) + return NULL; + + char *tail = (char *)m->buf_addr + m->data_off + m->data_len; + m->data_len = (uint16_t)(m->data_len + len); + m_head->pkt_len = (m_head->pkt_len + len); + return tail; +} + +static inline int +enqueue_enc_one_op_cb(struct fpga_queue *q, struct rte_bbdev_enc_op *op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input; + struct rte_mbuf *output; + int ret; + uint16_t k, e, ncb, ring_offset; + uint32_t total_left, in_length, out_length, in_offset, out_offset; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_enc_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo encoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_enc.input.data; + output = op->turbo_enc.output.data; + in_offset = op->turbo_enc.input.offset; + out_offset = op->turbo_enc.output.offset; + total_left = op->turbo_enc.input.length; + k = op->turbo_enc.cb_params.k; + e = op->turbo_enc.cb_params.e; + ncb = op->turbo_enc.cb_params.ncb; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_CRC_24B_ATTACH)) + in_length = ((k - 24) >> 3); + else + in_length = k >> 3; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_RATE_MATCH)) + out_length = (e + 7) >> 3; + else + out_length = (k >> 3) * 3 + 2; + + mbuf_append(output, output, out_length); + + /* Offset into the ring */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + /* Setup DMA Descriptor */ + desc = q->ring_addr + ring_offset; + + ret = fpga_dma_desc_te_fill(op, &desc->enc_req, input, output, k, e, + ncb, in_offset, out_offset, ring_offset, 1); + if (unlikely(ret < 0)) + return ret; + + /* Update lengths */ + total_left -= in_length; + op->turbo_enc.output.length += out_length; + + if (total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + total_left, in_length); + return -1; + } + + return 1; +} + +static inline int +enqueue_enc_one_op_tb(struct fpga_queue *q, struct rte_bbdev_enc_op *op, + uint16_t desc_offset, uint8_t cbs_in_op) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input, *output_head, *output; + int ret; + uint8_t r, c, crc24_bits = 0; + uint16_t k, e, ncb, ring_offset; + uint32_t mbuf_total_left, in_length, out_length, in_offset, out_offset; + uint32_t seg_total_left; + uint16_t current_enqueued_cbs = 0; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_enc_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo encoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_enc.input.data; + output_head = output = op->turbo_enc.output.data; + in_offset = op->turbo_enc.input.offset; + out_offset = op->turbo_enc.output.offset; + mbuf_total_left = op->turbo_enc.input.length; + + c = op->turbo_enc.tb_params.c; + r = op->turbo_enc.tb_params.r; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_CRC_24B_ATTACH)) + crc24_bits = 24; + + while (mbuf_total_left > 0 && r < c && input != NULL) { + seg_total_left = rte_pktmbuf_data_len(input) - in_offset; + + e = (r < op->turbo_enc.tb_params.cab) ? + op->turbo_enc.tb_params.ea : + op->turbo_enc.tb_params.eb; + k = (r < op->turbo_enc.tb_params.c_neg) ? + op->turbo_enc.tb_params.k_neg : + op->turbo_enc.tb_params.k_pos; + ncb = (r < op->turbo_enc.tb_params.c_neg) ? + op->turbo_enc.tb_params.ncb_neg : + op->turbo_enc.tb_params.ncb_pos; + + in_length = ((k - crc24_bits) >> 3); + + if (check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_RATE_MATCH)) + out_length = (e + 7) >> 3; + else + out_length = (k >> 3) * 3 + 2; + + mbuf_append(output_head, output, out_length); + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_te_fill(op, &desc->enc_req, input, output, + k, e, ncb, in_offset, out_offset, ring_offset, + cbs_in_op); + if (unlikely(ret < 0)) + return ret; + + rte_bbdev_log_debug("DMA request desc %p", desc); + + /* Update lengths */ + op->turbo_enc.output.length += out_length; + mbuf_total_left -= in_length; + + /* Update offsets */ + if (seg_total_left == in_length) { + /* Go to the next mbuf */ + input = input->next; + output = output->next; + in_offset = 0; + out_offset = 0; + } else { + in_offset += in_length; + out_offset += out_length; + } + + r++; + desc_offset++; + current_enqueued_cbs++; + } + + if (mbuf_total_left > 0) { + rte_bbdev_log(ERR, + "Some date still left for processing: mbuf_total_left = %u", + mbuf_total_left); + return -1; + } + + return current_enqueued_cbs; +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Validates turbo decoder parameters */ +static int +validate_dec_op(struct rte_bbdev_dec_op *op) +{ + struct rte_bbdev_op_turbo_dec *turbo_dec = &op->turbo_dec; + struct rte_bbdev_op_dec_cb_params *cb = NULL; + struct rte_bbdev_op_dec_tb_params *tb = NULL; + + if (op->mempool == NULL) { + rte_bbdev_log(ERR, "Invalid mempool pointer"); + return -1; + } + if (turbo_dec->input.data == NULL) { + rte_bbdev_log(ERR, "Invalid input pointer"); + return -1; + } + if (turbo_dec->hard_output.data == NULL) { + rte_bbdev_log(ERR, "Invalid hard_output pointer"); + return -1; + } + if (turbo_dec->rv_index > 3) { + rte_bbdev_log(ERR, + "rv_index (%u) is out of range 0 <= value <= 3", + turbo_dec->rv_index); + return -1; + } + if (turbo_dec->iter_min < 1) { + rte_bbdev_log(ERR, + "iter_min (%u) is less than 1", + turbo_dec->iter_min); + return -1; + } + if (turbo_dec->iter_max <= 2) { + rte_bbdev_log(ERR, + "iter_max (%u) is less than or equal to 2", + turbo_dec->iter_max); + return -1; + } + if (turbo_dec->iter_min > turbo_dec->iter_max) { + rte_bbdev_log(ERR, + "iter_min (%u) is greater than iter_max (%u)", + turbo_dec->iter_min, turbo_dec->iter_max); + return -1; + } + if (turbo_dec->code_block_mode != 0 && + turbo_dec->code_block_mode != 1) { + rte_bbdev_log(ERR, + "code_block_mode (%u) is out of range 0 <= value <= 1", + turbo_dec->code_block_mode); + return -1; + } + + if (turbo_dec->code_block_mode == 0) { + + if ((turbo_dec->op_flags & + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP) && + !(turbo_dec->op_flags & RTE_BBDEV_TURBO_CRC_TYPE_24B)) { + rte_bbdev_log(ERR, + "RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP should accompany RTE_BBDEV_TURBO_CRC_TYPE_24B"); + return -1; + } + + tb = &turbo_dec->tb_params; + if ((tb->k_neg < RTE_BBDEV_MIN_CB_SIZE + || tb->k_neg > RTE_BBDEV_MAX_CB_SIZE) + && tb->c_neg > 0) { + rte_bbdev_log(ERR, + "k_neg (%u) is out of range %u <= value <= %u", + tb->k_neg, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if ((tb->k_pos < RTE_BBDEV_MIN_CB_SIZE + || tb->k_pos > RTE_BBDEV_MAX_CB_SIZE) + && tb->c > tb->c_neg) { + rte_bbdev_log(ERR, + "k_pos (%u) is out of range %u <= value <= %u", + tb->k_pos, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->c_neg > (RTE_BBDEV_MAX_CODE_BLOCKS - 1)) + rte_bbdev_log(ERR, + "c_neg (%u) is out of range 0 <= value <= %u", + tb->c_neg, + RTE_BBDEV_MAX_CODE_BLOCKS - 1); + if (tb->c < 1 || tb->c > RTE_BBDEV_MAX_CODE_BLOCKS) { + rte_bbdev_log(ERR, + "c (%u) is out of range 1 <= value <= %u", + tb->c, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + if (tb->cab > tb->c) { + rte_bbdev_log(ERR, + "cab (%u) is greater than c (%u)", + tb->cab, tb->c); + return -1; + } + } else { + + if (turbo_dec->op_flags & RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP) { + rte_bbdev_log(ERR, + "RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP is invalid in CB-mode"); + return -1; + } + + cb = &turbo_dec->cb_params; + if (cb->k < RTE_BBDEV_MIN_CB_SIZE + || cb->k > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k (%u) is out of range %u <= value <= %u", + cb->k, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + } + + return 0; +} +#endif + +static inline int +enqueue_dec_one_op_cb(struct fpga_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input; + struct rte_mbuf *output; + int ret; + uint16_t k, kw, ring_offset; + uint32_t total_left, in_length, out_length, in_offset, out_offset; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_dec_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo decoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_dec.input.data; + output = op->turbo_dec.hard_output.data; + total_left = op->turbo_dec.input.length; + in_offset = op->turbo_dec.input.offset; + out_offset = op->turbo_dec.hard_output.offset; + + k = op->turbo_dec.cb_params.k; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + in_length = kw; + out_length = k >> 3; + + mbuf_append(output, output, out_length); + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_td_fill(op, &desc->dec_req, input, output, + in_length, k, in_offset, out_offset, ring_offset, 1); + if (unlikely(ret < 0)) + return ret; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_dma_dec_desc_debug_info(desc); +#endif + + /* Update lengths */ + total_left -= in_length; + op->turbo_dec.hard_output.length += out_length; + + if (total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + total_left, in_length); + return -1; + } + + return 1; +} + + +static inline int +enqueue_dec_one_op_tb(struct fpga_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset, uint8_t cbs_in_op) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input, *output_head, *output; + int ret; + uint8_t r, c; + uint16_t k, kw, in_length, out_length, ring_offset; + uint32_t mbuf_total_left, seg_total_left, in_offset, out_offset; + uint16_t current_enqueued_cbs = 0; + uint16_t crc24_overlap = 0; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_dec_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo decoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_dec.input.data; + output_head = output = op->turbo_dec.hard_output.data; + mbuf_total_left = op->turbo_dec.input.length; + in_offset = op->turbo_dec.input.offset; + out_offset = op->turbo_dec.hard_output.offset; + + if (!check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP)) + crc24_overlap = 24; + + c = op->turbo_dec.tb_params.c; + r = op->turbo_dec.tb_params.r; + + while (mbuf_total_left > 0 && r < c && input != NULL) { + seg_total_left = rte_pktmbuf_data_len(input) - in_offset; + k = (r < op->turbo_dec.tb_params.c_neg) ? + op->turbo_dec.tb_params.k_neg : + op->turbo_dec.tb_params.k_pos; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + + in_length = kw; + out_length = (k - crc24_overlap) >> 3; + + mbuf_append(output_head, output, out_length); + + if (seg_total_left < in_length) { + rte_bbdev_log(ERR, + "Partial CB found in a TB. FPGA Driver doesn't support scatter-gather operations!"); + return -1; + } + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_td_fill(op, &desc->dec_req, input, output, + in_length, k, in_offset, out_offset, + ring_offset, cbs_in_op); + if (unlikely(ret < 0)) + return ret; + + /* Update lengths */ + ret = rte_pktmbuf_trim(op->turbo_dec.hard_output.data, + (crc24_overlap >> 3)); +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (ret < 0) { + rte_bbdev_log(ERR, + "The length to remove is greater than the length of the last segment"); + return -EINVAL; + } +#endif + op->turbo_dec.hard_output.length += out_length; + mbuf_total_left -= in_length; + + /* Update offsets */ + if (seg_total_left == in_length) { + /* Go to the next mbuf */ + input = input->next; + output = output->next; + in_offset = 0; + out_offset = 0; + } else { + in_offset += in_length; + out_offset += out_length; + } + + r++; + desc_offset++; + current_enqueued_cbs++; + } + + if (mbuf_total_left > 0) { + rte_bbdev_log(ERR, + "Some date still left for processing: mbuf_total_left = %u", + mbuf_total_left); + return -1; + } + + return current_enqueued_cbs; +} + +static uint16_t +fpga_enqueue_enc(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num) +{ + uint8_t cbs_in_op; + uint16_t i, total_enqueued_cbs = 0; + int32_t avail; + int enqueued_cbs; + struct fpga_queue *q = q_data->queue_private; + union fpga_dma_desc *desc; + + /* Check if queue is not full */ + if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == + q->head_free_desc)) + return 0; + + /* Calculates available space */ + avail = (q->head_free_desc > q->tail) ? + q->head_free_desc - q->tail - 1 : + q->ring_ctrl_reg.ring_size + q->head_free_desc - q->tail - 1; + + for (i = 0; i < num; ++i) { + if (ops[i]->turbo_enc.code_block_mode == 0) { + cbs_in_op = get_num_cbs_in_op_enc(&ops[i]->turbo_enc); + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - cbs_in_op < 0)) + break; + avail -= cbs_in_op; + enqueued_cbs = enqueue_enc_one_op_tb(q, ops[i], + total_enqueued_cbs, cbs_in_op); + } else { + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - 1 < 0)) + break; + avail -= 1; + enqueued_cbs = enqueue_enc_one_op_cb(q, ops[i], + total_enqueued_cbs); + } + + if (enqueued_cbs < 0) + break; + + total_enqueued_cbs += enqueued_cbs; + + rte_bbdev_log_debug("enqueuing enc ops [%d/%d] | head %d | tail %d", + total_enqueued_cbs, num, + q->head_free_desc, q->tail); + } + + /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt + * only when all previous CBs were already processed. + */ + desc = q->ring_addr + ((q->tail + total_enqueued_cbs - 1) + & q->sw_ring_wrap_mask); + desc->enc_req.irq_en = q->irq_enable; + + fpga_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); + + /* Update stats */ + q_data->queue_stats.enqueued_count += i; + q_data->queue_stats.enqueue_err_count += num - i; + + return i; +} + +static uint16_t +fpga_enqueue_dec(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num) +{ + uint8_t cbs_in_op; + uint16_t i, total_enqueued_cbs = 0; + int32_t avail; + int enqueued_cbs; + struct fpga_queue *q = q_data->queue_private; + union fpga_dma_desc *desc; + + /* Check if queue is not full */ + if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == + q->head_free_desc)) + return 0; + + /* Calculates available space */ + avail = (q->head_free_desc > q->tail) ? + q->head_free_desc - q->tail - 1 : + q->ring_ctrl_reg.ring_size + q->head_free_desc - q->tail - 1; + + for (i = 0; i < num; ++i) { + if (ops[i]->turbo_dec.code_block_mode == 0) { + cbs_in_op = get_num_cbs_in_op_dec(&ops[i]->turbo_dec); + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - cbs_in_op < 0)) + break; + avail -= cbs_in_op; + enqueued_cbs = enqueue_dec_one_op_tb(q, ops[i], + total_enqueued_cbs, cbs_in_op); + } else { + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - 1 < 0)) + break; + avail -= 1; + enqueued_cbs = enqueue_dec_one_op_cb(q, ops[i], + total_enqueued_cbs); + } + + if (enqueued_cbs < 0) + break; + + total_enqueued_cbs += enqueued_cbs; + + rte_bbdev_log_debug("enqueuing dec ops [%d/%d] | head %d | tail %d", + total_enqueued_cbs, num, + q->head_free_desc, q->tail); + } + + /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt + * only when all previous CBs were already processed. + */ + desc = q->ring_addr + ((q->tail + total_enqueued_cbs - 1) + & q->sw_ring_wrap_mask); + desc->dec_req.irq_en = q->irq_enable; + + fpga_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); + + /* Update stats */ + q_data->queue_stats.enqueued_count += i; + q_data->queue_stats.enqueue_err_count += num - i; + + return i; +} + +static inline int +dequeue_enc_one_op_cb(struct fpga_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + int desc_error = 0; + + /* Set current desc */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /*check if done */ + if (desc->enc_req.done == 0) + return -1; + + /* make sure the response is read atomically */ + rte_smp_rmb(); + + rte_bbdev_log_debug("DMA response desc %p", desc); + + *op = desc->enc_req.op_addr; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + (*op)->status = desc_error << RTE_BBDEV_DATA_ERROR; + + return 1; +} + +static inline int +dequeue_enc_one_op_tb(struct fpga_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + uint8_t cbs_in_op, cb_idx; + int desc_error = 0; + int status = 0; + + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify if done bit is set */ + if (desc->enc_req.done == 0) + return -1; + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + /* Verify if done bit in all CBs is set */ + cbs_in_op = desc->enc_req.cbs_in_op; + for (cb_idx = 1; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + if (desc->enc_req.done == 0) + return -1; + } + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + for (cb_idx = 0; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + status |= desc_error << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log_debug("DMA response desc %p", desc); + } + + *op = desc->enc_req.op_addr; + (*op)->status = status; + return cbs_in_op; +} + +static inline int +dequeue_dec_one_op_cb(struct fpga_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + int desc_error = 0; + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify done bit is set */ + if (desc->dec_req.done == 0) + return -1; + + /* make sure the response is read atomically */ + rte_smp_rmb(); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_dma_dec_desc_debug_info(desc); + +#endif + + *op = desc->dec_req.op_addr; + /* FPGA reports in half-iterations, from 0 to 31. get ceiling */ + (*op)->turbo_dec.iter_count = (desc->dec_req.iter + 2) >> 1; + /* crc_pass = 0 when decoder fails */ + (*op)->status = !(desc->dec_req.crc_pass) << RTE_BBDEV_CRC_ERROR; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + (*op)->status |= desc_error << RTE_BBDEV_DATA_ERROR; + return 1; +} + +static inline int +dequeue_dec_one_op_tb(struct fpga_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + uint8_t cbs_in_op, cb_idx, iter_count = 0; + int status = 0; + int desc_error = 0; + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify if done bit is set */ + if (desc->dec_req.done == 0) + return -1; + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + /* Verify if done bit in all CBs is set */ + cbs_in_op = desc->dec_req.cbs_in_op; + for (cb_idx = 1; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + if (desc->dec_req.done == 0) + return -1; + } + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + for (cb_idx = 0; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + /* get max iter_count for all CBs in op */ + iter_count = RTE_MAX(iter_count, (uint8_t) desc->dec_req.iter); + /* crc_pass = 0 when decoder fails, one fails all */ + status |= !(desc->dec_req.crc_pass) << RTE_BBDEV_CRC_ERROR; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + status |= desc_error << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log_debug("DMA response desc %p", desc); + } + + *op = desc->dec_req.op_addr; + + /* FPGA reports in half-iterations, get ceiling */ + (*op)->turbo_dec.iter_count = (iter_count + 2) >> 1; + (*op)->status = status; + return cbs_in_op; +} + +static uint16_t +fpga_dequeue_enc(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num) +{ + struct fpga_queue *q = q_data->queue_private; + uint32_t avail = (q->tail - q->head_free_desc) & q->sw_ring_wrap_mask; + uint16_t i; + uint16_t dequeued_cbs = 0; + struct rte_bbdev_enc_op *op; + int ret; + + for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { + op = (q->ring_addr + ((q->head_free_desc + dequeued_cbs) + & q->sw_ring_wrap_mask))->enc_req.op_addr; + if (op->turbo_enc.code_block_mode == 0) + ret = dequeue_enc_one_op_tb(q, &ops[i], dequeued_cbs); + else + ret = dequeue_enc_one_op_cb(q, &ops[i], dequeued_cbs); + + if (ret < 0) + break; + + dequeued_cbs += ret; + + rte_bbdev_log_debug("dequeuing enc ops [%d/%d] | head %d | tail %d", + dequeued_cbs, num, q->head_free_desc, q->tail); + } + + /* Update head */ + q->head_free_desc = (q->head_free_desc + dequeued_cbs) & + q->sw_ring_wrap_mask; + + /* Update stats */ + q_data->queue_stats.dequeued_count += i; + + return i; +} + +static uint16_t +fpga_dequeue_dec(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num) +{ + struct fpga_queue *q = q_data->queue_private; + uint32_t avail = (q->tail - q->head_free_desc) & q->sw_ring_wrap_mask; + uint16_t i; + uint16_t dequeued_cbs = 0; + struct rte_bbdev_dec_op *op; + int ret; + + for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { + op = (q->ring_addr + ((q->head_free_desc + dequeued_cbs) + & q->sw_ring_wrap_mask))->dec_req.op_addr; + if (op->turbo_dec.code_block_mode == 0) + ret = dequeue_dec_one_op_tb(q, &ops[i], dequeued_cbs); + else + ret = dequeue_dec_one_op_cb(q, &ops[i], dequeued_cbs); + + if (ret < 0) + break; + + dequeued_cbs += ret; + + rte_bbdev_log_debug("dequeuing dec ops [%d/%d] | head %d | tail %d", + dequeued_cbs, num, q->head_free_desc, q->tail); + } + + /* Update head */ + q->head_free_desc = (q->head_free_desc + dequeued_cbs) & + q->sw_ring_wrap_mask; + + /* Update stats */ + q_data->queue_stats.dequeued_count += i; + + return i; +} + +/* Initialization Function */ +static void +fpga_lte_fec_init(struct rte_bbdev *dev) +{ + struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev->device); + + dev->dev_ops = &fpga_ops; + dev->enqueue_enc_ops = fpga_enqueue_enc; + dev->enqueue_dec_ops = fpga_enqueue_dec; + dev->dequeue_enc_ops = fpga_dequeue_enc; + dev->dequeue_dec_ops = fpga_dequeue_dec; + + ((struct fpga_lte_fec_device *) dev->data->dev_private)->pf_device = + !strcmp(dev->device->driver->name, + RTE_STR(FPGA_LTE_FEC_PF_DRIVER_NAME)); + ((struct fpga_lte_fec_device *) dev->data->dev_private)->mmio_base = + pci_dev->mem_resource[0].addr; + + rte_bbdev_log_debug( + "Init device %s [%s] @ virtaddr %p phyaddr %#"PRIx64, + dev->device->driver->name, dev->data->name, + (void *)pci_dev->mem_resource[0].addr, + pci_dev->mem_resource[0].phys_addr); +} + +static int +fpga_lte_fec_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct rte_bbdev *bbdev = NULL; + char dev_name[RTE_BBDEV_NAME_MAX_LEN]; + + if (pci_dev == NULL) { + rte_bbdev_log(ERR, "NULL PCI device"); + return -EINVAL; + } + + rte_pci_device_name(&pci_dev->addr, dev_name, sizeof(dev_name)); + + /* Allocate memory to be used privately by drivers */ + bbdev = rte_bbdev_allocate(pci_dev->device.name); + if (bbdev == NULL) + return -ENODEV; + + /* allocate device private memory */ + bbdev->data->dev_private = rte_zmalloc_socket(dev_name, + sizeof(struct fpga_lte_fec_device), RTE_CACHE_LINE_SIZE, + pci_dev->device.numa_node); + + if (bbdev->data->dev_private == NULL) { + rte_bbdev_log(CRIT, + "Allocate of %zu bytes for device \"%s\" failed", + sizeof(struct fpga_lte_fec_device), dev_name); + rte_bbdev_release(bbdev); + return -ENOMEM; + } + + /* Fill HW specific part of device structure */ + bbdev->device = &pci_dev->device; + bbdev->intr_handle = &pci_dev->intr_handle; + bbdev->data->socket_id = pci_dev->device.numa_node; + + /* Invoke FEC FPGA device initialization function */ + fpga_lte_fec_init(bbdev); + + rte_bbdev_log_debug("bbdev id = %u [%s]", + bbdev->data->dev_id, dev_name); + + struct fpga_lte_fec_device *d = bbdev->data->dev_private; + uint32_t version_id = fpga_reg_read_32(d->mmio_base, + FPGA_LTE_FEC_VERSION_ID); + rte_bbdev_log(INFO, "FEC FPGA RTL v%u.%u", + ((uint16_t)(version_id >> 16)), ((uint16_t)version_id)); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (!strcmp(bbdev->device->driver->name, + RTE_STR(FPGA_LTE_FEC_PF_DRIVER_NAME))) + print_static_reg_debug_info(d->mmio_base); +#endif + return 0; +} + +static int +fpga_lte_fec_remove(struct rte_pci_device *pci_dev) +{ + struct rte_bbdev *bbdev; + int ret; + uint8_t dev_id; + + if (pci_dev == NULL) + return -EINVAL; + + /* Find device */ + bbdev = rte_bbdev_get_named_dev(pci_dev->device.name); + if (bbdev == NULL) { + rte_bbdev_log(CRIT, + "Couldn't find HW dev \"%s\" to uninitialise it", + pci_dev->device.name); + return -ENODEV; + } + dev_id = bbdev->data->dev_id; + + /* free device private memory before close */ + rte_free(bbdev->data->dev_private); + + /* Close device */ + ret = rte_bbdev_close(dev_id); + if (ret < 0) + rte_bbdev_log(ERR, + "Device %i failed to close during uninit: %i", + dev_id, ret); + + /* release bbdev from library */ + ret = rte_bbdev_release(bbdev); + if (ret) + rte_bbdev_log(ERR, "Device %i failed to uninit: %i", dev_id, + ret); + + rte_bbdev_log_debug("Destroyed bbdev = %u", dev_id); + + return 0; +} + +static inline void +set_default_fpga_conf(struct fpga_lte_fec_conf *def_conf) +{ + /* clear default configuration before initialization */ + memset(def_conf, 0, sizeof(struct fpga_lte_fec_conf)); + /* Set pf mode to true */ + def_conf->pf_mode_en = true; + + /* Set ratio between UL and DL to 1:1 (unit of weight is 3 CBs) */ + def_conf->ul_bandwidth = 3; + def_conf->dl_bandwidth = 3; + + /* Set Load Balance Factor to 64 */ + def_conf->dl_load_balance = 64; + def_conf->ul_load_balance = 64; +} + +/* Initial configuration of FPGA LTE FEC device */ +int +fpga_lte_fec_configure(const char *dev_name, + const struct fpga_lte_fec_conf *conf) +{ + uint32_t payload_32, address; + uint16_t payload_16; + uint8_t payload_8; + uint16_t q_id, vf_id, total_q_id, total_ul_q_id, total_dl_q_id; + struct rte_bbdev *bbdev = rte_bbdev_get_named_dev(dev_name); + struct fpga_lte_fec_conf def_conf; + + if (bbdev == NULL) { + rte_bbdev_log(ERR, + "Invalid dev_name (%s), or device is not yet initialised", + dev_name); + return -ENODEV; + } + + struct fpga_lte_fec_device *d = bbdev->data->dev_private; + + if (conf == NULL) { + rte_bbdev_log(ERR, + "FPGA Configuration was not provided. Default configuration will be loaded."); + set_default_fpga_conf(&def_conf); + conf = &def_conf; + } + + /* + * Configure UL:DL ratio. + * [7:0]: UL weight + * [15:8]: DL weight + */ + payload_16 = (conf->dl_bandwidth << 8) | conf->ul_bandwidth; + address = FPGA_LTE_FEC_CONFIGURATION; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Clear all queues registers */ + payload_32 = FPGA_INVALID_HW_QUEUE_ID; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + address = (q_id << 2) + FPGA_LTE_FEC_QUEUE_MAP; + fpga_reg_write_32(d->mmio_base, address, payload_32); + } + + /* + * If PF mode is enabled allocate all queues for PF only. + * + * For VF mode each VF can have different number of UL and DL queues. + * Total number of queues to configure cannot exceed FPGA + * capabilities - 64 queues - 32 queues for UL and 32 queues for DL. + * Queues mapping is done according to configuration: + * + * UL queues: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | ... | 0 | + * | conf->vf_dl_queues_number[0] - 1 | 0 | + * | conf->vf_dl_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_dl_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_dl_queues_number[7] - 1 | 7 | + * + * DL queues: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | ... | 0 | + * | conf->vf_ul_queues_number[0] - 1 | 0 | + * | conf->vf_ul_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_ul_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_ul_queues_number[7] - 1 | 7 | + * + * Example of configuration: + * conf->vf_ul_queues_number[0] = 4; -> 4 UL queues for VF0 + * conf->vf_dl_queues_number[0] = 4; -> 4 DL queues for VF0 + * conf->vf_ul_queues_number[1] = 2; -> 2 UL queues for VF1 + * conf->vf_dl_queues_number[1] = 2; -> 2 DL queues for VF1 + * + * UL: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | 1 | 0 | + * | 2 | 0 | + * | 3 | 0 | + * | 4 | 1 | + * | 5 | 1 | + * + * DL: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | 33 | 0 | + * | 34 | 0 | + * | 35 | 0 | + * | 36 | 1 | + * | 37 | 1 | + */ + if (conf->pf_mode_en) { + payload_32 = 0x1; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + address = (q_id << 2) + FPGA_LTE_FEC_QUEUE_MAP; + fpga_reg_write_32(d->mmio_base, address, payload_32); + } + } else { + /* Calculate total number of UL and DL queues to configure */ + total_ul_q_id = total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + total_ul_q_id += conf->vf_ul_queues_number[vf_id]; + total_dl_q_id += conf->vf_dl_queues_number[vf_id]; + } + total_q_id = total_dl_q_id + total_ul_q_id; + /* + * Check if total number of queues to configure does not exceed + * FPGA capabilities (64 queues - 32 UL and 32 DL queues) + */ + if ((total_ul_q_id > FPGA_NUM_UL_QUEUES) || + (total_dl_q_id > FPGA_NUM_DL_QUEUES) || + (total_q_id > FPGA_TOTAL_NUM_QUEUES)) { + rte_bbdev_log(ERR, + "FPGA Configuration failed. Too many queues to configure: UL_Q %u, DL_Q %u, FPGA_Q %u", + total_ul_q_id, total_dl_q_id, + FPGA_TOTAL_NUM_QUEUES); + return -EINVAL; + } + total_ul_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_ul_queues_number[vf_id]; + ++q_id, ++total_ul_q_id) { + address = (total_ul_q_id << 2) + + FPGA_LTE_FEC_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_reg_write_32(d->mmio_base, address, + payload_32); + } + } + total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_dl_queues_number[vf_id]; + ++q_id, ++total_dl_q_id) { + address = ((total_dl_q_id + FPGA_NUM_UL_QUEUES) + << 2) + FPGA_LTE_FEC_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_reg_write_32(d->mmio_base, address, + payload_32); + } + } + } + + /* Setting Load Balance Factor */ + payload_16 = (conf->dl_load_balance << 8) | (conf->ul_load_balance); + address = FPGA_LTE_FEC_LOAD_BALANCE_FACTOR; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting length of ring descriptor entry */ + payload_16 = FPGA_RING_DESC_ENTRY_LENGTH; + address = FPGA_LTE_FEC_RING_DESC_LEN; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting FLR timeout value */ + payload_16 = conf->flr_time_out; + address = FPGA_LTE_FEC_FLR_TIME_OUT; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Queue PF/VF mapping table is ready */ + payload_8 = 0x1; + address = FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE; + fpga_reg_write_8(d->mmio_base, address, payload_8); + + rte_bbdev_log_debug("PF FPGA LTE FEC configuration complete for %s", + dev_name); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_static_reg_debug_info(d->mmio_base); +#endif + return 0; +} + +/* FPGA LTE FEC PCI PF address map */ +static struct rte_pci_id pci_id_fpga_lte_fec_pf_map[] = { + { + RTE_PCI_DEVICE(FPGA_LTE_FEC_VENDOR_ID, + FPGA_LTE_FEC_PF_DEVICE_ID) + }, + {.device_id = 0}, +}; + +static struct rte_pci_driver fpga_lte_fec_pci_pf_driver = { + .probe = fpga_lte_fec_probe, + .remove = fpga_lte_fec_remove, + .id_table = pci_id_fpga_lte_fec_pf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING +}; + +/* FPGA LTE FEC PCI VF address map */ +static struct rte_pci_id pci_id_fpga_lte_fec_vf_map[] = { + { + RTE_PCI_DEVICE(FPGA_LTE_FEC_VENDOR_ID, + FPGA_LTE_FEC_VF_DEVICE_ID) + }, + {.device_id = 0}, +}; + +static struct rte_pci_driver fpga_lte_fec_pci_vf_driver = { + .probe = fpga_lte_fec_probe, + .remove = fpga_lte_fec_remove, + .id_table = pci_id_fpga_lte_fec_vf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING +}; + + +RTE_PMD_REGISTER_PCI(FPGA_LTE_FEC_PF_DRIVER_NAME, fpga_lte_fec_pci_pf_driver); +RTE_PMD_REGISTER_PCI_TABLE(FPGA_LTE_FEC_PF_DRIVER_NAME, + pci_id_fpga_lte_fec_pf_map); +RTE_PMD_REGISTER_PCI(FPGA_LTE_FEC_VF_DRIVER_NAME, fpga_lte_fec_pci_vf_driver); +RTE_PMD_REGISTER_PCI_TABLE(FPGA_LTE_FEC_VF_DRIVER_NAME, + pci_id_fpga_lte_fec_vf_map); + +RTE_INIT(fpga_lte_fec_init_log) +{ + fpga_lte_fec_logtype = rte_log_register("pmd.bb.fpga_lte_fec"); + if (fpga_lte_fec_logtype >= 0) +#ifdef RTE_LIBRTE_BBDEV_DEBUG + rte_log_set_level(fpga_lte_fec_logtype, RTE_LOG_DEBUG); +#else + rte_log_set_level(fpga_lte_fec_logtype, RTE_LOG_NOTICE); +#endif +} diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h new file mode 100644 index 0000000..9ae8b12 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation + */ + +#ifndef _FPGA_LTE_FEC_H_ +#define _FPGA_LTE_FEC_H_ + +#include <stdint.h> +#include <stdbool.h> + +/** + * @file fpga_lte_fec.h + * + * Interface for Intel(R) FGPA LTE FEC device configuration at the host level, + * directly accessible by the application. + * Configuration related to LTE Turbo coding functionality is done through + * librte_bbdev library. + * + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + */ + +#ifdef __cplusplus +extern "C" { +#endif + +/**< Number of Virtual Functions FGPA 4G FEC supports */ +#define FPGA_LTE_FEC_NUM_VFS 8 + +/** + * Structure to pass FPGA 4G FEC configuration. + */ +struct fpga_lte_fec_conf { + /**< 1 if PF is used for dataplane, 0 for VFs */ + bool pf_mode_en; + /**< Number of UL queues per VF */ + uint8_t vf_ul_queues_number[FPGA_LTE_FEC_NUM_VFS]; + /**< Number of DL queues per VF */ + uint8_t vf_dl_queues_number[FPGA_LTE_FEC_NUM_VFS]; + /**< UL bandwidth. Needed for schedule algorithm */ + uint8_t ul_bandwidth; + /**< DL bandwidth. Needed for schedule algorithm */ + uint8_t dl_bandwidth; + /**< UL Load Balance */ + uint8_t ul_load_balance; + /**< DL Load Balance */ + uint8_t dl_load_balance; + /**< FLR timeout value */ + uint16_t flr_time_out; +}; + +/** + * Configure Intel(R) FPGA LTE FEC device + * + * @param dev_name + * The name of the device. This is the short form of PCI BDF, e.g. 00:01.0. + * It can also be retrieved for a bbdev device from the dev_name field in the + * rte_bbdev_info structure returned by rte_bbdev_info_get(). + * @param conf + * Configuration to apply to FPGA 4G FEC. + * + * @return + * Zero on success, negative value on failure. + */ +int +fpga_lte_fec_configure(const char *dev_name, + const struct fpga_lte_fec_conf *conf); + +#ifdef __cplusplus +} +#endif + +#endif /* _FPGA_LTE_FEC_H_ */ diff --git a/drivers/baseband/fpga_lte_fec/meson.build b/drivers/baseband/fpga_lte_fec/meson.build new file mode 100644 index 0000000..bf44e6b --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +deps += ['bbdev', 'bus_vdev', 'ring', 'pci', 'bus_pci'] +name = 'bbdev_fpga_lte_fec' +allow_experimental_apis = true +sources = files('fpga_lte_fec.c') diff --git a/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map new file mode 100644 index 0000000..e923270 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map @@ -0,0 +1,3 @@ +DPDK_19.08 { + local: *; +}; diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build index 52489df..a17d549 100644 --- a/drivers/baseband/meson.build +++ b/drivers/baseband/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <bluca@debian.org> -drivers = ['null'] +drivers = ['null', 'fpga_lte_fec'] config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' driver_name_fmt = 'rte_pmd_@0@' diff --git a/mk/rte.app.mk b/mk/rte.app.mk index b6106e1..73bba4a 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -215,6 +215,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_NETVSC_PMD) += -lrte_pmd_netvsc ifeq ($(CONFIG_RTE_LIBRTE_BBDEV),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += -lrte_pmd_fpga_lte_fec # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v6] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-10 17:01 ` Nicolas Chautru ` (2 preceding siblings ...) 2019-06-14 16:12 ` [dpdk-dev] [PATCH v6] " Nicolas Chautru @ 2019-06-14 16:17 ` Nicolas Chautru 2019-06-14 16:17 ` Nicolas Chautru 3 siblings, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-14 16:17 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, Nicolas Chautru Update v6: Update of one copyright date in documentation file. Update v5: Update date and version from dpdk review. Rebased to latest. Update v4: Fix warning for the DEBUG configuration. Update v3: Squashing 3 previous patches into one as recommended. This is adding a new PMD driver for BBDEV device based on FPGA implementation on PAC N3000 HW to provide FEC 4G acceleration. v1 was shared earlier on this patchwork : https://patches.dpdk.org/patch/53409/ The existing BBDEV test framework is still supported. Main updates based on feedback during the last v1 were: - correct the documentation which was arguably misleading in hinting that ICC was required. There is no dependency whatsoever. - add meson build support Note that a number of other BBDEV patches are in v1 here and will be resubmitted for 19.08 rc1 https://patches.dpdk.org/project/dpdk/list/?series=4657 Nicolas Chautru (1): baseband/fpga_lte_fec: adding driver for FEC on FPGA config/common_base | 6 + doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++ doc/guides/bbdevs/index.rst | 1 + drivers/baseband/Makefile | 2 + drivers/baseband/fpga_lte_fec/Makefile | 29 + drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 ++++++++++++++++++++ drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + drivers/baseband/fpga_lte_fec/meson.build | 7 + .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + drivers/baseband/meson.build | 2 +- mk/rte.app.mk | 1 + 11 files changed, 3115 insertions(+), 1 deletion(-) create mode 100644 doc/guides/bbdevs/fpga_lte_fec.rst create mode 100644 drivers/baseband/fpga_lte_fec/Makefile create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h create mode 100644 drivers/baseband/fpga_lte_fec/meson.build create mode 100644 drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v6] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-14 16:17 ` [dpdk-dev] [PATCH v6] " Nicolas Chautru @ 2019-06-14 16:17 ` Nicolas Chautru 2019-06-19 15:20 ` Chautru, Nicolas [not found] ` <EEA9FF629BF25B47BD67ADE995041EE2496B2975@IRSMSX103.ger.corp.intel.com> 0 siblings, 2 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-14 16:17 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, Nicolas Chautru Supports for FEC 4G PMD Driver on FPGA card PAC N3000 Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 6 + doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++ doc/guides/bbdevs/index.rst | 1 + drivers/baseband/Makefile | 2 + drivers/baseband/fpga_lte_fec/Makefile | 29 + drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2674 ++++++++++++++++++++ drivers/baseband/fpga_lte_fec/fpga_lte_fec.h | 73 + drivers/baseband/fpga_lte_fec/meson.build | 7 + .../rte_pmd_bbdev_fpga_lte_fec_version.map | 3 + drivers/baseband/meson.build | 2 +- mk/rte.app.mk | 1 + 11 files changed, 3115 insertions(+), 1 deletion(-) create mode 100644 doc/guides/bbdevs/fpga_lte_fec.rst create mode 100644 drivers/baseband/fpga_lte_fec/Makefile create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.c create mode 100644 drivers/baseband/fpga_lte_fec/fpga_lte_fec.h create mode 100644 drivers/baseband/fpga_lte_fec/meson.build create mode 100644 drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map diff --git a/config/common_base b/config/common_base index 6f19ad5..9632a07 100644 --- a/config/common_base +++ b/config/common_base @@ -521,6 +521,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y # EXPERIMENTAL: API may change without prior notice # CONFIG_RTE_LIBRTE_BBDEV=y +CONFIG_RTE_LIBRTE_BBDEV_DEBUG=n CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y @@ -535,6 +536,11 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n # +# Compile PMD for Intel FPGA LTE FEC bbdev device +# +CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC=y + +# # Compile generic crypto device library # CONFIG_RTE_LIBRTE_CRYPTODEV=y diff --git a/doc/guides/bbdevs/fpga_lte_fec.rst b/doc/guides/bbdevs/fpga_lte_fec.rst new file mode 100644 index 0000000..bdfbd75 --- /dev/null +++ b/doc/guides/bbdevs/fpga_lte_fec.rst @@ -0,0 +1,318 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation + +Intel(R) FPGA LTE FEC Poll Mode Driver +====================================== + +The BBDEV FPGA LTE FEC poll mode driver (PMD) supports an FPGA implementation of a VRAN +Turbo Encode / Decode LTE wireless acceleration function, using Intel's PCI-e and FPGA +based Vista Creek device. + +Features +-------- + +FPGA LTE FEC PMD supports the following features: + +- Turbo Encode in the DL with total throughput of 4.5 Gbits/s +- Turbo Decode in the UL with total throughput of 1.5 Gbits/s assuming 8 decoder iterations +- 8 VFs per PF (physical device) +- Maximum of 32 UL queues per VF +- Maximum of 32 DL queues per VF +- PCIe Gen-3 x8 Interface +- MSI-X +- SR-IOV + + +FPGA LTE FEC PMD supports the following BBDEV capabilities: + +* For the turbo encode operation: + - ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` : set to attach CRC24B to CB(s) + - ``RTE_BBDEV_TURBO_RATE_MATCH`` : if set then do not do Rate Match bypass + - ``RTE_BBDEV_TURBO_ENC_INTERRUPTS`` : set for encoder dequeue interrupts + + +* For the turbo decode operation: + - ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` : check CRC24B from CB(s) + - ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` : perform subblock de-interleave + - ``RTE_BBDEV_TURBO_DEC_INTERRUPTS`` : set for decoder dequeue interrupts + - ``RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN`` : set if negative LLR encoder i/p is supported + - ``RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP`` : keep CRC24B bits appended while decoding + + +Limitations +----------- + +FPGA LTE FEC does not support the following: + +- Scatter-Gather function + + +Installation +-------------- + +Section 3 of the DPDK manual provides instuctions on installing and compiling DPDK. The +default set of bbdev compile flags may be found in config/common_base, where for example +the flag to build the FPGA LTE FEC device, ``CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC``, is already +set. It is assumed DPDK has been compiled using for instance: + +.. code-block:: console + + make install T=x86_64-native-linuxapp-gcc + + +DPDK requires hugepages to be configured as detailed in section 2 of the DPDK manual. +The bbdev test application has been tested with a configuration 40 x 1GB hugepages. The +hugepage configuration of a server may be examined using: + +.. code-block:: console + + grep Huge* /proc/meminfo + + +Initialization +-------------- + +When the device first powers up, its PCI Physical Functions (PF) can be listed through this command: + +.. code-block:: console + + sudo lspci -vd1172:5052 + +The physical and virtual functions are compatible with Linux UIO drivers: +``vfio`` and ``igb_uio``. However, in order to work the FPGA LTE FEC device firstly needs +to be bound to one of these linux drivers through DPDK. + + +Bind PF UIO driver(s) +~~~~~~~~~~~~~~~~~~~~~ + +Install the DPDK igb_uio driver, bind it with the PF PCI device ID and use +``lspci`` to confirm the PF device is under use by ``igb_uio`` DPDK UIO driver. + +The igb_uio driver may be bound to the PF PCI device using one of three methods: + + +1. PCI functions (physical or virtual, depending on the use case) can be bound to +the UIO driver by repeating this command for every function. + +.. code-block:: console + + cd <dpdk-top-level-directory> + insmod ./build/kmod/igb_uio.ko + echo "1172 5052" > /sys/bus/pci/drivers/igb_uio/new_id + lspci -vd1172: + + +2. Another way to bind PF with DPDK UIO driver is by using the ``dpdk-devbind.py`` tool + +.. code-block:: console + + cd <dpdk-top-level-directory> + ./usertools/dpdk-devbind.py -b igb_uio 0000:06:00.0 + +where the PCI device ID (example: 0000:06:00.0) is obtained using lspci -vd1172: + + +3. A third way to bind is to use ``dpdk-setup.sh`` tool + +.. code-block:: console + + cd <dpdk-top-level-directory> + ./usertools/dpdk-setup.sh + + select 'Bind Ethernet/Crypto/Baseband device to IGB UIO module' + or + select 'Bind Ethernet/Crypto/Baseband device to VFIO module' depending on driver required + enter PCI device ID + select 'Display current Ethernet/Crypto/Baseband device settings' to confirm binding + + +In the same way the FPGA LTE FEC PF can be bound with vfio, but vfio driver does not +support SR-IOV configuration right out of the box, so it will need to be patched. + + +Enable Virtual Functions +~~~~~~~~~~~~~~~~~~~~~~~~ + +Now, it should be visible in the printouts that PCI PF is under igb_uio control +"``Kernel driver in use: igb_uio``" + +To show the number of available VFs on the device, read ``sriov_totalvfs`` file.. + +.. code-block:: console + + cat /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_totalvfs + + where 0000\:<b>\:<d>.<f> is the PCI device ID + + +To enable VFs via igb_uio, echo the number of virtual functions intended to +enable to ``max_vfs`` file.. + +.. code-block:: console + + echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/max_vfs + + +Afterwards, all VFs must be bound to appropriate UIO drivers as required, same +way it was done with the physical function previously. + +Enabling SR-IOV via vfio driver is pretty much the same, except that the file +name is different: + +.. code-block:: console + + echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_numvfs + + +Configure the VFs through PF +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The PCI virtual functions must be configured before working or getting assigned +to VMs/Containers. The configuration involves allocating the number of hardware +queues, priorities, load balance, bandwidth and other settings necessary for the +device to perform FEC functions. + +This configuration needs to be executed at least once after reboot or PCI FLR and can +be achieved by using the function ``fpga_lte_fec_configure()``, which sets up the +parameters defined in ``fpga_lte_fec_conf`` structure: + +.. code-block:: c + + struct fpga_lte_fec_conf { + bool pf_mode_en; + uint8_t vf_ul_queues_number[FPGA_LTE_FEC_NUM_VFS]; + uint8_t vf_dl_queues_number[FPGA_LTE_FEC_NUM_VFS]; + uint8_t ul_bandwidth; + uint8_t dl_bandwidth; + uint8_t ul_load_balance; + uint8_t dl_load_balance; + uint16_t flr_time_out; + }; + +- ``pf_mode_en``: identifies whether only PF is to be used, or the VFs. PF and + VFs are mutually exclusive and cannot run simultaneously. + Set to 1 for PF mode enabled. + If PF mode is enabled all queues available in the device are assigned + exclusively to PF and 0 queues given to VFs. + +- ``vf_*l_queues_number``: defines the hardware queue mapping for every VF. + +- ``*l_bandwidth``: in case of congestion on PCIe interface. The device + allocates different bandwidth to UL and DL. The weight is configured by this + setting. The unit of weight is 3 code blocks. For example, if the code block + cbps (code block per second) ratio between UL and DL is 12:1, then the + configuration value should be set to 36:3. The schedule algorithm is based + on code block regardless the length of each block. + +- ``*l_load_balance``: hardware queues are load-balanced in a round-robin + fashion. Queues get filled first-in first-out until they reach a pre-defined + watermark level, if exceeded, they won't get assigned new code blocks.. + This watermark is defined by this setting. + + If all hardware queues exceeds the watermark, no code blocks will be + streamed in from UL/DL code block FIFO. + +- ``flr_time_out``: specifies how many 16.384us to be FLR time out. The + time_out = flr_time_out x 16.384us. For instance, if you want to set 10ms for + the FLR time out then set this setting to 0x262=610. + + +An example configuration code calling the function ``fpga_lte_fec_configure()`` is shown +below: + +.. code-block:: c + + struct fpga_lte_fec_conf conf; + unsigned int i; + + memset(&conf, 0, sizeof(struct fpga_lte_fec_conf)); + conf.pf_mode_en = 1; + + for (i = 0; i < FPGA_LTE_FEC_NUM_VFS; ++i) { + conf.vf_ul_queues_number[i] = 4; + conf.vf_dl_queues_number[i] = 4; + } + conf.ul_bandwidth = 12; + conf.dl_bandwidth = 5; + conf.dl_load_balance = 64; + conf.ul_load_balance = 64; + + /* setup FPGA PF */ + ret = fpga_lte_fec_configure(info->dev_name, &conf); + TEST_ASSERT_SUCCESS(ret, + "Failed to configure 4G FPGA PF for bbdev %s", + info->dev_name); + + +Test Application +---------------- + +BBDEV provides a test application, ``test-bbdev.py`` and range of test data for testing +the functionality of FPGA LTE FEC turbo encode and turbo decode, depending on the device's +capabilities. The test application is located under app->test-bbdev folder and has the +following options: + +.. code-block:: console + + "-p", "--testapp-path": specifies path to the bbdev test app. + "-e", "--eal-params" : EAL arguments which are passed to the test app. + "-t", "--timeout" : Timeout in seconds (default=300). + "-c", "--test-cases" : Defines test cases to run. Run all if not specified. + "-v", "--test-vector" : Test vector path (default=dpdk_path+/app/test-bbdev/test_vectors/bbdev_null.data). + "-n", "--num-ops" : Number of operations to process on device (default=32). + "-b", "--burst-size" : Operations enqueue/dequeue burst size (default=32). + "-l", "--num-lcores" : Number of lcores to run (default=16). + "-i", "--init-device" : Initialise PF device with default values. + + +To execute the test application tool using simple turbo decode or turbo encode data, +type one of the following: + +.. code-block:: console + + ./test-bbdev.py -c validation -n 64 -b 8 -v ./turbo_dec_default.data + ./test-bbdev.py -c validation -n 64 -b 8 -v ./turbo_enc_default.data + + +The test application ``test-bbdev.py``, supports the ability to configure the PF device with +a default set of values, if the "-i" or "- -init-device" option is included. The default values +are defined in test_bbdev_perf.c as: + +- VF_UL_QUEUE_VALUE 4 +- VF_DL_QUEUE_VALUE 4 +- UL_BANDWIDTH 3 +- DL_BANDWIDTH 3 +- UL_LOAD_BALANCE 128 +- DL_LOAD_BALANCE 128 +- FLR_TIMEOUT 610 + + +Test Vectors +~~~~~~~~~~~~ + +In addition to the simple turbo decoder and turbo encoder tests, bbdev also provides +a range of additional tests under the test_vectors folder, which may be useful. The results +of these tests will depend on the FPGA LTE FEC capabilities: + +* turbo decoder tests: + - ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_high_snr.data`` + - ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_low_snr.data`` + - ``turbo_dec_c1_k6144_r0_e34560_negllr.data`` + - ``turbo_dec_c1_k6144_r0_e34560_sbd_negllr.data`` + - ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr_crc24b.data`` + - ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr.data`` + + +* turbo encoder tests: + - ``turbo_enc_c1_k40_r0_e1190_rm.data`` + - ``turbo_enc_c1_k40_r0_e1194_rm.data`` + - ``turbo_enc_c1_k40_r0_e1196_rm.data`` + - ``turbo_enc_c1_k40_r0_e272_rm.data`` + - ``turbo_enc_c1_k6144_r0_e18444.data`` + - ``turbo_enc_c1_k6144_r0_e32256_crc24b_rm.data`` + - ``turbo_enc_c2_k5952_r0_e17868_crc24b.data`` + - ``turbo_enc_c3_k4800_r2_e14412_crc24b.data`` + - ``turbo_enc_c4_k4800_r2_e14412_crc24b.data`` + + diff --git a/doc/guides/bbdevs/index.rst b/doc/guides/bbdevs/index.rst index 93276ed..005b95e 100644 --- a/doc/guides/bbdevs/index.rst +++ b/doc/guides/bbdevs/index.rst @@ -10,3 +10,4 @@ Baseband Device Drivers null turbo_sw + fpga_lte_fec diff --git a/drivers/baseband/Makefile b/drivers/baseband/Makefile index 4ec83b0..ceffc7d 100644 --- a/drivers/baseband/Makefile +++ b/drivers/baseband/Makefile @@ -10,5 +10,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += null DEPDIRS-null = $(core-libs) DIRS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += turbo_sw DEPDIRS-turbo_sw = $(core-libs) +DIRS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += fpga_lte_fec +DEPDIRS-fpga_lte_fec = $(core-libs) include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/drivers/baseband/fpga_lte_fec/Makefile b/drivers/baseband/fpga_lte_fec/Makefile new file mode 100644 index 0000000..a38a396 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/Makefile @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_pmd_fpga_lte_fec.a + +# build flags +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring +LDLIBS += -lrte_bbdev +LDLIBS += -lrte_pci -lrte_bus_pci + +# versioning export map +EXPORT_MAP := rte_pmd_bbdev_fpga_lte_fec_version.map + +# library version +LIBABIVER := 1 + +# library source files +SRCS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += fpga_lte_fec.c + +# export include files +SYMLINK-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC)-include += fpga_lte_fec.h + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c new file mode 100644 index 0000000..19e7689 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c @@ -0,0 +1,2674 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation + */ + +#include <unistd.h> + +#include <rte_common.h> +#include <rte_log.h> +#include <rte_dev.h> +#include <rte_malloc.h> +#include <rte_mempool.h> +#include <rte_errno.h> +#include <rte_pci.h> +#include <rte_bus_pci.h> +#include <rte_byteorder.h> +#ifdef RTE_BBDEV_OFFLOAD_COST +#include <rte_cycles.h> +#endif + +#include <rte_bbdev.h> +#include <rte_bbdev_pmd.h> + +#include "fpga_lte_fec.h" + +/* Turbo SW PMD logging ID */ +static int fpga_lte_fec_logtype; + +/* Helper macro for logging */ +#define rte_bbdev_log(level, fmt, ...) \ + rte_log(RTE_LOG_ ## level, fpga_lte_fec_logtype, fmt "\n", \ + ##__VA_ARGS__) + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +#define rte_bbdev_log_debug(fmt, ...) \ + rte_bbdev_log(DEBUG, "fpga_lte_fec: " fmt, \ + ##__VA_ARGS__) +#else +#define rte_bbdev_log_debug(fmt, ...) +#endif + +/* FPGA LTE FEC driver names */ +#define FPGA_LTE_FEC_PF_DRIVER_NAME intel_fpga_lte_fec_pf +#define FPGA_LTE_FEC_VF_DRIVER_NAME intel_fpga_lte_fec_vf + +/* FPGA LTE FEC PCI vendor & device IDs */ +#define FPGA_LTE_FEC_VENDOR_ID (0x1172) +#define FPGA_LTE_FEC_PF_DEVICE_ID (0x5052) +#define FPGA_LTE_FEC_VF_DEVICE_ID (0x5050) + +/* Align DMA descriptors to 256 bytes - cache-aligned */ +#define FPGA_RING_DESC_ENTRY_LENGTH (8) +/* Ring size is in 256 bits (32 bytes) units */ +#define FPGA_RING_DESC_LEN_UNIT_BYTES (32) +/* Maximum size of queue */ +#define FPGA_RING_MAX_SIZE (1024) +#define FPGA_FLR_TIMEOUT_UNIT (16.384) + +#define FPGA_NUM_UL_QUEUES (32) +#define FPGA_NUM_DL_QUEUES (32) +#define FPGA_TOTAL_NUM_QUEUES (FPGA_NUM_UL_QUEUES + FPGA_NUM_DL_QUEUES) +#define FPGA_NUM_INTR_VEC (FPGA_TOTAL_NUM_QUEUES - RTE_INTR_VEC_RXTX_OFFSET) + +#define FPGA_INVALID_HW_QUEUE_ID (0xFFFFFFFF) + +#define FPGA_QUEUE_FLUSH_TIMEOUT_US (1000) +#define FPGA_TIMEOUT_CHECK_INTERVAL (5) + +/* FPGA LTE FEC Register mapping on BAR0 */ +enum { + FPGA_LTE_FEC_VERSION_ID = 0x00000000, /* len: 4B */ + FPGA_LTE_FEC_CONFIGURATION = 0x00000004, /* len: 2B */ + FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE = 0x00000008, /* len: 1B */ + FPGA_LTE_FEC_LOAD_BALANCE_FACTOR = 0x0000000a, /* len: 2B */ + FPGA_LTE_FEC_RING_DESC_LEN = 0x0000000c, /* len: 2B */ + FPGA_LTE_FEC_FLR_TIME_OUT = 0x0000000e, /* len: 2B */ + FPGA_LTE_FEC_VFQ_FLUSH_STATUS_LW = 0x00000018, /* len: 4B */ + FPGA_LTE_FEC_VFQ_FLUSH_STATUS_HI = 0x0000001c, /* len: 4B */ + FPGA_LTE_FEC_VF0_DEBUG = 0x00000020, /* len: 4B */ + FPGA_LTE_FEC_VF1_DEBUG = 0x00000024, /* len: 4B */ + FPGA_LTE_FEC_VF2_DEBUG = 0x00000028, /* len: 4B */ + FPGA_LTE_FEC_VF3_DEBUG = 0x0000002c, /* len: 4B */ + FPGA_LTE_FEC_VF4_DEBUG = 0x00000030, /* len: 4B */ + FPGA_LTE_FEC_VF5_DEBUG = 0x00000034, /* len: 4B */ + FPGA_LTE_FEC_VF6_DEBUG = 0x00000038, /* len: 4B */ + FPGA_LTE_FEC_VF7_DEBUG = 0x0000003c, /* len: 4B */ + FPGA_LTE_FEC_QUEUE_MAP = 0x00000040, /* len: 256B */ + FPGA_LTE_FEC_RING_CTRL_REGS = 0x00000200 /* len: 2048B */ +}; + +/* FPGA LTE FEC Ring Control Registers */ +enum { + FPGA_LTE_FEC_RING_HEAD_ADDR = 0x00000008, + FPGA_LTE_FEC_RING_SIZE = 0x00000010, + FPGA_LTE_FEC_RING_MISC = 0x00000014, + FPGA_LTE_FEC_RING_ENABLE = 0x00000015, + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN = 0x00000016, + FPGA_LTE_FEC_RING_SHADOW_TAIL = 0x00000018, + FPGA_LTE_FEC_RING_HEAD_POINT = 0x0000001C +}; + +/* FPGA LTE FEC DESCRIPTOR ERROR */ +enum { + DESC_ERR_NO_ERR = 0x0, + DESC_ERR_K_OUT_OF_RANGE = 0x1, + DESC_ERR_K_NOT_NORMAL = 0x2, + DESC_ERR_KPAI_NOT_NORMAL = 0x3, + DESC_ERR_DESC_OFFSET_ERR = 0x4, + DESC_ERR_DESC_READ_FAIL = 0x8, + DESC_ERR_DESC_READ_TIMEOUT = 0x9, + DESC_ERR_DESC_READ_TLP_POISONED = 0xA, + DESC_ERR_CB_READ_FAIL = 0xC, + DESC_ERR_CB_READ_TIMEOUT = 0xD, + DESC_ERR_CB_READ_TLP_POISONED = 0xE +}; + +/* FPGA LTE FEC DMA Encoding Request Descriptor */ +struct __attribute__((__packed__)) fpga_dma_enc_desc { + uint32_t done:1, + rsrvd0:11, + error:4, + rsrvd1:16; + uint32_t ncb:16, + rsrvd2:14, + rv:2; + uint32_t bypass_rm:1, + irq_en:1, + crc_en:1, + rsrvd3:13, + offset:10, + rsrvd4:6; + uint16_t e; + uint16_t k; + uint32_t out_addr_lw; + uint32_t out_addr_hi; + uint32_t in_addr_lw; + uint32_t in_addr_hi; + + union { + struct { + /* Virtual addresses used to retrieve SW context info */ + void *op_addr; + /* Stores information about total number of Code Blocks + * in currently processed Transport Block + */ + uint64_t cbs_in_op; + }; + + uint8_t sw_ctxt[FPGA_RING_DESC_LEN_UNIT_BYTES * + (FPGA_RING_DESC_ENTRY_LENGTH - 1)]; + }; +}; + +/* FPGA LTE FEC DMA Decoding Request Descriptor */ +struct __attribute__((__packed__)) fpga_dma_dec_desc { + uint32_t done:1, + iter:5, + rsrvd0:2, + crc_pass:1, + rsrvd1:3, + error:4, + crc_type:1, + rsrvd2:7, + max_iter:5, + rsrvd3:3; + uint32_t rsrvd4; + uint32_t bypass_rm:1, + irq_en:1, + drop_crc:1, + rsrvd5:13, + offset:10, + rsrvd6:6; + uint16_t k; + uint16_t in_len; + uint32_t out_addr_lw; + uint32_t out_addr_hi; + uint32_t in_addr_lw; + uint32_t in_addr_hi; + + union { + struct { + /* Virtual addresses used to retrieve SW context info */ + void *op_addr; + /* Stores information about total number of Code Blocks + * in currently processed Transport Block + */ + uint8_t cbs_in_op; + }; + + uint32_t sw_ctxt[8 * (FPGA_RING_DESC_ENTRY_LENGTH - 1)]; + }; +}; + +/* FPGA LTE DMA Descriptor */ +union fpga_dma_desc { + struct fpga_dma_enc_desc enc_req; + struct fpga_dma_dec_desc dec_req; +}; + +/* FPGA LTE FEC Ring Control Register */ +struct __attribute__((__packed__)) fpga_ring_ctrl_reg { + uint64_t ring_base_addr; + uint64_t ring_head_addr; + uint16_t ring_size:11; + uint16_t rsrvd0; + union { /* Miscellaneous register */ + uint8_t misc; + uint8_t max_ul_dec:5, + max_ul_dec_en:1, + rsrvd1:2; + }; + uint8_t enable; + uint8_t flush_queue_en; + uint8_t rsrvd2; + uint16_t shadow_tail; + uint16_t rsrvd3; + uint16_t head_point; + uint16_t rsrvd4; + +}; + +/* Private data structure for each FPGA FEC device */ +struct fpga_lte_fec_device { + /** Base address of MMIO registers (BAR0) */ + void *mmio_base; + /** Base address of memory for sw rings */ + void *sw_rings; + /** Physical address of sw_rings */ + rte_iova_t sw_rings_phys; + /** Number of bytes available for each queue in device. */ + uint32_t sw_ring_size; + /** Max number of entries available for each queue in device */ + uint32_t sw_ring_max_depth; + /** Base address of response tail pointer buffer */ + uint32_t *tail_ptrs; + /** Physical address of tail pointers */ + rte_iova_t tail_ptr_phys; + /** Queues flush completion flag */ + uint64_t *flush_queue_status; + /* Bitmap capturing which Queues are bound to the PF/VF */ + uint64_t q_bound_bit_map; + /* Bitmap capturing which Queues have already been assigned */ + uint64_t q_assigned_bit_map; + /** True if this is a PF FPGA FEC device */ + bool pf_device; +}; + +/* Structure associated with each queue. */ +struct __rte_cache_aligned fpga_queue { + struct fpga_ring_ctrl_reg ring_ctrl_reg; /* Ring Control Register */ + union fpga_dma_desc *ring_addr; /* Virtual address of software ring */ + uint64_t *ring_head_addr; /* Virtual address of completion_head */ + uint64_t shadow_completion_head; /* Shadow completion head value */ + uint16_t head_free_desc; /* Ring head */ + uint16_t tail; /* Ring tail */ + /* Mask used to wrap enqueued descriptors on the sw ring */ + uint32_t sw_ring_wrap_mask; + uint32_t irq_enable; /* Enable ops dequeue interrupts if set to 1 */ + uint8_t q_idx; /* Queue index */ + struct fpga_lte_fec_device *d; + /* MMIO register of shadow_tail used to enqueue descriptors */ + void *shadow_tail_addr; +}; + +/* Write to 16 bit MMIO register address */ +static inline void +mmio_write_16(void *addr, uint16_t value) +{ + *((volatile uint16_t *)(addr)) = rte_cpu_to_le_16(value); +} + +/* Write to 32 bit MMIO register address */ +static inline void +mmio_write_32(void *addr, uint32_t value) +{ + *((volatile uint32_t *)(addr)) = rte_cpu_to_le_32(value); +} + +/* Write to 64 bit MMIO register address */ +static inline void +mmio_write_64(void *addr, uint64_t value) +{ + *((volatile uint64_t *)(addr)) = rte_cpu_to_le_64(value); +} + +/* Write a 8 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_8(void *mmio_base, uint32_t offset, uint8_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + *((volatile uint8_t *)(reg_addr)) = payload; +} + +/* Write a 16 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_16(void *mmio_base, uint32_t offset, uint16_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_16(reg_addr, payload); +} + +/* Write a 32 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_32(void *mmio_base, uint32_t offset, uint32_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_32(reg_addr, payload); +} + +/* Write a 64 bit register of a FPGA LTE FEC device */ +static inline void +fpga_reg_write_64(void *mmio_base, uint32_t offset, uint64_t payload) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + mmio_write_64(reg_addr, payload); +} + +/* Write a ring control register of a FPGA LTE FEC device */ +static inline void +fpga_ring_reg_write(void *mmio_base, uint32_t offset, + struct fpga_ring_ctrl_reg payload) +{ + fpga_reg_write_64(mmio_base, offset, payload.ring_base_addr); + fpga_reg_write_64(mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_ADDR, + payload.ring_head_addr); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_SIZE, + payload.ring_size); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_POINT, + payload.head_point); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN, + payload.flush_queue_en); + fpga_reg_write_16(mmio_base, offset + FPGA_LTE_FEC_RING_SHADOW_TAIL, + payload.shadow_tail); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_MISC, + payload.misc); + fpga_reg_write_8(mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + payload.enable); +} + +/* Read a register of FPGA LTE FEC device */ +static uint32_t +fpga_reg_read_32(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint32_t ret = *((volatile uint32_t *)(reg_addr)); + return rte_le_to_cpu_32(ret); +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Read a register of FPGA LTE FEC device */ +static uint8_t +fpga_reg_read_8(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + return *((volatile uint8_t *)(reg_addr)); +} + +/* Read a register of FPGA LTE FEC device */ +static uint16_t +fpga_reg_read_16(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint16_t ret = *((volatile uint16_t *)(reg_addr)); + return rte_le_to_cpu_16(ret); +} + +/* Read a register of FPGA LTE FEC device */ +static uint64_t +fpga_reg_read_64(void *mmio_base, uint32_t offset) +{ + void *reg_addr = RTE_PTR_ADD(mmio_base, offset); + uint64_t ret = *((volatile uint64_t *)(reg_addr)); + return rte_le_to_cpu_64(ret); +} + +/* Read Ring Control Register of FPGA LTE FEC device */ +static inline void +print_ring_reg_debug_info(void *mmio_base, uint32_t offset) +{ + rte_bbdev_log_debug( + "FPGA MMIO base address @ %p | Ring Control Register @ offset = 0x%08" + PRIx32, mmio_base, offset); + rte_bbdev_log_debug( + "RING_BASE_ADDR = 0x%016"PRIx64, + fpga_reg_read_64(mmio_base, offset)); + rte_bbdev_log_debug( + "RING_HEAD_ADDR = 0x%016"PRIx64, + fpga_reg_read_64(mmio_base, offset + + FPGA_LTE_FEC_RING_HEAD_ADDR)); + rte_bbdev_log_debug( + "RING_SIZE = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_SIZE)); + rte_bbdev_log_debug( + "RING_MISC = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_MISC)); + rte_bbdev_log_debug( + "RING_ENABLE = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_ENABLE)); + rte_bbdev_log_debug( + "RING_FLUSH_QUEUE_EN = 0x%02"PRIx8, + fpga_reg_read_8(mmio_base, offset + + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN)); + rte_bbdev_log_debug( + "RING_SHADOW_TAIL = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_SHADOW_TAIL)); + rte_bbdev_log_debug( + "RING_HEAD_POINT = 0x%04"PRIx16, + fpga_reg_read_16(mmio_base, offset + + FPGA_LTE_FEC_RING_HEAD_POINT)); +} + +/* Read Static Register of FPGA LTE FEC device */ +static inline void +print_static_reg_debug_info(void *mmio_base) +{ + uint16_t config = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_CONFIGURATION); + uint8_t qmap_done = fpga_reg_read_8(mmio_base, + FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE); + uint16_t lb_factor = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_LOAD_BALANCE_FACTOR); + uint16_t ring_desc_len = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_RING_DESC_LEN); + uint16_t flr_time_out = fpga_reg_read_16(mmio_base, + FPGA_LTE_FEC_FLR_TIME_OUT); + + rte_bbdev_log_debug("UL.DL Weights = %u.%u", + ((uint8_t)config), ((uint8_t)(config >> 8))); + rte_bbdev_log_debug("UL.DL Load Balance = %u.%u", + ((uint8_t)lb_factor), ((uint8_t)(lb_factor >> 8))); + rte_bbdev_log_debug("Queue-PF/VF Mapping Table = %s", + (qmap_done > 0) ? "READY" : "NOT-READY"); + rte_bbdev_log_debug("Ring Descriptor Size = %u bytes", + ring_desc_len*FPGA_RING_DESC_LEN_UNIT_BYTES); + rte_bbdev_log_debug("FLR Timeout = %f usec", + (float)flr_time_out*FPGA_FLR_TIMEOUT_UNIT); +} + +/* Print decode DMA Descriptor of FPGA LTE FEC device */ +static void +print_dma_dec_desc_debug_info(union fpga_dma_desc *desc) +{ + rte_bbdev_log_debug("DMA response desc %p\n" + "\t-- done(%"PRIu32") | iter(%"PRIu32") | crc_pass(%"PRIu32")" + " | error (%"PRIu32") | crc_type(%"PRIu32")\n" + "\t-- max_iter(%"PRIu32") | bypass_rm(%"PRIu32") | " + "irq_en (%"PRIu32") | drop_crc(%"PRIu32") | offset(%"PRIu32")\n" + "\t-- k(%"PRIu32") | in_len (%"PRIu16") | op_add(%p)\n" + "\t-- cbs_in_op(%"PRIu32") | in_add (0x%08"PRIx32"%08"PRIx32") | " + "out_add (0x%08"PRIx32"%08"PRIx32")", + desc, + (uint32_t)desc->dec_req.done, + (uint32_t)desc->dec_req.iter, + (uint32_t)desc->dec_req.crc_pass, + (uint32_t)desc->dec_req.error, + (uint32_t)desc->dec_req.crc_type, + (uint32_t)desc->dec_req.max_iter, + (uint32_t)desc->dec_req.bypass_rm, + (uint32_t)desc->dec_req.irq_en, + (uint32_t)desc->dec_req.drop_crc, + (uint32_t)desc->dec_req.offset, + (uint32_t)desc->dec_req.k, + (uint16_t)desc->dec_req.in_len, + desc->dec_req.op_addr, + (uint32_t)desc->dec_req.cbs_in_op, + (uint32_t)desc->dec_req.in_addr_hi, + (uint32_t)desc->dec_req.in_addr_lw, + (uint32_t)desc->dec_req.out_addr_hi, + (uint32_t)desc->dec_req.out_addr_lw); +} +#endif + +static int +fpga_setup_queues(struct rte_bbdev *dev, uint16_t num_queues, int socket_id) +{ + /* Number of queues bound to a PF/VF */ + uint32_t hw_q_num = 0; + uint32_t ring_size, payload, address, q_id, offset; + rte_iova_t phys_addr; + struct fpga_ring_ctrl_reg ring_reg; + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + + address = FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE; + if (!(fpga_reg_read_32(fpga_dev->mmio_base, address) & 0x1)) { + rte_bbdev_log(ERR, + "Queue-PF/VF mapping is not set! Was PF configured for device (%s) ?", + dev->data->name); + return -EPERM; + } + + /* Clear queue registers structure */ + memset(&ring_reg, 0, sizeof(struct fpga_ring_ctrl_reg)); + + /* Scan queue map. + * If a queue is valid and mapped to a calling PF/VF the read value is + * replaced with a queue ID and if it's not then + * FPGA_INVALID_HW_QUEUE_ID is returned. + */ + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + uint32_t hw_q_id = fpga_reg_read_32(fpga_dev->mmio_base, + FPGA_LTE_FEC_QUEUE_MAP + (q_id << 2)); + + rte_bbdev_log_debug("%s: queue ID: %u, registry queue ID: %u", + dev->device->name, q_id, hw_q_id); + + if (hw_q_id != FPGA_INVALID_HW_QUEUE_ID) { + fpga_dev->q_bound_bit_map |= (1ULL << q_id); + /* Clear queue register of found queue */ + offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q_id); + fpga_ring_reg_write(fpga_dev->mmio_base, + offset, ring_reg); + ++hw_q_num; + } + } + if (hw_q_num == 0) { + rte_bbdev_log(ERR, + "No HW queues assigned to this device. Probably this is a VF configured for PF mode. Check device configuration!"); + return -ENODEV; + } + + if (num_queues > hw_q_num) { + rte_bbdev_log(ERR, + "Not enough queues for device %s! Requested: %u, available: %u", + dev->device->name, num_queues, hw_q_num); + return -EINVAL; + } + + ring_size = FPGA_RING_MAX_SIZE * sizeof(struct fpga_dma_dec_desc); + + /* Enforce 32 byte alignment */ + RTE_BUILD_BUG_ON((RTE_CACHE_LINE_SIZE % 32) != 0); + + /* Allocate memory for SW descriptor rings */ + fpga_dev->sw_rings = rte_zmalloc_socket(dev->device->driver->name, + num_queues * ring_size, RTE_CACHE_LINE_SIZE, + socket_id); + if (fpga_dev->sw_rings == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u sw_rings", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + + fpga_dev->sw_rings_phys = rte_malloc_virt2iova(fpga_dev->sw_rings); + fpga_dev->sw_ring_size = ring_size; + fpga_dev->sw_ring_max_depth = FPGA_RING_MAX_SIZE; + + /* Allocate memory for ring flush status */ + fpga_dev->flush_queue_status = rte_zmalloc_socket(NULL, + sizeof(uint64_t), RTE_CACHE_LINE_SIZE, socket_id); + if (fpga_dev->flush_queue_status == NULL) { + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u flush_queue_status", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + + /* Set the flush status address registers */ + phys_addr = rte_malloc_virt2iova(fpga_dev->flush_queue_status); + + address = FPGA_LTE_FEC_VFQ_FLUSH_STATUS_LW; + payload = (uint32_t)(phys_addr); + fpga_reg_write_32(fpga_dev->mmio_base, address, payload); + + address = FPGA_LTE_FEC_VFQ_FLUSH_STATUS_HI; + payload = (uint32_t)(phys_addr >> 32); + fpga_reg_write_32(fpga_dev->mmio_base, address, payload); + + return 0; +} + +static int +fpga_dev_close(struct rte_bbdev *dev) +{ + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + + rte_free(fpga_dev->sw_rings); + rte_free(fpga_dev->flush_queue_status); + + return 0; +} + +static void +fpga_dev_info_get(struct rte_bbdev *dev, + struct rte_bbdev_driver_info *dev_info) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + uint32_t q_id = 0; + + /* TODO RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN and numbers of buffers are set + * to temporary values as they are required by test application while + * validation phase. + */ + static const struct rte_bbdev_op_cap bbdev_capabilities[] = { + { + .type = RTE_BBDEV_OP_TURBO_DEC, + .cap.turbo_dec = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_TYPE_24B | + RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE | + RTE_BBDEV_TURBO_DEC_INTERRUPTS | + RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN | + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP, + .max_llr_modulus = INT8_MAX, + .num_buffers_src = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_hard_out = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_soft_out = 0 + } + }, + { + .type = RTE_BBDEV_OP_TURBO_ENC, + .cap.turbo_enc = { + .capability_flags = + RTE_BBDEV_TURBO_CRC_24B_ATTACH | + RTE_BBDEV_TURBO_RATE_MATCH | + RTE_BBDEV_TURBO_ENC_INTERRUPTS, + .num_buffers_src = + RTE_BBDEV_MAX_CODE_BLOCKS, + .num_buffers_dst = + RTE_BBDEV_MAX_CODE_BLOCKS + } + }, + RTE_BBDEV_END_OF_CAPABILITIES_LIST() + }; + + static struct rte_bbdev_queue_conf default_queue_conf; + default_queue_conf.socket = dev->data->socket_id; + default_queue_conf.queue_size = FPGA_RING_MAX_SIZE; + + + dev_info->driver_name = dev->device->driver->name; + dev_info->queue_size_lim = FPGA_RING_MAX_SIZE; + dev_info->hardware_accelerated = true; + dev_info->min_alignment = 64; + dev_info->default_queue_conf = default_queue_conf; + dev_info->capabilities = bbdev_capabilities; + dev_info->cpu_flag_reqs = NULL; + + /* Calculates number of queues assigned to device */ + dev_info->max_num_queues = 0; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + uint32_t hw_q_id = fpga_reg_read_32(d->mmio_base, + FPGA_LTE_FEC_QUEUE_MAP + (q_id << 2)); + if (hw_q_id != FPGA_INVALID_HW_QUEUE_ID) + dev_info->max_num_queues++; + } +} + +/** + * Find index of queue bound to current PF/VF which is unassigned. Return -1 + * when there is no available queue + */ +static int +fpga_find_free_queue_idx(struct rte_bbdev *dev, + const struct rte_bbdev_queue_conf *conf) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + uint64_t q_idx; + uint8_t i = 0; + uint8_t range = FPGA_TOTAL_NUM_QUEUES >> 1; + + if (conf->op_type == RTE_BBDEV_OP_TURBO_ENC) { + i = FPGA_NUM_DL_QUEUES; + range = FPGA_TOTAL_NUM_QUEUES; + } + + for (; i < range; ++i) { + q_idx = 1ULL << i; + /* Check if index of queue is bound to current PF/VF */ + if (d->q_bound_bit_map & q_idx) + /* Check if found queue was not already assigned */ + if (!(d->q_assigned_bit_map & q_idx)) { + d->q_assigned_bit_map |= q_idx; + return i; + } + } + + rte_bbdev_log(INFO, "Failed to find free queue on %s", dev->data->name); + + return -1; +} + +static int +fpga_queue_setup(struct rte_bbdev *dev, uint16_t queue_id, + const struct rte_bbdev_queue_conf *conf) +{ + uint32_t address, ring_offset; + struct fpga_lte_fec_device *d = dev->data->dev_private; + struct fpga_queue *q; + int8_t q_idx; + + /* Check if there is a free queue to assign */ + q_idx = fpga_find_free_queue_idx(dev, conf); + if (q_idx == -1) + return -1; + + /* Allocate the queue data structure. */ + q = rte_zmalloc_socket(dev->device->driver->name, sizeof(*q), + RTE_CACHE_LINE_SIZE, conf->socket); + if (q == NULL) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_bbdev_log(ERR, "Failed to allocate queue memory"); + return -ENOMEM; + } + + q->d = d; + q->q_idx = q_idx; + + /* Set ring_base_addr */ + q->ring_addr = RTE_PTR_ADD(d->sw_rings, (d->sw_ring_size * queue_id)); + q->ring_ctrl_reg.ring_base_addr = d->sw_rings_phys + + (d->sw_ring_size * queue_id); + + /* Allocate memory for Completion Head variable*/ + q->ring_head_addr = rte_zmalloc_socket(dev->device->driver->name, + sizeof(uint64_t), RTE_CACHE_LINE_SIZE, conf->socket); + if (q->ring_head_addr == NULL) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_free(q); + rte_bbdev_log(ERR, + "Failed to allocate memory for %s:%u completion_head", + dev->device->driver->name, dev->data->dev_id); + return -ENOMEM; + } + /* Set ring_head_addr */ + q->ring_ctrl_reg.ring_head_addr = + rte_malloc_virt2iova(q->ring_head_addr); + + /* Clear shadow_completion_head */ + q->shadow_completion_head = 0; + + /* Set ring_size */ + if (conf->queue_size > FPGA_RING_MAX_SIZE) { + /* Mark queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q_idx)); + rte_free(q->ring_head_addr); + rte_free(q); + rte_bbdev_log(ERR, + "Size of queue is too big %d (MAX: %d ) for %s:%u", + conf->queue_size, FPGA_RING_MAX_SIZE, + dev->device->driver->name, dev->data->dev_id); + return -EINVAL; + } + q->ring_ctrl_reg.ring_size = conf->queue_size; + + /* Set Miscellaneous FPGA register*/ + /* Max iteration number for TTI mitigation - todo */ + q->ring_ctrl_reg.max_ul_dec = 0; + /* Enable max iteration number for TTI - todo */ + q->ring_ctrl_reg.max_ul_dec_en = 0; + + /* Enable the ring */ + q->ring_ctrl_reg.enable = 1; + + /* Set FPGA head_point and tail registers */ + q->ring_ctrl_reg.head_point = q->tail = 0; + + /* Set FPGA shadow_tail register */ + q->ring_ctrl_reg.shadow_tail = q->tail; + + /* Calculates the ring offset for found queue */ + ring_offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q_idx); + + /* Set FPGA Ring Control Registers */ + fpga_ring_reg_write(d->mmio_base, ring_offset, q->ring_ctrl_reg); + + /* Store MMIO register of shadow_tail */ + address = ring_offset + FPGA_LTE_FEC_RING_SHADOW_TAIL; + q->shadow_tail_addr = RTE_PTR_ADD(d->mmio_base, address); + + q->head_free_desc = q->tail; + + /* Set wrap mask */ + q->sw_ring_wrap_mask = conf->queue_size - 1; + + rte_bbdev_log_debug("Setup dev%u q%u: queue_idx=%u", + dev->data->dev_id, queue_id, q->q_idx); + + dev->data->queues[queue_id].queue_private = q; + + rte_bbdev_log_debug("BBDEV queue[%d] set up for FPGA queue[%d]", + queue_id, q_idx); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Read FPGA Ring Control Registers after configuration*/ + print_ring_reg_debug_info(d->mmio_base, ring_offset); +#endif + return 0; +} + +static int +fpga_queue_release(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + struct fpga_ring_ctrl_reg ring_reg; + uint32_t offset; + + rte_bbdev_log_debug("FPGA Queue[%d] released", queue_id); + + if (q != NULL) { + memset(&ring_reg, 0, sizeof(struct fpga_ring_ctrl_reg)); + offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + /* Disable queue */ + fpga_reg_write_8(d->mmio_base, + offset + FPGA_LTE_FEC_RING_ENABLE, 0x00); + /* Clear queue registers */ + fpga_ring_reg_write(d->mmio_base, offset, ring_reg); + + /* Mark the Queue as un-assigned */ + d->q_assigned_bit_map &= (0xFFFFFFFF - (1ULL << q->q_idx)); + rte_free(q->ring_head_addr); + rte_free(q); + dev->data->queues[queue_id].queue_private = NULL; + } + + return 0; +} + +/* Function starts a device queue. */ +static int +fpga_queue_start(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (d == NULL) { + rte_bbdev_log(ERR, "Invalid device pointer"); + return -1; + } +#endif + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + uint32_t offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + uint8_t enable = 0x01; + uint16_t zero = 0x0000; + + /* Clear queue head and tail variables */ + q->tail = q->head_free_desc = 0; + + /* Clear FPGA head_point and tail registers */ + fpga_reg_write_16(d->mmio_base, offset + FPGA_LTE_FEC_RING_HEAD_POINT, + zero); + fpga_reg_write_16(d->mmio_base, offset + FPGA_LTE_FEC_RING_SHADOW_TAIL, + zero); + + /* Enable queue */ + fpga_reg_write_8(d->mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + enable); + + rte_bbdev_log_debug("FPGA Queue[%d] started", queue_id); + return 0; +} + +/* Function stops a device queue. */ +static int +fpga_queue_stop(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_lte_fec_device *d = dev->data->dev_private; +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (d == NULL) { + rte_bbdev_log(ERR, "Invalid device pointer"); + return -1; + } +#endif + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + uint32_t offset = FPGA_LTE_FEC_RING_CTRL_REGS + + (sizeof(struct fpga_ring_ctrl_reg) * q->q_idx); + uint8_t payload = 0x01; + uint8_t counter = 0; + uint8_t timeout = FPGA_QUEUE_FLUSH_TIMEOUT_US / + FPGA_TIMEOUT_CHECK_INTERVAL; + + /* Set flush_queue_en bit to trigger queue flushing */ + fpga_reg_write_8(d->mmio_base, + offset + FPGA_LTE_FEC_RING_FLUSH_QUEUE_EN, payload); + + /** Check if queue flush is completed. + * FPGA will update the completion flag after queue flushing is + * completed. If completion flag is not updated within 1ms it is + * considered as a failure. + */ + while (!(*((uint8_t *)d->flush_queue_status + q->q_idx) & payload)) { + if (counter > timeout) { + rte_bbdev_log(ERR, "FPGA Queue Flush failed for queue %d", + queue_id); + return -1; + } + usleep(FPGA_TIMEOUT_CHECK_INTERVAL); + counter++; + } + + /* Disable queue */ + payload = 0x00; + fpga_reg_write_8(d->mmio_base, offset + FPGA_LTE_FEC_RING_ENABLE, + payload); + + rte_bbdev_log_debug("FPGA Queue[%d] stopped", queue_id); + return 0; +} + +static inline uint16_t +get_queue_id(struct rte_bbdev_data *data, uint8_t q_idx) +{ + uint16_t queue_id; + + for (queue_id = 0; queue_id < data->num_queues; ++queue_id) { + struct fpga_queue *q = data->queues[queue_id].queue_private; + if (q != NULL && q->q_idx == q_idx) + return queue_id; + } + + return -1; +} + +/* Interrupt handler triggered by FPGA dev for handling specific interrupt */ +static void +fpga_dev_interrupt_handler(void *cb_arg) +{ + struct rte_bbdev *dev = cb_arg; + struct fpga_lte_fec_device *fpga_dev = dev->data->dev_private; + struct fpga_queue *q; + uint64_t ring_head; + uint64_t q_idx; + uint16_t queue_id; + uint8_t i; + + /* Scan queue assigned to this device */ + for (i = 0; i < FPGA_TOTAL_NUM_QUEUES; ++i) { + q_idx = 1ULL << i; + if (fpga_dev->q_bound_bit_map & q_idx) { + queue_id = get_queue_id(dev->data, i); + if (queue_id == (uint16_t) -1) + continue; + + /* Check if completion head was changed */ + q = dev->data->queues[queue_id].queue_private; + ring_head = *q->ring_head_addr; + if (q->shadow_completion_head != ring_head && + q->irq_enable == 1) { + q->shadow_completion_head = ring_head; + rte_bbdev_pmd_callback_process( + dev, + RTE_BBDEV_EVENT_DEQUEUE, + &queue_id); + } + } + } +} + +static int +fpga_queue_intr_enable(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + + if (!rte_intr_cap_multiple(dev->intr_handle)) + return -ENOTSUP; + + q->irq_enable = 1; + + return 0; +} + +static int +fpga_queue_intr_disable(struct rte_bbdev *dev, uint16_t queue_id) +{ + struct fpga_queue *q = dev->data->queues[queue_id].queue_private; + q->irq_enable = 0; + + return 0; +} + +static int +fpga_intr_enable(struct rte_bbdev *dev) +{ + int ret; + uint8_t i; + + if (!rte_intr_cap_multiple(dev->intr_handle)) { + rte_bbdev_log(ERR, "Multiple intr vector is not supported by FPGA (%s)", + dev->data->name); + return -ENOTSUP; + } + + /* Create event file descriptors for each of 64 queue. Event fds will be + * mapped to FPGA IRQs in rte_intr_enable(). This is a 1:1 mapping where + * the IRQ number is a direct translation to the queue number. + * + * 63 (FPGA_NUM_INTR_VEC) event fds are created as rte_intr_enable() + * mapped the first IRQ to already created interrupt event file + * descriptor (intr_handle->fd). + */ + if (rte_intr_efd_enable(dev->intr_handle, FPGA_NUM_INTR_VEC)) { + rte_bbdev_log(ERR, "Failed to create fds for %u queues", + dev->data->num_queues); + return -1; + } + + /* TODO Each event file descriptor is overwritten by interrupt event + * file descriptor. That descriptor is added to epoll observed list. + * It ensures that callback function assigned to that descriptor will + * invoked when any FPGA queue issues interrupt. + */ + for (i = 0; i < FPGA_NUM_INTR_VEC; ++i) + dev->intr_handle->efds[i] = dev->intr_handle->fd; + + if (!dev->intr_handle->intr_vec) { + dev->intr_handle->intr_vec = rte_zmalloc("intr_vec", + dev->data->num_queues * sizeof(int), 0); + if (!dev->intr_handle->intr_vec) { + rte_bbdev_log(ERR, "Failed to allocate %u vectors", + dev->data->num_queues); + return -ENOMEM; + } + } + + ret = rte_intr_enable(dev->intr_handle); + if (ret < 0) { + rte_bbdev_log(ERR, + "Couldn't enable interrupts for device: %s", + dev->data->name); + return ret; + } + + ret = rte_intr_callback_register(dev->intr_handle, + fpga_dev_interrupt_handler, dev); + if (ret < 0) { + rte_bbdev_log(ERR, + "Couldn't register interrupt callback for device: %s", + dev->data->name); + return ret; + } + + return 0; +} + +static const struct rte_bbdev_ops fpga_ops = { + .setup_queues = fpga_setup_queues, + .intr_enable = fpga_intr_enable, + .close = fpga_dev_close, + .info_get = fpga_dev_info_get, + .queue_setup = fpga_queue_setup, + .queue_stop = fpga_queue_stop, + .queue_start = fpga_queue_start, + .queue_release = fpga_queue_release, + .queue_intr_enable = fpga_queue_intr_enable, + .queue_intr_disable = fpga_queue_intr_disable +}; + +static inline void +fpga_dma_enqueue(struct fpga_queue *q, uint16_t num_desc, + struct rte_bbdev_stats *queue_stats) +{ +#ifdef RTE_BBDEV_OFFLOAD_COST + uint64_t start_time = 0; + queue_stats->acc_offload_cycles = 0; +#else + RTE_SET_USED(queue_stats); +#endif + + /* Update tail and shadow_tail register */ + q->tail = (q->tail + num_desc) & q->sw_ring_wrap_mask; + + rte_wmb(); + +#ifdef RTE_BBDEV_OFFLOAD_COST + /* Start time measurement for enqueue function offload. */ + start_time = rte_rdtsc_precise(); +#endif + mmio_write_16(q->shadow_tail_addr, q->tail); + +#ifdef RTE_BBDEV_OFFLOAD_COST + rte_wmb(); + queue_stats->acc_offload_cycles += rte_rdtsc_precise() - start_time; +#endif +} + +/* Calculates number of CBs in processed encoder TB based on 'r' and input + * length. + */ +static inline uint8_t +get_num_cbs_in_op_enc(struct rte_bbdev_op_turbo_enc *turbo_enc) +{ + uint8_t c, c_neg, r, crc24_bits = 0; + uint16_t k, k_neg, k_pos; + uint8_t cbs_in_op = 0; + int32_t length; + + length = turbo_enc->input.length; + r = turbo_enc->tb_params.r; + c = turbo_enc->tb_params.c; + c_neg = turbo_enc->tb_params.c_neg; + k_neg = turbo_enc->tb_params.k_neg; + k_pos = turbo_enc->tb_params.k_pos; + crc24_bits = 24; + while (length > 0 && r < c) { + k = (r < c_neg) ? k_neg : k_pos; + length -= (k - crc24_bits) >> 3; + r++; + cbs_in_op++; + } + + return cbs_in_op; +} + +/* Calculates number of CBs in processed decoder TB based on 'r' and input + * length. + */ +static inline uint16_t +get_num_cbs_in_op_dec(struct rte_bbdev_op_turbo_dec *turbo_dec) +{ + uint8_t c, c_neg, r = 0; + uint16_t kw, k, k_neg, k_pos, cbs_in_op = 0; + int32_t length; + + length = turbo_dec->input.length; + r = turbo_dec->tb_params.r; + c = turbo_dec->tb_params.c; + c_neg = turbo_dec->tb_params.c_neg; + k_neg = turbo_dec->tb_params.k_neg; + k_pos = turbo_dec->tb_params.k_pos; + while (length > 0 && r < c) { + k = (r < c_neg) ? k_neg : k_pos; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + length -= kw; + r++; + cbs_in_op++; + } + + return cbs_in_op; +} + +/* Read flag value 0/1/ from bitmap */ +static inline bool +check_bit(uint32_t bitmap, uint32_t bitmask) +{ + return bitmap & bitmask; +} + +/* Print an error if a descriptor error has occurred. + * Return 0 on success, 1 on failure + */ +static inline int +check_desc_error(uint32_t error_code) { + switch (error_code) { + case DESC_ERR_NO_ERR: + return 0; + case DESC_ERR_K_OUT_OF_RANGE: + rte_bbdev_log(ERR, "Block_size_k is out of range (k<40 or k>6144)"); + break; + case DESC_ERR_K_NOT_NORMAL: + rte_bbdev_log(ERR, "Block_size_k is not a normal value within normal range"); + break; + case DESC_ERR_KPAI_NOT_NORMAL: + rte_bbdev_log(ERR, "Three_kpai is not a normal value for UL only"); + break; + case DESC_ERR_DESC_OFFSET_ERR: + rte_bbdev_log(ERR, "Queue offset does not meet the expectation in the FPGA"); + break; + case (DESC_ERR_K_OUT_OF_RANGE | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Block_size_k is out of range (k<40 or k>6144) and queue offset error"); + break; + case (DESC_ERR_K_NOT_NORMAL | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Block_size_k is not a normal value within normal range and queue offset error"); + break; + case (DESC_ERR_KPAI_NOT_NORMAL | DESC_ERR_DESC_OFFSET_ERR): + rte_bbdev_log(ERR, "Three_kpai is not a normal value for UL only and queue offset error"); + break; + case DESC_ERR_DESC_READ_FAIL: + rte_bbdev_log(ERR, "Unsuccessful completion for descriptor read"); + break; + case DESC_ERR_DESC_READ_TIMEOUT: + rte_bbdev_log(ERR, "Descriptor read time-out"); + break; + case DESC_ERR_DESC_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Descriptor read TLP poisoned"); + break; + case DESC_ERR_CB_READ_FAIL: + rte_bbdev_log(ERR, "Unsuccessful completion for code block"); + break; + case DESC_ERR_CB_READ_TIMEOUT: + rte_bbdev_log(ERR, "Code block read time-out"); + break; + case DESC_ERR_CB_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Code block read TLP poisoned"); + break; + default: + rte_bbdev_log(ERR, "Descriptor error unknown error code %u", + error_code); + break; + } + return 1; +} + +/** + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param k + * K value (length of input in bits). + * @param e + * E value (length of output in bits). + * @param ncb + * Ncb value (size of the soft buffer). + * @param out_length + * Length of output buffer + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +fpga_dma_desc_te_fill(struct rte_bbdev_enc_op *op, + struct fpga_dma_enc_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t k, uint16_t e, uint16_t ncb, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) + +{ + /* reset */ + desc->done = 0; + desc->crc_en = check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_CRC_24B_ATTACH); + desc->bypass_rm = !check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_RATE_MATCH); + desc->k = k; + desc->e = e; + desc->ncb = ncb; + desc->rv = op->turbo_enc.rv_index; + desc->offset = desc_offset; + /* Set inbound data buffer address */ + desc->in_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset) >> 32); + desc->in_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset)); + + desc->out_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset) >> 32); + desc->out_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset)); + + /* Save software context needed for dequeue */ + desc->op_addr = op; + + /* Set total number of CBs in an op */ + desc->cbs_in_op = cbs_in_op; + + return 0; +} + +/** + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param in_length + * Length of an input. + * @param k + * K value (length of an output in bits). + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +fpga_dma_desc_td_fill(struct rte_bbdev_dec_op *op, + struct fpga_dma_dec_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t in_length, uint16_t k, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) +{ + /* reset */ + desc->done = 0; + /* Set inbound data buffer address */ + desc->in_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset) >> 32); + desc->in_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(input, in_offset)); + desc->in_len = in_length; + desc->k = k; + desc->crc_type = !check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_CRC_TYPE_24B); + if ((op->turbo_dec.code_block_mode == 0) + && !check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP)) + desc->drop_crc = 1; + desc->max_iter = op->turbo_dec.iter_max * 2; + desc->offset = desc_offset; + desc->out_addr_hi = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset) >> 32); + desc->out_addr_lw = (uint32_t)( + rte_pktmbuf_mtophys_offset(output, out_offset)); + + /* Save software context needed for dequeue */ + desc->op_addr = op; + + /* Set total number of CBs in an op */ + desc->cbs_in_op = cbs_in_op; + + return 0; +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Validates turbo encoder parameters */ +static int +validate_enc_op(struct rte_bbdev_enc_op *op) +{ + struct rte_bbdev_op_turbo_enc *turbo_enc = &op->turbo_enc; + struct rte_bbdev_op_enc_cb_params *cb = NULL; + struct rte_bbdev_op_enc_tb_params *tb = NULL; + uint16_t kw, kw_neg, kw_pos; + + if (turbo_enc->input.length > + RTE_BBDEV_MAX_TB_SIZE >> 3) { + rte_bbdev_log(ERR, "TB size (%u) is too big, max: %d", + turbo_enc->input.length, RTE_BBDEV_MAX_TB_SIZE); + op->status = 1 << RTE_BBDEV_DATA_ERROR; + return -1; + } + + if (op->mempool == NULL) { + rte_bbdev_log(ERR, "Invalid mempool pointer"); + return -1; + } + if (turbo_enc->input.data == NULL) { + rte_bbdev_log(ERR, "Invalid input pointer"); + return -1; + } + if (turbo_enc->output.data == NULL) { + rte_bbdev_log(ERR, "Invalid output pointer"); + return -1; + } + if (turbo_enc->rv_index > 3) { + rte_bbdev_log(ERR, + "rv_index (%u) is out of range 0 <= value <= 3", + turbo_enc->rv_index); + return -1; + } + if (turbo_enc->code_block_mode != 0 && + turbo_enc->code_block_mode != 1) { + rte_bbdev_log(ERR, + "code_block_mode (%u) is out of range 0 <= value <= 1", + turbo_enc->code_block_mode); + return -1; + } + + if (turbo_enc->code_block_mode == 0) { + tb = &turbo_enc->tb_params; + if ((tb->k_neg < RTE_BBDEV_MIN_CB_SIZE + || tb->k_neg > RTE_BBDEV_MAX_CB_SIZE) + && tb->c_neg > 0) { + rte_bbdev_log(ERR, + "k_neg (%u) is out of range %u <= value <= %u", + tb->k_neg, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->k_pos < RTE_BBDEV_MIN_CB_SIZE + || tb->k_pos > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k_pos (%u) is out of range %u <= value <= %u", + tb->k_pos, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->c_neg > (RTE_BBDEV_MAX_CODE_BLOCKS - 1)) + rte_bbdev_log(ERR, + "c_neg (%u) is out of range 0 <= value <= %u", + tb->c_neg, + RTE_BBDEV_MAX_CODE_BLOCKS - 1); + if (tb->c < 1 || tb->c > RTE_BBDEV_MAX_CODE_BLOCKS) { + rte_bbdev_log(ERR, + "c (%u) is out of range 1 <= value <= %u", + tb->c, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + if (tb->cab > tb->c) { + rte_bbdev_log(ERR, + "cab (%u) is greater than c (%u)", + tb->cab, tb->c); + return -1; + } + if ((tb->ea < RTE_BBDEV_MIN_CB_SIZE || (tb->ea % 2)) + && tb->r < tb->cab) { + rte_bbdev_log(ERR, + "ea (%u) is less than %u or it is not even", + tb->ea, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + if ((tb->eb < RTE_BBDEV_MIN_CB_SIZE || (tb->eb % 2)) + && tb->c > tb->cab) { + rte_bbdev_log(ERR, + "eb (%u) is less than %u or it is not even", + tb->eb, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + + kw_neg = 3 * RTE_ALIGN_CEIL(tb->k_neg + 4, + RTE_BBDEV_C_SUBBLOCK); + if (tb->ncb_neg < tb->k_neg || tb->ncb_neg > kw_neg) { + rte_bbdev_log(ERR, + "ncb_neg (%u) is out of range (%u) k_neg <= value <= (%u) kw_neg", + tb->ncb_neg, tb->k_neg, kw_neg); + return -1; + } + + kw_pos = 3 * RTE_ALIGN_CEIL(tb->k_pos + 4, + RTE_BBDEV_C_SUBBLOCK); + if (tb->ncb_pos < tb->k_pos || tb->ncb_pos > kw_pos) { + rte_bbdev_log(ERR, + "ncb_pos (%u) is out of range (%u) k_pos <= value <= (%u) kw_pos", + tb->ncb_pos, tb->k_pos, kw_pos); + return -1; + } + if (tb->r > (tb->c - 1)) { + rte_bbdev_log(ERR, + "r (%u) is greater than c - 1 (%u)", + tb->r, tb->c - 1); + return -1; + } + } else { + cb = &turbo_enc->cb_params; + if (cb->k < RTE_BBDEV_MIN_CB_SIZE + || cb->k > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k (%u) is out of range %u <= value <= %u", + cb->k, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + + if (cb->e < RTE_BBDEV_MIN_CB_SIZE || (cb->e % 2)) { + rte_bbdev_log(ERR, + "e (%u) is less than %u or it is not even", + cb->e, RTE_BBDEV_MIN_CB_SIZE); + return -1; + } + + kw = RTE_ALIGN_CEIL(cb->k + 4, RTE_BBDEV_C_SUBBLOCK) * 3; + if (cb->ncb < cb->k || cb->ncb > kw) { + rte_bbdev_log(ERR, + "ncb (%u) is out of range (%u) k <= value <= (%u) kw", + cb->ncb, cb->k, kw); + return -1; + } + } + + return 0; +} +#endif + +static inline char * +mbuf_append(struct rte_mbuf *m_head, struct rte_mbuf *m, uint16_t len) +{ + if (unlikely(len > rte_pktmbuf_tailroom(m))) + return NULL; + + char *tail = (char *)m->buf_addr + m->data_off + m->data_len; + m->data_len = (uint16_t)(m->data_len + len); + m_head->pkt_len = (m_head->pkt_len + len); + return tail; +} + +static inline int +enqueue_enc_one_op_cb(struct fpga_queue *q, struct rte_bbdev_enc_op *op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input; + struct rte_mbuf *output; + int ret; + uint16_t k, e, ncb, ring_offset; + uint32_t total_left, in_length, out_length, in_offset, out_offset; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_enc_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo encoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_enc.input.data; + output = op->turbo_enc.output.data; + in_offset = op->turbo_enc.input.offset; + out_offset = op->turbo_enc.output.offset; + total_left = op->turbo_enc.input.length; + k = op->turbo_enc.cb_params.k; + e = op->turbo_enc.cb_params.e; + ncb = op->turbo_enc.cb_params.ncb; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_CRC_24B_ATTACH)) + in_length = ((k - 24) >> 3); + else + in_length = k >> 3; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_RATE_MATCH)) + out_length = (e + 7) >> 3; + else + out_length = (k >> 3) * 3 + 2; + + mbuf_append(output, output, out_length); + + /* Offset into the ring */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + /* Setup DMA Descriptor */ + desc = q->ring_addr + ring_offset; + + ret = fpga_dma_desc_te_fill(op, &desc->enc_req, input, output, k, e, + ncb, in_offset, out_offset, ring_offset, 1); + if (unlikely(ret < 0)) + return ret; + + /* Update lengths */ + total_left -= in_length; + op->turbo_enc.output.length += out_length; + + if (total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + total_left, in_length); + return -1; + } + + return 1; +} + +static inline int +enqueue_enc_one_op_tb(struct fpga_queue *q, struct rte_bbdev_enc_op *op, + uint16_t desc_offset, uint8_t cbs_in_op) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input, *output_head, *output; + int ret; + uint8_t r, c, crc24_bits = 0; + uint16_t k, e, ncb, ring_offset; + uint32_t mbuf_total_left, in_length, out_length, in_offset, out_offset; + uint32_t seg_total_left; + uint16_t current_enqueued_cbs = 0; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_enc_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo encoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_enc.input.data; + output_head = output = op->turbo_enc.output.data; + in_offset = op->turbo_enc.input.offset; + out_offset = op->turbo_enc.output.offset; + mbuf_total_left = op->turbo_enc.input.length; + + c = op->turbo_enc.tb_params.c; + r = op->turbo_enc.tb_params.r; + + if (check_bit(op->turbo_enc.op_flags, RTE_BBDEV_TURBO_CRC_24B_ATTACH)) + crc24_bits = 24; + + while (mbuf_total_left > 0 && r < c && input != NULL) { + seg_total_left = rte_pktmbuf_data_len(input) - in_offset; + + e = (r < op->turbo_enc.tb_params.cab) ? + op->turbo_enc.tb_params.ea : + op->turbo_enc.tb_params.eb; + k = (r < op->turbo_enc.tb_params.c_neg) ? + op->turbo_enc.tb_params.k_neg : + op->turbo_enc.tb_params.k_pos; + ncb = (r < op->turbo_enc.tb_params.c_neg) ? + op->turbo_enc.tb_params.ncb_neg : + op->turbo_enc.tb_params.ncb_pos; + + in_length = ((k - crc24_bits) >> 3); + + if (check_bit(op->turbo_enc.op_flags, + RTE_BBDEV_TURBO_RATE_MATCH)) + out_length = (e + 7) >> 3; + else + out_length = (k >> 3) * 3 + 2; + + mbuf_append(output_head, output, out_length); + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_te_fill(op, &desc->enc_req, input, output, + k, e, ncb, in_offset, out_offset, ring_offset, + cbs_in_op); + if (unlikely(ret < 0)) + return ret; + + rte_bbdev_log_debug("DMA request desc %p", desc); + + /* Update lengths */ + op->turbo_enc.output.length += out_length; + mbuf_total_left -= in_length; + + /* Update offsets */ + if (seg_total_left == in_length) { + /* Go to the next mbuf */ + input = input->next; + output = output->next; + in_offset = 0; + out_offset = 0; + } else { + in_offset += in_length; + out_offset += out_length; + } + + r++; + desc_offset++; + current_enqueued_cbs++; + } + + if (mbuf_total_left > 0) { + rte_bbdev_log(ERR, + "Some date still left for processing: mbuf_total_left = %u", + mbuf_total_left); + return -1; + } + + return current_enqueued_cbs; +} + +#ifdef RTE_LIBRTE_BBDEV_DEBUG +/* Validates turbo decoder parameters */ +static int +validate_dec_op(struct rte_bbdev_dec_op *op) +{ + struct rte_bbdev_op_turbo_dec *turbo_dec = &op->turbo_dec; + struct rte_bbdev_op_dec_cb_params *cb = NULL; + struct rte_bbdev_op_dec_tb_params *tb = NULL; + + if (op->mempool == NULL) { + rte_bbdev_log(ERR, "Invalid mempool pointer"); + return -1; + } + if (turbo_dec->input.data == NULL) { + rte_bbdev_log(ERR, "Invalid input pointer"); + return -1; + } + if (turbo_dec->hard_output.data == NULL) { + rte_bbdev_log(ERR, "Invalid hard_output pointer"); + return -1; + } + if (turbo_dec->rv_index > 3) { + rte_bbdev_log(ERR, + "rv_index (%u) is out of range 0 <= value <= 3", + turbo_dec->rv_index); + return -1; + } + if (turbo_dec->iter_min < 1) { + rte_bbdev_log(ERR, + "iter_min (%u) is less than 1", + turbo_dec->iter_min); + return -1; + } + if (turbo_dec->iter_max <= 2) { + rte_bbdev_log(ERR, + "iter_max (%u) is less than or equal to 2", + turbo_dec->iter_max); + return -1; + } + if (turbo_dec->iter_min > turbo_dec->iter_max) { + rte_bbdev_log(ERR, + "iter_min (%u) is greater than iter_max (%u)", + turbo_dec->iter_min, turbo_dec->iter_max); + return -1; + } + if (turbo_dec->code_block_mode != 0 && + turbo_dec->code_block_mode != 1) { + rte_bbdev_log(ERR, + "code_block_mode (%u) is out of range 0 <= value <= 1", + turbo_dec->code_block_mode); + return -1; + } + + if (turbo_dec->code_block_mode == 0) { + + if ((turbo_dec->op_flags & + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP) && + !(turbo_dec->op_flags & RTE_BBDEV_TURBO_CRC_TYPE_24B)) { + rte_bbdev_log(ERR, + "RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP should accompany RTE_BBDEV_TURBO_CRC_TYPE_24B"); + return -1; + } + + tb = &turbo_dec->tb_params; + if ((tb->k_neg < RTE_BBDEV_MIN_CB_SIZE + || tb->k_neg > RTE_BBDEV_MAX_CB_SIZE) + && tb->c_neg > 0) { + rte_bbdev_log(ERR, + "k_neg (%u) is out of range %u <= value <= %u", + tb->k_neg, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if ((tb->k_pos < RTE_BBDEV_MIN_CB_SIZE + || tb->k_pos > RTE_BBDEV_MAX_CB_SIZE) + && tb->c > tb->c_neg) { + rte_bbdev_log(ERR, + "k_pos (%u) is out of range %u <= value <= %u", + tb->k_pos, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + if (tb->c_neg > (RTE_BBDEV_MAX_CODE_BLOCKS - 1)) + rte_bbdev_log(ERR, + "c_neg (%u) is out of range 0 <= value <= %u", + tb->c_neg, + RTE_BBDEV_MAX_CODE_BLOCKS - 1); + if (tb->c < 1 || tb->c > RTE_BBDEV_MAX_CODE_BLOCKS) { + rte_bbdev_log(ERR, + "c (%u) is out of range 1 <= value <= %u", + tb->c, RTE_BBDEV_MAX_CODE_BLOCKS); + return -1; + } + if (tb->cab > tb->c) { + rte_bbdev_log(ERR, + "cab (%u) is greater than c (%u)", + tb->cab, tb->c); + return -1; + } + } else { + + if (turbo_dec->op_flags & RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP) { + rte_bbdev_log(ERR, + "RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP is invalid in CB-mode"); + return -1; + } + + cb = &turbo_dec->cb_params; + if (cb->k < RTE_BBDEV_MIN_CB_SIZE + || cb->k > RTE_BBDEV_MAX_CB_SIZE) { + rte_bbdev_log(ERR, + "k (%u) is out of range %u <= value <= %u", + cb->k, RTE_BBDEV_MIN_CB_SIZE, + RTE_BBDEV_MAX_CB_SIZE); + return -1; + } + } + + return 0; +} +#endif + +static inline int +enqueue_dec_one_op_cb(struct fpga_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input; + struct rte_mbuf *output; + int ret; + uint16_t k, kw, ring_offset; + uint32_t total_left, in_length, out_length, in_offset, out_offset; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_dec_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo decoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_dec.input.data; + output = op->turbo_dec.hard_output.data; + total_left = op->turbo_dec.input.length; + in_offset = op->turbo_dec.input.offset; + out_offset = op->turbo_dec.hard_output.offset; + + k = op->turbo_dec.cb_params.k; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + in_length = kw; + out_length = k >> 3; + + mbuf_append(output, output, out_length); + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_td_fill(op, &desc->dec_req, input, output, + in_length, k, in_offset, out_offset, ring_offset, 1); + if (unlikely(ret < 0)) + return ret; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_dma_dec_desc_debug_info(desc); +#endif + + /* Update lengths */ + total_left -= in_length; + op->turbo_dec.hard_output.length += out_length; + + if (total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + total_left, in_length); + return -1; + } + + return 1; +} + + +static inline int +enqueue_dec_one_op_tb(struct fpga_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset, uint8_t cbs_in_op) +{ + union fpga_dma_desc *desc; + struct rte_mbuf *input, *output_head, *output; + int ret; + uint8_t r, c; + uint16_t k, kw, in_length, out_length, ring_offset; + uint32_t mbuf_total_left, seg_total_left, in_offset, out_offset; + uint16_t current_enqueued_cbs = 0; + uint16_t crc24_overlap = 0; + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + /* Validate op structure */ + if (validate_dec_op(op) == -1) { + rte_bbdev_log(ERR, "Turbo decoder validation failed"); + return -EINVAL; + } +#endif + + input = op->turbo_dec.input.data; + output_head = output = op->turbo_dec.hard_output.data; + mbuf_total_left = op->turbo_dec.input.length; + in_offset = op->turbo_dec.input.offset; + out_offset = op->turbo_dec.hard_output.offset; + + if (!check_bit(op->turbo_dec.op_flags, + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP)) + crc24_overlap = 24; + + c = op->turbo_dec.tb_params.c; + r = op->turbo_dec.tb_params.r; + + while (mbuf_total_left > 0 && r < c && input != NULL) { + seg_total_left = rte_pktmbuf_data_len(input) - in_offset; + k = (r < op->turbo_dec.tb_params.c_neg) ? + op->turbo_dec.tb_params.k_neg : + op->turbo_dec.tb_params.k_pos; + kw = RTE_ALIGN_CEIL(k + 4, 32) * 3; + + in_length = kw; + out_length = (k - crc24_overlap) >> 3; + + mbuf_append(output_head, output, out_length); + + if (seg_total_left < in_length) { + rte_bbdev_log(ERR, + "Partial CB found in a TB. FPGA Driver doesn't support scatter-gather operations!"); + return -1; + } + + /* Setup DMA Descriptor */ + ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); + desc = q->ring_addr + ring_offset; + ret = fpga_dma_desc_td_fill(op, &desc->dec_req, input, output, + in_length, k, in_offset, out_offset, + ring_offset, cbs_in_op); + if (unlikely(ret < 0)) + return ret; + + /* Update lengths */ + ret = rte_pktmbuf_trim(op->turbo_dec.hard_output.data, + (crc24_overlap >> 3)); +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (ret < 0) { + rte_bbdev_log(ERR, + "The length to remove is greater than the length of the last segment"); + return -EINVAL; + } +#endif + op->turbo_dec.hard_output.length += out_length; + mbuf_total_left -= in_length; + + /* Update offsets */ + if (seg_total_left == in_length) { + /* Go to the next mbuf */ + input = input->next; + output = output->next; + in_offset = 0; + out_offset = 0; + } else { + in_offset += in_length; + out_offset += out_length; + } + + r++; + desc_offset++; + current_enqueued_cbs++; + } + + if (mbuf_total_left > 0) { + rte_bbdev_log(ERR, + "Some date still left for processing: mbuf_total_left = %u", + mbuf_total_left); + return -1; + } + + return current_enqueued_cbs; +} + +static uint16_t +fpga_enqueue_enc(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num) +{ + uint8_t cbs_in_op; + uint16_t i, total_enqueued_cbs = 0; + int32_t avail; + int enqueued_cbs; + struct fpga_queue *q = q_data->queue_private; + union fpga_dma_desc *desc; + + /* Check if queue is not full */ + if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == + q->head_free_desc)) + return 0; + + /* Calculates available space */ + avail = (q->head_free_desc > q->tail) ? + q->head_free_desc - q->tail - 1 : + q->ring_ctrl_reg.ring_size + q->head_free_desc - q->tail - 1; + + for (i = 0; i < num; ++i) { + if (ops[i]->turbo_enc.code_block_mode == 0) { + cbs_in_op = get_num_cbs_in_op_enc(&ops[i]->turbo_enc); + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - cbs_in_op < 0)) + break; + avail -= cbs_in_op; + enqueued_cbs = enqueue_enc_one_op_tb(q, ops[i], + total_enqueued_cbs, cbs_in_op); + } else { + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - 1 < 0)) + break; + avail -= 1; + enqueued_cbs = enqueue_enc_one_op_cb(q, ops[i], + total_enqueued_cbs); + } + + if (enqueued_cbs < 0) + break; + + total_enqueued_cbs += enqueued_cbs; + + rte_bbdev_log_debug("enqueuing enc ops [%d/%d] | head %d | tail %d", + total_enqueued_cbs, num, + q->head_free_desc, q->tail); + } + + /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt + * only when all previous CBs were already processed. + */ + desc = q->ring_addr + ((q->tail + total_enqueued_cbs - 1) + & q->sw_ring_wrap_mask); + desc->enc_req.irq_en = q->irq_enable; + + fpga_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); + + /* Update stats */ + q_data->queue_stats.enqueued_count += i; + q_data->queue_stats.enqueue_err_count += num - i; + + return i; +} + +static uint16_t +fpga_enqueue_dec(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num) +{ + uint8_t cbs_in_op; + uint16_t i, total_enqueued_cbs = 0; + int32_t avail; + int enqueued_cbs; + struct fpga_queue *q = q_data->queue_private; + union fpga_dma_desc *desc; + + /* Check if queue is not full */ + if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == + q->head_free_desc)) + return 0; + + /* Calculates available space */ + avail = (q->head_free_desc > q->tail) ? + q->head_free_desc - q->tail - 1 : + q->ring_ctrl_reg.ring_size + q->head_free_desc - q->tail - 1; + + for (i = 0; i < num; ++i) { + if (ops[i]->turbo_dec.code_block_mode == 0) { + cbs_in_op = get_num_cbs_in_op_dec(&ops[i]->turbo_dec); + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - cbs_in_op < 0)) + break; + avail -= cbs_in_op; + enqueued_cbs = enqueue_dec_one_op_tb(q, ops[i], + total_enqueued_cbs, cbs_in_op); + } else { + /* Check if there is available space for further + * processing + */ + if (unlikely(avail - 1 < 0)) + break; + avail -= 1; + enqueued_cbs = enqueue_dec_one_op_cb(q, ops[i], + total_enqueued_cbs); + } + + if (enqueued_cbs < 0) + break; + + total_enqueued_cbs += enqueued_cbs; + + rte_bbdev_log_debug("enqueuing dec ops [%d/%d] | head %d | tail %d", + total_enqueued_cbs, num, + q->head_free_desc, q->tail); + } + + /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt + * only when all previous CBs were already processed. + */ + desc = q->ring_addr + ((q->tail + total_enqueued_cbs - 1) + & q->sw_ring_wrap_mask); + desc->dec_req.irq_en = q->irq_enable; + + fpga_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); + + /* Update stats */ + q_data->queue_stats.enqueued_count += i; + q_data->queue_stats.enqueue_err_count += num - i; + + return i; +} + +static inline int +dequeue_enc_one_op_cb(struct fpga_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + int desc_error = 0; + + /* Set current desc */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /*check if done */ + if (desc->enc_req.done == 0) + return -1; + + /* make sure the response is read atomically */ + rte_smp_rmb(); + + rte_bbdev_log_debug("DMA response desc %p", desc); + + *op = desc->enc_req.op_addr; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + (*op)->status = desc_error << RTE_BBDEV_DATA_ERROR; + + return 1; +} + +static inline int +dequeue_enc_one_op_tb(struct fpga_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + uint8_t cbs_in_op, cb_idx; + int desc_error = 0; + int status = 0; + + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify if done bit is set */ + if (desc->enc_req.done == 0) + return -1; + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + /* Verify if done bit in all CBs is set */ + cbs_in_op = desc->enc_req.cbs_in_op; + for (cb_idx = 1; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + if (desc->enc_req.done == 0) + return -1; + } + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + for (cb_idx = 0; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + status |= desc_error << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log_debug("DMA response desc %p", desc); + } + + *op = desc->enc_req.op_addr; + (*op)->status = status; + return cbs_in_op; +} + +static inline int +dequeue_dec_one_op_cb(struct fpga_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + int desc_error = 0; + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify done bit is set */ + if (desc->dec_req.done == 0) + return -1; + + /* make sure the response is read atomically */ + rte_smp_rmb(); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_dma_dec_desc_debug_info(desc); + +#endif + + *op = desc->dec_req.op_addr; + /* FPGA reports in half-iterations, from 0 to 31. get ceiling */ + (*op)->turbo_dec.iter_count = (desc->dec_req.iter + 2) >> 1; + /* crc_pass = 0 when decoder fails */ + (*op)->status = !(desc->dec_req.crc_pass) << RTE_BBDEV_CRC_ERROR; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + (*op)->status |= desc_error << RTE_BBDEV_DATA_ERROR; + return 1; +} + +static inline int +dequeue_dec_one_op_tb(struct fpga_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union fpga_dma_desc *desc; + uint8_t cbs_in_op, cb_idx, iter_count = 0; + int status = 0; + int desc_error = 0; + /* Set descriptor */ + desc = q->ring_addr + ((q->head_free_desc + desc_offset) + & q->sw_ring_wrap_mask); + + /* Verify if done bit is set */ + if (desc->dec_req.done == 0) + return -1; + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + /* Verify if done bit in all CBs is set */ + cbs_in_op = desc->dec_req.cbs_in_op; + for (cb_idx = 1; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + if (desc->dec_req.done == 0) + return -1; + } + + /* Make sure the response is read atomically */ + rte_smp_rmb(); + + for (cb_idx = 0; cb_idx < cbs_in_op; ++cb_idx) { + desc = q->ring_addr + ((q->head_free_desc + desc_offset + + cb_idx) & q->sw_ring_wrap_mask); + /* get max iter_count for all CBs in op */ + iter_count = RTE_MAX(iter_count, (uint8_t) desc->dec_req.iter); + /* crc_pass = 0 when decoder fails, one fails all */ + status |= !(desc->dec_req.crc_pass) << RTE_BBDEV_CRC_ERROR; + /* Check the decriptor error field, return 1 on error */ + desc_error = check_desc_error(desc->enc_req.error); + status |= desc_error << RTE_BBDEV_DATA_ERROR; + rte_bbdev_log_debug("DMA response desc %p", desc); + } + + *op = desc->dec_req.op_addr; + + /* FPGA reports in half-iterations, get ceiling */ + (*op)->turbo_dec.iter_count = (iter_count + 2) >> 1; + (*op)->status = status; + return cbs_in_op; +} + +static uint16_t +fpga_dequeue_enc(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_enc_op **ops, uint16_t num) +{ + struct fpga_queue *q = q_data->queue_private; + uint32_t avail = (q->tail - q->head_free_desc) & q->sw_ring_wrap_mask; + uint16_t i; + uint16_t dequeued_cbs = 0; + struct rte_bbdev_enc_op *op; + int ret; + + for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { + op = (q->ring_addr + ((q->head_free_desc + dequeued_cbs) + & q->sw_ring_wrap_mask))->enc_req.op_addr; + if (op->turbo_enc.code_block_mode == 0) + ret = dequeue_enc_one_op_tb(q, &ops[i], dequeued_cbs); + else + ret = dequeue_enc_one_op_cb(q, &ops[i], dequeued_cbs); + + if (ret < 0) + break; + + dequeued_cbs += ret; + + rte_bbdev_log_debug("dequeuing enc ops [%d/%d] | head %d | tail %d", + dequeued_cbs, num, q->head_free_desc, q->tail); + } + + /* Update head */ + q->head_free_desc = (q->head_free_desc + dequeued_cbs) & + q->sw_ring_wrap_mask; + + /* Update stats */ + q_data->queue_stats.dequeued_count += i; + + return i; +} + +static uint16_t +fpga_dequeue_dec(struct rte_bbdev_queue_data *q_data, + struct rte_bbdev_dec_op **ops, uint16_t num) +{ + struct fpga_queue *q = q_data->queue_private; + uint32_t avail = (q->tail - q->head_free_desc) & q->sw_ring_wrap_mask; + uint16_t i; + uint16_t dequeued_cbs = 0; + struct rte_bbdev_dec_op *op; + int ret; + + for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { + op = (q->ring_addr + ((q->head_free_desc + dequeued_cbs) + & q->sw_ring_wrap_mask))->dec_req.op_addr; + if (op->turbo_dec.code_block_mode == 0) + ret = dequeue_dec_one_op_tb(q, &ops[i], dequeued_cbs); + else + ret = dequeue_dec_one_op_cb(q, &ops[i], dequeued_cbs); + + if (ret < 0) + break; + + dequeued_cbs += ret; + + rte_bbdev_log_debug("dequeuing dec ops [%d/%d] | head %d | tail %d", + dequeued_cbs, num, q->head_free_desc, q->tail); + } + + /* Update head */ + q->head_free_desc = (q->head_free_desc + dequeued_cbs) & + q->sw_ring_wrap_mask; + + /* Update stats */ + q_data->queue_stats.dequeued_count += i; + + return i; +} + +/* Initialization Function */ +static void +fpga_lte_fec_init(struct rte_bbdev *dev) +{ + struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev->device); + + dev->dev_ops = &fpga_ops; + dev->enqueue_enc_ops = fpga_enqueue_enc; + dev->enqueue_dec_ops = fpga_enqueue_dec; + dev->dequeue_enc_ops = fpga_dequeue_enc; + dev->dequeue_dec_ops = fpga_dequeue_dec; + + ((struct fpga_lte_fec_device *) dev->data->dev_private)->pf_device = + !strcmp(dev->device->driver->name, + RTE_STR(FPGA_LTE_FEC_PF_DRIVER_NAME)); + ((struct fpga_lte_fec_device *) dev->data->dev_private)->mmio_base = + pci_dev->mem_resource[0].addr; + + rte_bbdev_log_debug( + "Init device %s [%s] @ virtaddr %p phyaddr %#"PRIx64, + dev->device->driver->name, dev->data->name, + (void *)pci_dev->mem_resource[0].addr, + pci_dev->mem_resource[0].phys_addr); +} + +static int +fpga_lte_fec_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct rte_bbdev *bbdev = NULL; + char dev_name[RTE_BBDEV_NAME_MAX_LEN]; + + if (pci_dev == NULL) { + rte_bbdev_log(ERR, "NULL PCI device"); + return -EINVAL; + } + + rte_pci_device_name(&pci_dev->addr, dev_name, sizeof(dev_name)); + + /* Allocate memory to be used privately by drivers */ + bbdev = rte_bbdev_allocate(pci_dev->device.name); + if (bbdev == NULL) + return -ENODEV; + + /* allocate device private memory */ + bbdev->data->dev_private = rte_zmalloc_socket(dev_name, + sizeof(struct fpga_lte_fec_device), RTE_CACHE_LINE_SIZE, + pci_dev->device.numa_node); + + if (bbdev->data->dev_private == NULL) { + rte_bbdev_log(CRIT, + "Allocate of %zu bytes for device \"%s\" failed", + sizeof(struct fpga_lte_fec_device), dev_name); + rte_bbdev_release(bbdev); + return -ENOMEM; + } + + /* Fill HW specific part of device structure */ + bbdev->device = &pci_dev->device; + bbdev->intr_handle = &pci_dev->intr_handle; + bbdev->data->socket_id = pci_dev->device.numa_node; + + /* Invoke FEC FPGA device initialization function */ + fpga_lte_fec_init(bbdev); + + rte_bbdev_log_debug("bbdev id = %u [%s]", + bbdev->data->dev_id, dev_name); + + struct fpga_lte_fec_device *d = bbdev->data->dev_private; + uint32_t version_id = fpga_reg_read_32(d->mmio_base, + FPGA_LTE_FEC_VERSION_ID); + rte_bbdev_log(INFO, "FEC FPGA RTL v%u.%u", + ((uint16_t)(version_id >> 16)), ((uint16_t)version_id)); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + if (!strcmp(bbdev->device->driver->name, + RTE_STR(FPGA_LTE_FEC_PF_DRIVER_NAME))) + print_static_reg_debug_info(d->mmio_base); +#endif + return 0; +} + +static int +fpga_lte_fec_remove(struct rte_pci_device *pci_dev) +{ + struct rte_bbdev *bbdev; + int ret; + uint8_t dev_id; + + if (pci_dev == NULL) + return -EINVAL; + + /* Find device */ + bbdev = rte_bbdev_get_named_dev(pci_dev->device.name); + if (bbdev == NULL) { + rte_bbdev_log(CRIT, + "Couldn't find HW dev \"%s\" to uninitialise it", + pci_dev->device.name); + return -ENODEV; + } + dev_id = bbdev->data->dev_id; + + /* free device private memory before close */ + rte_free(bbdev->data->dev_private); + + /* Close device */ + ret = rte_bbdev_close(dev_id); + if (ret < 0) + rte_bbdev_log(ERR, + "Device %i failed to close during uninit: %i", + dev_id, ret); + + /* release bbdev from library */ + ret = rte_bbdev_release(bbdev); + if (ret) + rte_bbdev_log(ERR, "Device %i failed to uninit: %i", dev_id, + ret); + + rte_bbdev_log_debug("Destroyed bbdev = %u", dev_id); + + return 0; +} + +static inline void +set_default_fpga_conf(struct fpga_lte_fec_conf *def_conf) +{ + /* clear default configuration before initialization */ + memset(def_conf, 0, sizeof(struct fpga_lte_fec_conf)); + /* Set pf mode to true */ + def_conf->pf_mode_en = true; + + /* Set ratio between UL and DL to 1:1 (unit of weight is 3 CBs) */ + def_conf->ul_bandwidth = 3; + def_conf->dl_bandwidth = 3; + + /* Set Load Balance Factor to 64 */ + def_conf->dl_load_balance = 64; + def_conf->ul_load_balance = 64; +} + +/* Initial configuration of FPGA LTE FEC device */ +int +fpga_lte_fec_configure(const char *dev_name, + const struct fpga_lte_fec_conf *conf) +{ + uint32_t payload_32, address; + uint16_t payload_16; + uint8_t payload_8; + uint16_t q_id, vf_id, total_q_id, total_ul_q_id, total_dl_q_id; + struct rte_bbdev *bbdev = rte_bbdev_get_named_dev(dev_name); + struct fpga_lte_fec_conf def_conf; + + if (bbdev == NULL) { + rte_bbdev_log(ERR, + "Invalid dev_name (%s), or device is not yet initialised", + dev_name); + return -ENODEV; + } + + struct fpga_lte_fec_device *d = bbdev->data->dev_private; + + if (conf == NULL) { + rte_bbdev_log(ERR, + "FPGA Configuration was not provided. Default configuration will be loaded."); + set_default_fpga_conf(&def_conf); + conf = &def_conf; + } + + /* + * Configure UL:DL ratio. + * [7:0]: UL weight + * [15:8]: DL weight + */ + payload_16 = (conf->dl_bandwidth << 8) | conf->ul_bandwidth; + address = FPGA_LTE_FEC_CONFIGURATION; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Clear all queues registers */ + payload_32 = FPGA_INVALID_HW_QUEUE_ID; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + address = (q_id << 2) + FPGA_LTE_FEC_QUEUE_MAP; + fpga_reg_write_32(d->mmio_base, address, payload_32); + } + + /* + * If PF mode is enabled allocate all queues for PF only. + * + * For VF mode each VF can have different number of UL and DL queues. + * Total number of queues to configure cannot exceed FPGA + * capabilities - 64 queues - 32 queues for UL and 32 queues for DL. + * Queues mapping is done according to configuration: + * + * UL queues: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | ... | 0 | + * | conf->vf_dl_queues_number[0] - 1 | 0 | + * | conf->vf_dl_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_dl_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_dl_queues_number[7] - 1 | 7 | + * + * DL queues: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | ... | 0 | + * | conf->vf_ul_queues_number[0] - 1 | 0 | + * | conf->vf_ul_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_ul_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_ul_queues_number[7] - 1 | 7 | + * + * Example of configuration: + * conf->vf_ul_queues_number[0] = 4; -> 4 UL queues for VF0 + * conf->vf_dl_queues_number[0] = 4; -> 4 DL queues for VF0 + * conf->vf_ul_queues_number[1] = 2; -> 2 UL queues for VF1 + * conf->vf_dl_queues_number[1] = 2; -> 2 DL queues for VF1 + * + * UL: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | 1 | 0 | + * | 2 | 0 | + * | 3 | 0 | + * | 4 | 1 | + * | 5 | 1 | + * + * DL: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | 33 | 0 | + * | 34 | 0 | + * | 35 | 0 | + * | 36 | 1 | + * | 37 | 1 | + */ + if (conf->pf_mode_en) { + payload_32 = 0x1; + for (q_id = 0; q_id < FPGA_TOTAL_NUM_QUEUES; ++q_id) { + address = (q_id << 2) + FPGA_LTE_FEC_QUEUE_MAP; + fpga_reg_write_32(d->mmio_base, address, payload_32); + } + } else { + /* Calculate total number of UL and DL queues to configure */ + total_ul_q_id = total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + total_ul_q_id += conf->vf_ul_queues_number[vf_id]; + total_dl_q_id += conf->vf_dl_queues_number[vf_id]; + } + total_q_id = total_dl_q_id + total_ul_q_id; + /* + * Check if total number of queues to configure does not exceed + * FPGA capabilities (64 queues - 32 UL and 32 DL queues) + */ + if ((total_ul_q_id > FPGA_NUM_UL_QUEUES) || + (total_dl_q_id > FPGA_NUM_DL_QUEUES) || + (total_q_id > FPGA_TOTAL_NUM_QUEUES)) { + rte_bbdev_log(ERR, + "FPGA Configuration failed. Too many queues to configure: UL_Q %u, DL_Q %u, FPGA_Q %u", + total_ul_q_id, total_dl_q_id, + FPGA_TOTAL_NUM_QUEUES); + return -EINVAL; + } + total_ul_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_ul_queues_number[vf_id]; + ++q_id, ++total_ul_q_id) { + address = (total_ul_q_id << 2) + + FPGA_LTE_FEC_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_reg_write_32(d->mmio_base, address, + payload_32); + } + } + total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_LTE_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_dl_queues_number[vf_id]; + ++q_id, ++total_dl_q_id) { + address = ((total_dl_q_id + FPGA_NUM_UL_QUEUES) + << 2) + FPGA_LTE_FEC_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_reg_write_32(d->mmio_base, address, + payload_32); + } + } + } + + /* Setting Load Balance Factor */ + payload_16 = (conf->dl_load_balance << 8) | (conf->ul_load_balance); + address = FPGA_LTE_FEC_LOAD_BALANCE_FACTOR; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting length of ring descriptor entry */ + payload_16 = FPGA_RING_DESC_ENTRY_LENGTH; + address = FPGA_LTE_FEC_RING_DESC_LEN; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting FLR timeout value */ + payload_16 = conf->flr_time_out; + address = FPGA_LTE_FEC_FLR_TIME_OUT; + fpga_reg_write_16(d->mmio_base, address, payload_16); + + /* Queue PF/VF mapping table is ready */ + payload_8 = 0x1; + address = FPGA_LTE_FEC_QUEUE_PF_VF_MAP_DONE; + fpga_reg_write_8(d->mmio_base, address, payload_8); + + rte_bbdev_log_debug("PF FPGA LTE FEC configuration complete for %s", + dev_name); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_static_reg_debug_info(d->mmio_base); +#endif + return 0; +} + +/* FPGA LTE FEC PCI PF address map */ +static struct rte_pci_id pci_id_fpga_lte_fec_pf_map[] = { + { + RTE_PCI_DEVICE(FPGA_LTE_FEC_VENDOR_ID, + FPGA_LTE_FEC_PF_DEVICE_ID) + }, + {.device_id = 0}, +}; + +static struct rte_pci_driver fpga_lte_fec_pci_pf_driver = { + .probe = fpga_lte_fec_probe, + .remove = fpga_lte_fec_remove, + .id_table = pci_id_fpga_lte_fec_pf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING +}; + +/* FPGA LTE FEC PCI VF address map */ +static struct rte_pci_id pci_id_fpga_lte_fec_vf_map[] = { + { + RTE_PCI_DEVICE(FPGA_LTE_FEC_VENDOR_ID, + FPGA_LTE_FEC_VF_DEVICE_ID) + }, + {.device_id = 0}, +}; + +static struct rte_pci_driver fpga_lte_fec_pci_vf_driver = { + .probe = fpga_lte_fec_probe, + .remove = fpga_lte_fec_remove, + .id_table = pci_id_fpga_lte_fec_vf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING +}; + + +RTE_PMD_REGISTER_PCI(FPGA_LTE_FEC_PF_DRIVER_NAME, fpga_lte_fec_pci_pf_driver); +RTE_PMD_REGISTER_PCI_TABLE(FPGA_LTE_FEC_PF_DRIVER_NAME, + pci_id_fpga_lte_fec_pf_map); +RTE_PMD_REGISTER_PCI(FPGA_LTE_FEC_VF_DRIVER_NAME, fpga_lte_fec_pci_vf_driver); +RTE_PMD_REGISTER_PCI_TABLE(FPGA_LTE_FEC_VF_DRIVER_NAME, + pci_id_fpga_lte_fec_vf_map); + +RTE_INIT(fpga_lte_fec_init_log) +{ + fpga_lte_fec_logtype = rte_log_register("pmd.bb.fpga_lte_fec"); + if (fpga_lte_fec_logtype >= 0) +#ifdef RTE_LIBRTE_BBDEV_DEBUG + rte_log_set_level(fpga_lte_fec_logtype, RTE_LOG_DEBUG); +#else + rte_log_set_level(fpga_lte_fec_logtype, RTE_LOG_NOTICE); +#endif +} diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h new file mode 100644 index 0000000..9ae8b12 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation + */ + +#ifndef _FPGA_LTE_FEC_H_ +#define _FPGA_LTE_FEC_H_ + +#include <stdint.h> +#include <stdbool.h> + +/** + * @file fpga_lte_fec.h + * + * Interface for Intel(R) FGPA LTE FEC device configuration at the host level, + * directly accessible by the application. + * Configuration related to LTE Turbo coding functionality is done through + * librte_bbdev library. + * + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + */ + +#ifdef __cplusplus +extern "C" { +#endif + +/**< Number of Virtual Functions FGPA 4G FEC supports */ +#define FPGA_LTE_FEC_NUM_VFS 8 + +/** + * Structure to pass FPGA 4G FEC configuration. + */ +struct fpga_lte_fec_conf { + /**< 1 if PF is used for dataplane, 0 for VFs */ + bool pf_mode_en; + /**< Number of UL queues per VF */ + uint8_t vf_ul_queues_number[FPGA_LTE_FEC_NUM_VFS]; + /**< Number of DL queues per VF */ + uint8_t vf_dl_queues_number[FPGA_LTE_FEC_NUM_VFS]; + /**< UL bandwidth. Needed for schedule algorithm */ + uint8_t ul_bandwidth; + /**< DL bandwidth. Needed for schedule algorithm */ + uint8_t dl_bandwidth; + /**< UL Load Balance */ + uint8_t ul_load_balance; + /**< DL Load Balance */ + uint8_t dl_load_balance; + /**< FLR timeout value */ + uint16_t flr_time_out; +}; + +/** + * Configure Intel(R) FPGA LTE FEC device + * + * @param dev_name + * The name of the device. This is the short form of PCI BDF, e.g. 00:01.0. + * It can also be retrieved for a bbdev device from the dev_name field in the + * rte_bbdev_info structure returned by rte_bbdev_info_get(). + * @param conf + * Configuration to apply to FPGA 4G FEC. + * + * @return + * Zero on success, negative value on failure. + */ +int +fpga_lte_fec_configure(const char *dev_name, + const struct fpga_lte_fec_conf *conf); + +#ifdef __cplusplus +} +#endif + +#endif /* _FPGA_LTE_FEC_H_ */ diff --git a/drivers/baseband/fpga_lte_fec/meson.build b/drivers/baseband/fpga_lte_fec/meson.build new file mode 100644 index 0000000..bf44e6b --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +deps += ['bbdev', 'bus_vdev', 'ring', 'pci', 'bus_pci'] +name = 'bbdev_fpga_lte_fec' +allow_experimental_apis = true +sources = files('fpga_lte_fec.c') diff --git a/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map new file mode 100644 index 0000000..e923270 --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/rte_pmd_bbdev_fpga_lte_fec_version.map @@ -0,0 +1,3 @@ +DPDK_19.08 { + local: *; +}; diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build index 52489df..a17d549 100644 --- a/drivers/baseband/meson.build +++ b/drivers/baseband/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <bluca@debian.org> -drivers = ['null'] +drivers = ['null', 'fpga_lte_fec'] config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' driver_name_fmt = 'rte_pmd_@0@' diff --git a/mk/rte.app.mk b/mk/rte.app.mk index b6106e1..73bba4a 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -215,6 +215,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_NETVSC_PMD) += -lrte_pmd_netvsc ifeq ($(CONFIG_RTE_LIBRTE_BBDEV),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC) += -lrte_pmd_fpga_lte_fec # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [dpdk-dev] [PATCH v6] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-14 16:17 ` Nicolas Chautru @ 2019-06-19 15:20 ` Chautru, Nicolas [not found] ` <EEA9FF629BF25B47BD67ADE995041EE2496B2975@IRSMSX103.ger.corp.intel.com> 1 sibling, 0 replies; 62+ messages in thread From: Chautru, Nicolas @ 2019-06-19 15:20 UTC (permalink / raw) To: akhil.goyal, dev; +Cc: Yigit, Ferruh, thomas, Mokhtar, Amr, Chalupnik, KamilX >-----Original Message----- >From: Chautru, Nicolas >Sent: Friday, June 14, 2019 9:18 AM >To: akhil.goyal@nxp.com; dev@dpdk.org vCc: Yigit, Ferruh <ferruh.yigit@intel.com>; thomas@monjalon.net; Mokhtar, Amr <amr.mokhtar@intel.com>; Chalupnik, KamilX <kamilx.chalupnik@intel.com>; Chautru, Nicolas <nicolas.chautru@intel.com> >Subject: [PATCH v6] baseband/fpga_lte_fec: adding driver for FEC on FPGA > >Supports for FEC 4G PMD Driver on FPGA card PAC N3000 > >Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> Acked-by: Nicolas Chautru <nicolas.chautru@intel.com> ^ permalink raw reply [flat|nested] 62+ messages in thread
[parent not found: <EEA9FF629BF25B47BD67ADE995041EE2496B2975@IRSMSX103.ger.corp.intel.com>]
* Re: [dpdk-dev] [PATCH v6] baseband/fpga_lte_fec: adding driver for FEC on FPGA [not found] ` <EEA9FF629BF25B47BD67ADE995041EE2496B2975@IRSMSX103.ger.corp.intel.com> @ 2019-06-20 13:35 ` Ferruh Yigit 2019-06-25 12:37 ` Akhil Goyal 0 siblings, 1 reply; 62+ messages in thread From: Ferruh Yigit @ 2019-06-20 13:35 UTC (permalink / raw) To: Chalupnik, KamilX, Chautru, Nicolas, akhil.goyal, dev Cc: thomas, Mokhtar, Amr On 6/18/2019 7:40 AM, Chalupnik, KamilX wrote: >> -----Original Message----- >> From: Chautru, Nicolas >> Sent: Friday, June 14, 2019 6:18 PM >> To: akhil.goyal@nxp.com; dev@dpdk.org >> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; thomas@monjalon.net; Mokhtar, >> Amr <amr.mokhtar@intel.com>; Chalupnik, KamilX >> <kamilx.chalupnik@intel.com>; Chautru, Nicolas >> <nicolas.chautru@intel.com> >> Subject: [PATCH v6] baseband/fpga_lte_fec: adding driver for FEC on FPGA >> >> Supports for FEC 4G PMD Driver on FPGA card PAC N3000 >> >> Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> > > Acked-by: Kamil Chalupnik <kamilx.chalupnik@intel.com> > Above is the ack from Kamil, for some reason seems not hit the mail list, will carry here so that patchwork can pick it. Acked-by: Kamil Chalupnik <kamilx.chalupnik@intel.com> ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [dpdk-dev] [PATCH v6] baseband/fpga_lte_fec: adding driver for FEC on FPGA 2019-06-20 13:35 ` Ferruh Yigit @ 2019-06-25 12:37 ` Akhil Goyal 0 siblings, 0 replies; 62+ messages in thread From: Akhil Goyal @ 2019-06-25 12:37 UTC (permalink / raw) To: Ferruh Yigit, Chalupnik, KamilX, Chautru, Nicolas, dev Cc: thomas, Mokhtar, Amr > > On 6/18/2019 7:40 AM, Chalupnik, KamilX wrote: > >> -----Original Message----- > >> From: Chautru, Nicolas > >> Sent: Friday, June 14, 2019 6:18 PM > >> To: akhil.goyal@nxp.com; dev@dpdk.org > >> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; thomas@monjalon.net; Mokhtar, > >> Amr <amr.mokhtar@intel.com>; Chalupnik, KamilX > >> <kamilx.chalupnik@intel.com>; Chautru, Nicolas > >> <nicolas.chautru@intel.com> > >> Subject: [PATCH v6] baseband/fpga_lte_fec: adding driver for FEC on FPGA > >> > >> Supports for FEC 4G PMD Driver on FPGA card PAC N3000 > >> > >> Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> > > > > Acked-by: Kamil Chalupnik <kamilx.chalupnik@intel.com> > > > > Above is the ack from Kamil, for some reason seems not hit the mail list, will > carry here so that patchwork can pick it. > > Acked-by: Kamil Chalupnik <kamilx.chalupnik@intel.com> Applied to dpdk-next-crypto Thanks ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v3 0/3] BBDEV turbo_sw PMD compilation fix 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 1/3] " Nicolas Chautru 2019-06-06 9:04 ` [dpdk-dev] [PATCH v3] " Nicolas Chautru @ 2019-06-06 10:17 ` Nicolas Chautru 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru ` (2 more replies) 1 sibling, 3 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-06 10:17 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Update v3: Cosmetic changes in documentation commit to be more 4G/AVX2 specific. Update v2: Splitting into 3 patches as recommended (ignore previous v2 which had a typo) Based on discussion with maintainer, pushing first a patch to help maintenance of the baseband_turbo_sw which had been lacking. The documentation is clarified to point to steps on building the SDK libraries which are now publicly available. (Cosmetic changes to the webpage containing these SDK will happen in parallel based on feedback from maintainer). A compile flag is added to be able to build the turbo_sw PMD when the SDK libraries for AVX2 are installed or not. In both cases this can be compiled with gcc RTE_TARGET. Missing meson build support is also added. Note that additional BBDEV changes pushed in previous v1 https://patches.dpdk.org/project/dpdk/list/?series=4657 will be added in a separate v2 patchset which will depend on this very patchset. Nicolas Chautru (3): baseband/turbo_sw: option to build turbosw PMD without SDK docs/guides: updating turbo_sw building steps baseband/turbo_sw: meson build support for PMD driver config/common_base | 3 +- doc/guides/bbdevs/turbo_sw.rst | 81 +++++++++++++----------- drivers/baseband/meson.build | 2 +- drivers/baseband/turbo_sw/Makefile | 13 ++-- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 39 +++++++++++- drivers/baseband/turbo_sw/meson.build | 30 +++++++++ mk/rte.app.mk | 3 + 7 files changed, 125 insertions(+), 46 deletions(-) create mode 100644 drivers/baseband/turbo_sw/meson.build -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v3 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru @ 2019-06-06 10:17 ` Nicolas Chautru 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2 siblings, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-06 10:17 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Adding compile flag to allow to build the turbo_sw PMD without dependency to have the SDK libraries installed. Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 3 +- drivers/baseband/turbo_sw/Makefile | 13 ++++---- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 39 +++++++++++++++++++++++- mk/rte.app.mk | 3 ++ 4 files changed, 50 insertions(+), 8 deletions(-) diff --git a/config/common_base b/config/common_base index 6b96e0e..bc80209 100644 --- a/config/common_base +++ b/config/common_base @@ -523,6 +523,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y CONFIG_RTE_LIBRTE_BBDEV=y CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y +CONFIG_RTE_BBDEV_SDK_AVX2=n # # Compile PMD for NULL bbdev device @@ -532,7 +533,7 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y # # Compile PMD for turbo software bbdev device # -CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n +CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y # # Compile generic crypto device library diff --git a/drivers/baseband/turbo_sw/Makefile b/drivers/baseband/turbo_sw/Makefile index d364677..414d0d9 100644 --- a/drivers/baseband/turbo_sw/Makefile +++ b/drivers/baseband/turbo_sw/Makefile @@ -3,9 +3,6 @@ include $(RTE_SDK)/mk/rte.vars.mk -ifeq ($(FLEXRAN_SDK),) -$(error "Please define FLEXRAN_SDK environment variable") -endif # library name LIB = librte_pmd_bbdev_turbo_sw.a @@ -21,17 +18,21 @@ LDLIBS += -lrte_bus_vdev # versioning export map EXPORT_MAP := rte_pmd_bbdev_turbo_sw_version.map -# external library dependencies +# external library dependencies if available +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +ifeq ($(FLEXRAN_SDK),) +$(error "Please define FLEXRAN_SDK environment variable") +endif CFLAGS += -I$(FLEXRAN_SDK)/lib_common CFLAGS += -I$(FLEXRAN_SDK)/lib_turbo CFLAGS += -I$(FLEXRAN_SDK)/lib_crc CFLAGS += -I$(FLEXRAN_SDK)/lib_rate_matching - LDLIBS += -L$(FLEXRAN_SDK)/lib_turbo -lturbo LDLIBS += -L$(FLEXRAN_SDK)/lib_crc -lcrc LDLIBS += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching LDLIBS += -L$(FLEXRAN_SDK)/lib_common -lcommon -LDLIBS += -lstdc++ -lirc -limf -lipps +LDLIBS += -lstdc++ -lirc -limf -lipps -lsvml +endif # library version LIBABIVER := 1 diff --git a/drivers/baseband/turbo_sw/bbdev_turbo_software.c b/drivers/baseband/turbo_sw/bbdev_turbo_software.c index 5204a77..4781de0 100644 --- a/drivers/baseband/turbo_sw/bbdev_turbo_software.c +++ b/drivers/baseband/turbo_sw/bbdev_turbo_software.c @@ -14,10 +14,11 @@ #include <rte_bbdev.h> #include <rte_bbdev_pmd.h> +#ifdef RTE_BBDEV_SDK_AVX2 #include <phy_turbo.h> #include <phy_crc.h> #include <phy_rate_match.h> -#include <divide.h> +#endif #define DRIVER_NAME baseband_turbo_sw @@ -143,6 +144,7 @@ struct turbo_sw_queue { struct bbdev_private *internals = dev->data->dev_private; static const struct rte_bbdev_op_cap bbdev_capabilities[] = { +#ifdef RTE_BBDEV_SDK_AVX2 { .type = RTE_BBDEV_OP_TURBO_DEC, .cap.turbo_dec = { @@ -172,6 +174,7 @@ struct turbo_sw_queue { .num_buffers_dst = RTE_BBDEV_MAX_CODE_BLOCKS, } }, +#endif RTE_BBDEV_END_OF_CAPABILITIES_LIST() }; @@ -472,6 +475,7 @@ struct turbo_sw_queue { struct rte_mbuf *m_out, uint16_t in_offset, uint16_t out_offset, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int16_t k_idx; uint16_t m; @@ -724,6 +728,22 @@ struct turbo_sw_queue { } *tmp_out = 0; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(r); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(ncb); + RTE_SET_USED(e); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void @@ -856,6 +876,7 @@ struct turbo_sw_queue { uint16_t crc24_overlap, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int32_t k_idx; int32_t iter_cnt; @@ -972,6 +993,22 @@ struct turbo_sw_queue { rte_bbdev_log(ERR, "Turbo Decoder failed"); return; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(kw); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(check_crc_24b); + RTE_SET_USED(crc24_overlap); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 7c9b4b5..878cc31 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -221,11 +221,14 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +# Dependency on the FLEXRAN SDK library if available _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_crc -lcrc _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_turbo -lturbo _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_common -lcommon _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lirc -limf -lstdc++ -lipps +endif # CONFIG_RTE_BBDEV_SDK_AVX2 endif # CONFIG_RTE_LIBRTE_BBDEV ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y) -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v4 0/3] BBDEV turbo_sw PMD compilation fix 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru @ 2019-06-07 23:54 ` Nicolas Chautru 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru ` (2 more replies) 0 siblings, 3 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-07 23:54 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Update v4: Missed one file for meson build path and minor change to prevent warning for some configurations due to unused symbols. Update v3: Cosmetic changes in documentation commit to be more 4G/AVX2 specific. Update v2: Splitting into 3 patches as recommended (ignore previous v2 which had a typo) Based on discussion with maintainer, pushing first a patch to help maintenance of the baseband_turbo_sw which had been lacking. The documentation is clarified to point to steps on building the SDK libraries which are now publicly available. (Cosmetic changes to the webpage containing these SDK will happen in parallel based on feedback from maintainer). A compile flag is added to be able to build the turbo_sw PMD when the SDK libraries for AVX2 are installed or not. In both cases this can be compiled with gcc RTE_TARGET. Missing meson build support is also added. Note that additional BBDEV changes pushed in previous v1 https://patches.dpdk.org/project/dpdk/list/?series=4657 will be added in a separate v2 patchset which will depend on this very patchset. Nicolas Chautru (3): baseband/turbo_sw: option to build turbosw PMD without SDK docs/guides: updating turbo_sw building steps baseband/turbo_sw: meson build support for PMD driver config/common_base | 3 +- doc/guides/bbdevs/turbo_sw.rst | 81 +++++++++++++----------- drivers/baseband/meson.build | 2 +- drivers/baseband/turbo_sw/Makefile | 13 ++-- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 45 ++++++++++++- drivers/baseband/turbo_sw/meson.build | 30 +++++++++ meson_options.txt | 2 + mk/rte.app.mk | 3 + 8 files changed, 133 insertions(+), 46 deletions(-) create mode 100644 drivers/baseband/turbo_sw/meson.build -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v4 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru @ 2019-06-07 23:54 ` Nicolas Chautru 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2 siblings, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-07 23:54 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Adding compile flag to allow to build the turbo_sw PMD without dependency to have the SDK libraries installed. Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 3 +- drivers/baseband/turbo_sw/Makefile | 13 +++---- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 45 +++++++++++++++++++++++- mk/rte.app.mk | 3 ++ 4 files changed, 56 insertions(+), 8 deletions(-) diff --git a/config/common_base b/config/common_base index 6b96e0e..bc80209 100644 --- a/config/common_base +++ b/config/common_base @@ -523,6 +523,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y CONFIG_RTE_LIBRTE_BBDEV=y CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y +CONFIG_RTE_BBDEV_SDK_AVX2=n # # Compile PMD for NULL bbdev device @@ -532,7 +533,7 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y # # Compile PMD for turbo software bbdev device # -CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n +CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y # # Compile generic crypto device library diff --git a/drivers/baseband/turbo_sw/Makefile b/drivers/baseband/turbo_sw/Makefile index d364677..414d0d9 100644 --- a/drivers/baseband/turbo_sw/Makefile +++ b/drivers/baseband/turbo_sw/Makefile @@ -3,9 +3,6 @@ include $(RTE_SDK)/mk/rte.vars.mk -ifeq ($(FLEXRAN_SDK),) -$(error "Please define FLEXRAN_SDK environment variable") -endif # library name LIB = librte_pmd_bbdev_turbo_sw.a @@ -21,17 +18,21 @@ LDLIBS += -lrte_bus_vdev # versioning export map EXPORT_MAP := rte_pmd_bbdev_turbo_sw_version.map -# external library dependencies +# external library dependencies if available +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +ifeq ($(FLEXRAN_SDK),) +$(error "Please define FLEXRAN_SDK environment variable") +endif CFLAGS += -I$(FLEXRAN_SDK)/lib_common CFLAGS += -I$(FLEXRAN_SDK)/lib_turbo CFLAGS += -I$(FLEXRAN_SDK)/lib_crc CFLAGS += -I$(FLEXRAN_SDK)/lib_rate_matching - LDLIBS += -L$(FLEXRAN_SDK)/lib_turbo -lturbo LDLIBS += -L$(FLEXRAN_SDK)/lib_crc -lcrc LDLIBS += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching LDLIBS += -L$(FLEXRAN_SDK)/lib_common -lcommon -LDLIBS += -lstdc++ -lirc -limf -lipps +LDLIBS += -lstdc++ -lirc -limf -lipps -lsvml +endif # library version LIBABIVER := 1 diff --git a/drivers/baseband/turbo_sw/bbdev_turbo_software.c b/drivers/baseband/turbo_sw/bbdev_turbo_software.c index 5204a77..cac8920 100644 --- a/drivers/baseband/turbo_sw/bbdev_turbo_software.c +++ b/drivers/baseband/turbo_sw/bbdev_turbo_software.c @@ -14,10 +14,11 @@ #include <rte_bbdev.h> #include <rte_bbdev_pmd.h> +#ifdef RTE_BBDEV_SDK_AVX2 #include <phy_turbo.h> #include <phy_crc.h> #include <phy_rate_match.h> -#include <divide.h> +#endif #define DRIVER_NAME baseband_turbo_sw @@ -83,6 +84,7 @@ struct turbo_sw_queue { enum rte_bbdev_op_type type; } __rte_cache_aligned; +#ifdef RTE_BBDEV_SDK_AVX2 static inline char * mbuf_append(struct rte_mbuf *m_head, struct rte_mbuf *m, uint16_t len) { @@ -128,6 +130,7 @@ struct turbo_sw_queue { return result; } +#endif /* Read flag value 0/1 from bitmap */ static inline bool @@ -143,6 +146,7 @@ struct turbo_sw_queue { struct bbdev_private *internals = dev->data->dev_private; static const struct rte_bbdev_op_cap bbdev_capabilities[] = { +#ifdef RTE_BBDEV_SDK_AVX2 { .type = RTE_BBDEV_OP_TURBO_DEC, .cap.turbo_dec = { @@ -172,6 +176,7 @@ struct turbo_sw_queue { .num_buffers_dst = RTE_BBDEV_MAX_CODE_BLOCKS, } }, +#endif RTE_BBDEV_END_OF_CAPABILITIES_LIST() }; @@ -410,6 +415,7 @@ struct turbo_sw_queue { .queue_release = q_release }; +#ifdef RTE_BBDEV_SDK_AVX2 /* Checks if the encoder input buffer is correct. * Returns 0 if it's valid, -1 otherwise. */ @@ -464,6 +470,7 @@ struct turbo_sw_queue { return 0; } +#endif static inline void process_enc_cb(struct turbo_sw_queue *q, struct rte_bbdev_enc_op *op, @@ -472,6 +479,7 @@ struct turbo_sw_queue { struct rte_mbuf *m_out, uint16_t in_offset, uint16_t out_offset, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int16_t k_idx; uint16_t m; @@ -724,6 +732,22 @@ struct turbo_sw_queue { } *tmp_out = 0; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(r); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(ncb); + RTE_SET_USED(e); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void @@ -835,6 +859,7 @@ struct turbo_sw_queue { NULL); } +#ifdef RTE_BBDEV_SDK_AVX2 static inline void move_padding_bytes(const uint8_t *in, uint8_t *out, uint16_t k, uint16_t ncb) @@ -847,6 +872,7 @@ struct turbo_sw_queue { rte_memcpy(&out[nd + kpi + 64], &in[kpi], d); rte_memcpy(&out[(nd - 1) + 2 * (kpi + 64)], &in[2 * kpi], d); } +#endif static inline void process_dec_cb(struct turbo_sw_queue *q, struct rte_bbdev_dec_op *op, @@ -856,6 +882,7 @@ struct turbo_sw_queue { uint16_t crc24_overlap, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int32_t k_idx; int32_t iter_cnt; @@ -972,6 +999,22 @@ struct turbo_sw_queue { rte_bbdev_log(ERR, "Turbo Decoder failed"); return; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(kw); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(check_crc_24b); + RTE_SET_USED(crc24_overlap); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 7c9b4b5..878cc31 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -221,11 +221,14 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +# Dependency on the FLEXRAN SDK library if available _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_crc -lcrc _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_turbo -lturbo _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_common -lcommon _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lirc -limf -lstdc++ -lipps +endif # CONFIG_RTE_BBDEV_SDK_AVX2 endif # CONFIG_RTE_LIBRTE_BBDEV ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y) -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v5 0/3] BBDEV turbo_sw PMD compilation fix 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru @ 2019-06-13 16:51 ` Nicolas Chautru 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru ` (2 more replies) 0 siblings, 3 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-13 16:51 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, Nicolas Chautru Update v5: Cosmetic change to remove trailing space and to commit message. Rebased to latest. Update v4: Missed one file for meson build path and minor change to prevent warning for some configurations due to unused symbols. Update v3: Cosmetic changes in documentation commit to be more 4G/AVX2 specific. Update v2: Splitting into 3 patches as recommended (ignore previous v2 which had a typo) Based on discussion with maintainer, pushing first a patch to help maintenance of the baseband_turbo_sw which had been lacking. The documentation is clarified to point to steps on building the SDK libraries which are now publicly available. (Cosmetic changes to the webpage containing these SDK will happen in parallel based on feedback from maintainer). A compile flag is added to be able to build the turbo_sw PMD when the SDK libraries for AVX2 are installed or not. In both cases this can be compiled with gcc RTE_TARGET. Missing meson build support is also added. Note that additional BBDEV changes pushed in previous v1 https://patches.dpdk.org/project/dpdk/list/?series=4657 will be added in a separate v2 patchset which will depend on this very patchset. Nicolas Chautru (3): baseband/turbo_sw: option to build turbosw PMD without SDK docs/guides: updating turbo_sw building steps baseband/turbo_sw: meson build support for PMD driver config/common_base | 3 +- doc/guides/bbdevs/turbo_sw.rst | 81 +++++++++++++----------- drivers/baseband/meson.build | 2 +- drivers/baseband/turbo_sw/Makefile | 13 ++-- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 45 ++++++++++++- drivers/baseband/turbo_sw/meson.build | 30 +++++++++ meson_options.txt | 2 + mk/rte.app.mk | 3 + 8 files changed, 133 insertions(+), 46 deletions(-) create mode 100644 drivers/baseband/turbo_sw/meson.build -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v5 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru @ 2019-06-13 16:51 ` Nicolas Chautru 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2 siblings, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-13 16:51 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, Nicolas Chautru Adding compile flag to allow to build the turbo_sw PMD without dependency to have the SDK libraries installed. Acked-by: Kamil Chalupnik <kamilx.chalupnik@intel.com> Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 3 +- drivers/baseband/turbo_sw/Makefile | 13 +++---- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 45 +++++++++++++++++++++++- mk/rte.app.mk | 3 ++ 4 files changed, 56 insertions(+), 8 deletions(-) diff --git a/config/common_base b/config/common_base index 6f19ad5..06c4847 100644 --- a/config/common_base +++ b/config/common_base @@ -523,6 +523,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y CONFIG_RTE_LIBRTE_BBDEV=y CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y +CONFIG_RTE_BBDEV_SDK_AVX2=n # # Compile PMD for NULL bbdev device @@ -532,7 +533,7 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y # # Compile PMD for turbo software bbdev device # -CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n +CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y # # Compile generic crypto device library diff --git a/drivers/baseband/turbo_sw/Makefile b/drivers/baseband/turbo_sw/Makefile index d364677..414d0d9 100644 --- a/drivers/baseband/turbo_sw/Makefile +++ b/drivers/baseband/turbo_sw/Makefile @@ -3,9 +3,6 @@ include $(RTE_SDK)/mk/rte.vars.mk -ifeq ($(FLEXRAN_SDK),) -$(error "Please define FLEXRAN_SDK environment variable") -endif # library name LIB = librte_pmd_bbdev_turbo_sw.a @@ -21,17 +18,21 @@ LDLIBS += -lrte_bus_vdev # versioning export map EXPORT_MAP := rte_pmd_bbdev_turbo_sw_version.map -# external library dependencies +# external library dependencies if available +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +ifeq ($(FLEXRAN_SDK),) +$(error "Please define FLEXRAN_SDK environment variable") +endif CFLAGS += -I$(FLEXRAN_SDK)/lib_common CFLAGS += -I$(FLEXRAN_SDK)/lib_turbo CFLAGS += -I$(FLEXRAN_SDK)/lib_crc CFLAGS += -I$(FLEXRAN_SDK)/lib_rate_matching - LDLIBS += -L$(FLEXRAN_SDK)/lib_turbo -lturbo LDLIBS += -L$(FLEXRAN_SDK)/lib_crc -lcrc LDLIBS += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching LDLIBS += -L$(FLEXRAN_SDK)/lib_common -lcommon -LDLIBS += -lstdc++ -lirc -limf -lipps +LDLIBS += -lstdc++ -lirc -limf -lipps -lsvml +endif # library version LIBABIVER := 1 diff --git a/drivers/baseband/turbo_sw/bbdev_turbo_software.c b/drivers/baseband/turbo_sw/bbdev_turbo_software.c index 5204a77..cac8920 100644 --- a/drivers/baseband/turbo_sw/bbdev_turbo_software.c +++ b/drivers/baseband/turbo_sw/bbdev_turbo_software.c @@ -14,10 +14,11 @@ #include <rte_bbdev.h> #include <rte_bbdev_pmd.h> +#ifdef RTE_BBDEV_SDK_AVX2 #include <phy_turbo.h> #include <phy_crc.h> #include <phy_rate_match.h> -#include <divide.h> +#endif #define DRIVER_NAME baseband_turbo_sw @@ -83,6 +84,7 @@ struct turbo_sw_queue { enum rte_bbdev_op_type type; } __rte_cache_aligned; +#ifdef RTE_BBDEV_SDK_AVX2 static inline char * mbuf_append(struct rte_mbuf *m_head, struct rte_mbuf *m, uint16_t len) { @@ -128,6 +130,7 @@ struct turbo_sw_queue { return result; } +#endif /* Read flag value 0/1 from bitmap */ static inline bool @@ -143,6 +146,7 @@ struct turbo_sw_queue { struct bbdev_private *internals = dev->data->dev_private; static const struct rte_bbdev_op_cap bbdev_capabilities[] = { +#ifdef RTE_BBDEV_SDK_AVX2 { .type = RTE_BBDEV_OP_TURBO_DEC, .cap.turbo_dec = { @@ -172,6 +176,7 @@ struct turbo_sw_queue { .num_buffers_dst = RTE_BBDEV_MAX_CODE_BLOCKS, } }, +#endif RTE_BBDEV_END_OF_CAPABILITIES_LIST() }; @@ -410,6 +415,7 @@ struct turbo_sw_queue { .queue_release = q_release }; +#ifdef RTE_BBDEV_SDK_AVX2 /* Checks if the encoder input buffer is correct. * Returns 0 if it's valid, -1 otherwise. */ @@ -464,6 +470,7 @@ struct turbo_sw_queue { return 0; } +#endif static inline void process_enc_cb(struct turbo_sw_queue *q, struct rte_bbdev_enc_op *op, @@ -472,6 +479,7 @@ struct turbo_sw_queue { struct rte_mbuf *m_out, uint16_t in_offset, uint16_t out_offset, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int16_t k_idx; uint16_t m; @@ -724,6 +732,22 @@ struct turbo_sw_queue { } *tmp_out = 0; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(r); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(ncb); + RTE_SET_USED(e); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void @@ -835,6 +859,7 @@ struct turbo_sw_queue { NULL); } +#ifdef RTE_BBDEV_SDK_AVX2 static inline void move_padding_bytes(const uint8_t *in, uint8_t *out, uint16_t k, uint16_t ncb) @@ -847,6 +872,7 @@ struct turbo_sw_queue { rte_memcpy(&out[nd + kpi + 64], &in[kpi], d); rte_memcpy(&out[(nd - 1) + 2 * (kpi + 64)], &in[2 * kpi], d); } +#endif static inline void process_dec_cb(struct turbo_sw_queue *q, struct rte_bbdev_dec_op *op, @@ -856,6 +882,7 @@ struct turbo_sw_queue { uint16_t crc24_overlap, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int32_t k_idx; int32_t iter_cnt; @@ -972,6 +999,22 @@ struct turbo_sw_queue { rte_bbdev_log(ERR, "Turbo Decoder failed"); return; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(kw); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(check_crc_24b); + RTE_SET_USED(crc24_overlap); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void diff --git a/mk/rte.app.mk b/mk/rte.app.mk index b6106e1..3a2183b 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -218,11 +218,14 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +# Dependency on the FLEXRAN SDK library if available _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_crc -lcrc _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_turbo -lturbo _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_common -lcommon _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lirc -limf -lstdc++ -lipps +endif # CONFIG_RTE_BBDEV_SDK_AVX2 endif # CONFIG_RTE_LIBRTE_BBDEV ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y) -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v6 0/3] BBDEV turbo_sw PMD compilation fix 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru @ 2019-06-19 17:11 ` Nicolas Chautru 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru ` (2 more replies) 0 siblings, 3 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-19 17:11 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, Nicolas Chautru Update v6: Cosmetic change to commit message to include previous Acked-by. Update v5: Cosmetic change to remove trailing space and to commit message. Rebased to latest. Update v4: Missed one file for meson build path and minor change to prevent warning for some configurations due to unused symbols. Update v3: Cosmetic changes in documentation commit to be more 4G/AVX2 specific. Update v2: Splitting into 3 patches as recommended (ignore previous v2 which had a typo) Based on discussion with maintainer, pushing first a patch to help maintenance of the baseband_turbo_sw which had been lacking. The documentation is clarified to point to steps on building the SDK libraries which are now publicly available. (Cosmetic changes to the webpage containing these SDK will happen in parallel based on feedback from maintainer). A compile flag is added to be able to build the turbo_sw PMD when the SDK libraries for AVX2 are installed or not. In both cases this can be compiled with gcc RTE_TARGET. Missing meson build support is also added. Note that additional BBDEV changes pushed in previous v1 https://patches.dpdk.org/project/dpdk/list/?series=4657 will be added in a separate v2 patchset which will depend on this very patchset. Nicolas Chautru (3): baseband/turbo_sw: option to build turbosw PMD without SDK docs/guides: updating turbo_sw building steps baseband/turbo_sw: meson build support for PMD driver config/common_base | 3 +- doc/guides/bbdevs/turbo_sw.rst | 81 +++++++++++++----------- drivers/baseband/meson.build | 2 +- drivers/baseband/turbo_sw/Makefile | 13 ++-- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 45 ++++++++++++- drivers/baseband/turbo_sw/meson.build | 30 +++++++++ meson_options.txt | 2 + mk/rte.app.mk | 3 + 8 files changed, 133 insertions(+), 46 deletions(-) create mode 100644 drivers/baseband/turbo_sw/meson.build -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v6 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru @ 2019-06-19 17:11 ` Nicolas Chautru 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2 siblings, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-19 17:11 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, Nicolas Chautru Adding compile flag to allow to build the turbo_sw PMD without dependency to have the SDK libraries installed. Acked-by: Kamil Chalupnik <kamilx.chalupnik@intel.com> Acked-by: Nicolas Chautru <nicolas.chautru@intel.com> Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 3 +- drivers/baseband/turbo_sw/Makefile | 13 +++---- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 45 +++++++++++++++++++++++- mk/rte.app.mk | 3 ++ 4 files changed, 56 insertions(+), 8 deletions(-) diff --git a/config/common_base b/config/common_base index 6b96e0e..bc80209 100644 --- a/config/common_base +++ b/config/common_base @@ -523,6 +523,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y CONFIG_RTE_LIBRTE_BBDEV=y CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y +CONFIG_RTE_BBDEV_SDK_AVX2=n # # Compile PMD for NULL bbdev device @@ -532,7 +533,7 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y # # Compile PMD for turbo software bbdev device # -CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n +CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y # # Compile generic crypto device library diff --git a/drivers/baseband/turbo_sw/Makefile b/drivers/baseband/turbo_sw/Makefile index d364677..414d0d9 100644 --- a/drivers/baseband/turbo_sw/Makefile +++ b/drivers/baseband/turbo_sw/Makefile @@ -3,9 +3,6 @@ include $(RTE_SDK)/mk/rte.vars.mk -ifeq ($(FLEXRAN_SDK),) -$(error "Please define FLEXRAN_SDK environment variable") -endif # library name LIB = librte_pmd_bbdev_turbo_sw.a @@ -21,17 +18,21 @@ LDLIBS += -lrte_bus_vdev # versioning export map EXPORT_MAP := rte_pmd_bbdev_turbo_sw_version.map -# external library dependencies +# external library dependencies if available +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +ifeq ($(FLEXRAN_SDK),) +$(error "Please define FLEXRAN_SDK environment variable") +endif CFLAGS += -I$(FLEXRAN_SDK)/lib_common CFLAGS += -I$(FLEXRAN_SDK)/lib_turbo CFLAGS += -I$(FLEXRAN_SDK)/lib_crc CFLAGS += -I$(FLEXRAN_SDK)/lib_rate_matching - LDLIBS += -L$(FLEXRAN_SDK)/lib_turbo -lturbo LDLIBS += -L$(FLEXRAN_SDK)/lib_crc -lcrc LDLIBS += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching LDLIBS += -L$(FLEXRAN_SDK)/lib_common -lcommon -LDLIBS += -lstdc++ -lirc -limf -lipps +LDLIBS += -lstdc++ -lirc -limf -lipps -lsvml +endif # library version LIBABIVER := 1 diff --git a/drivers/baseband/turbo_sw/bbdev_turbo_software.c b/drivers/baseband/turbo_sw/bbdev_turbo_software.c index 5204a77..cac8920 100644 --- a/drivers/baseband/turbo_sw/bbdev_turbo_software.c +++ b/drivers/baseband/turbo_sw/bbdev_turbo_software.c @@ -14,10 +14,11 @@ #include <rte_bbdev.h> #include <rte_bbdev_pmd.h> +#ifdef RTE_BBDEV_SDK_AVX2 #include <phy_turbo.h> #include <phy_crc.h> #include <phy_rate_match.h> -#include <divide.h> +#endif #define DRIVER_NAME baseband_turbo_sw @@ -83,6 +84,7 @@ struct turbo_sw_queue { enum rte_bbdev_op_type type; } __rte_cache_aligned; +#ifdef RTE_BBDEV_SDK_AVX2 static inline char * mbuf_append(struct rte_mbuf *m_head, struct rte_mbuf *m, uint16_t len) { @@ -128,6 +130,7 @@ struct turbo_sw_queue { return result; } +#endif /* Read flag value 0/1 from bitmap */ static inline bool @@ -143,6 +146,7 @@ struct turbo_sw_queue { struct bbdev_private *internals = dev->data->dev_private; static const struct rte_bbdev_op_cap bbdev_capabilities[] = { +#ifdef RTE_BBDEV_SDK_AVX2 { .type = RTE_BBDEV_OP_TURBO_DEC, .cap.turbo_dec = { @@ -172,6 +176,7 @@ struct turbo_sw_queue { .num_buffers_dst = RTE_BBDEV_MAX_CODE_BLOCKS, } }, +#endif RTE_BBDEV_END_OF_CAPABILITIES_LIST() }; @@ -410,6 +415,7 @@ struct turbo_sw_queue { .queue_release = q_release }; +#ifdef RTE_BBDEV_SDK_AVX2 /* Checks if the encoder input buffer is correct. * Returns 0 if it's valid, -1 otherwise. */ @@ -464,6 +470,7 @@ struct turbo_sw_queue { return 0; } +#endif static inline void process_enc_cb(struct turbo_sw_queue *q, struct rte_bbdev_enc_op *op, @@ -472,6 +479,7 @@ struct turbo_sw_queue { struct rte_mbuf *m_out, uint16_t in_offset, uint16_t out_offset, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int16_t k_idx; uint16_t m; @@ -724,6 +732,22 @@ struct turbo_sw_queue { } *tmp_out = 0; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(r); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(ncb); + RTE_SET_USED(e); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void @@ -835,6 +859,7 @@ struct turbo_sw_queue { NULL); } +#ifdef RTE_BBDEV_SDK_AVX2 static inline void move_padding_bytes(const uint8_t *in, uint8_t *out, uint16_t k, uint16_t ncb) @@ -847,6 +872,7 @@ struct turbo_sw_queue { rte_memcpy(&out[nd + kpi + 64], &in[kpi], d); rte_memcpy(&out[(nd - 1) + 2 * (kpi + 64)], &in[2 * kpi], d); } +#endif static inline void process_dec_cb(struct turbo_sw_queue *q, struct rte_bbdev_dec_op *op, @@ -856,6 +882,7 @@ struct turbo_sw_queue { uint16_t crc24_overlap, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int32_t k_idx; int32_t iter_cnt; @@ -972,6 +999,22 @@ struct turbo_sw_queue { rte_bbdev_log(ERR, "Turbo Decoder failed"); return; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(kw); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(check_crc_24b); + RTE_SET_USED(crc24_overlap); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 7c9b4b5..878cc31 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -221,11 +221,14 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +# Dependency on the FLEXRAN SDK library if available _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_crc -lcrc _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_turbo -lturbo _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_common -lcommon _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lirc -limf -lstdc++ -lipps +endif # CONFIG_RTE_BBDEV_SDK_AVX2 endif # CONFIG_RTE_LIBRTE_BBDEV ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y) -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v7 0/3] BBDEV turbo_sw PMD compilation fix 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru @ 2019-06-19 17:48 ` Nicolas Chautru 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru ` (3 more replies) 0 siblings, 4 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-19 17:48 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, aconole, Nicolas Chautru Update v7: Remove architecture specific reference to RTE_CPUFLAG_SSE4_2 reported on travis-ci Update v6: Cosmetic change to commit message to include previous Acked-by. Update v5: Cosmetic change to remove trailing space and to commit message. Rebased to latest. Update v4: Missed one file for meson build path and minor change to prevent warning for some configurations due to unused symbols. Update v3: Cosmetic changes in documentation commit to be more 4G/AVX2 specific. Update v2: Splitting into 3 patches as recommended (ignore previous v2 which had a typo) Based on discussion with maintainer, pushing first a patch to help maintenance of the baseband_turbo_sw which had been lacking. The documentation is clarified to point to steps on building the SDK libraries which are now publicly available. (Cosmetic changes to the webpage containing these SDK will happen in parallel based on feedback from maintainer). A compile flag is added to be able to build the turbo_sw PMD when the SDK libraries for AVX2 are installed or not. In both cases this can be compiled with gcc RTE_TARGET. Missing meson build support is also added. Note that additional BBDEV changes pushed in previous v1 https://patches.dpdk.org/project/dpdk/list/?series=4657 will be added in a separate v2 patchset which will depend on this very patchset. Nicolas Chautru (3): baseband/turbo_sw: option to build turbosw PMD without SDK docs/guides: updating turbo_sw building steps baseband/turbo_sw: meson build support for PMD driver config/common_base | 3 +- doc/guides/bbdevs/turbo_sw.rst | 81 +++++++++++++----------- drivers/baseband/meson.build | 2 +- drivers/baseband/turbo_sw/Makefile | 13 ++-- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 51 ++++++++++++++- drivers/baseband/turbo_sw/meson.build | 30 +++++++++ meson_options.txt | 2 + mk/rte.app.mk | 3 + 8 files changed, 138 insertions(+), 47 deletions(-) create mode 100644 drivers/baseband/turbo_sw/meson.build -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v7 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru @ 2019-06-19 17:48 ` Nicolas Chautru 2019-06-20 22:42 ` Mokhtar, Amr 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru ` (2 subsequent siblings) 3 siblings, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-19 17:48 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, aconole, Nicolas Chautru Adding compile flag to allow to build the turbo_sw PMD without dependency to have the SDK libraries installed. Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- config/common_base | 3 +- drivers/baseband/turbo_sw/Makefile | 13 +++--- drivers/baseband/turbo_sw/bbdev_turbo_software.c | 51 +++++++++++++++++++++++- mk/rte.app.mk | 3 ++ 4 files changed, 61 insertions(+), 9 deletions(-) diff --git a/config/common_base b/config/common_base index e406e78..21ab606 100644 --- a/config/common_base +++ b/config/common_base @@ -528,6 +528,7 @@ CONFIG_RTE_PMD_PACKET_PREFETCH=y CONFIG_RTE_LIBRTE_BBDEV=y CONFIG_RTE_BBDEV_MAX_DEVS=128 CONFIG_RTE_BBDEV_OFFLOAD_COST=y +CONFIG_RTE_BBDEV_SDK_AVX2=n # # Compile PMD for NULL bbdev device @@ -537,7 +538,7 @@ CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL=y # # Compile PMD for turbo software bbdev device # -CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=n +CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y # # Compile generic crypto device library diff --git a/drivers/baseband/turbo_sw/Makefile b/drivers/baseband/turbo_sw/Makefile index d364677..414d0d9 100644 --- a/drivers/baseband/turbo_sw/Makefile +++ b/drivers/baseband/turbo_sw/Makefile @@ -3,9 +3,6 @@ include $(RTE_SDK)/mk/rte.vars.mk -ifeq ($(FLEXRAN_SDK),) -$(error "Please define FLEXRAN_SDK environment variable") -endif # library name LIB = librte_pmd_bbdev_turbo_sw.a @@ -21,17 +18,21 @@ LDLIBS += -lrte_bus_vdev # versioning export map EXPORT_MAP := rte_pmd_bbdev_turbo_sw_version.map -# external library dependencies +# external library dependencies if available +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +ifeq ($(FLEXRAN_SDK),) +$(error "Please define FLEXRAN_SDK environment variable") +endif CFLAGS += -I$(FLEXRAN_SDK)/lib_common CFLAGS += -I$(FLEXRAN_SDK)/lib_turbo CFLAGS += -I$(FLEXRAN_SDK)/lib_crc CFLAGS += -I$(FLEXRAN_SDK)/lib_rate_matching - LDLIBS += -L$(FLEXRAN_SDK)/lib_turbo -lturbo LDLIBS += -L$(FLEXRAN_SDK)/lib_crc -lcrc LDLIBS += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching LDLIBS += -L$(FLEXRAN_SDK)/lib_common -lcommon -LDLIBS += -lstdc++ -lirc -limf -lipps +LDLIBS += -lstdc++ -lirc -limf -lipps -lsvml +endif # library version LIBABIVER := 1 diff --git a/drivers/baseband/turbo_sw/bbdev_turbo_software.c b/drivers/baseband/turbo_sw/bbdev_turbo_software.c index 5204a77..94aa536 100644 --- a/drivers/baseband/turbo_sw/bbdev_turbo_software.c +++ b/drivers/baseband/turbo_sw/bbdev_turbo_software.c @@ -14,10 +14,11 @@ #include <rte_bbdev.h> #include <rte_bbdev_pmd.h> +#ifdef RTE_BBDEV_SDK_AVX2 #include <phy_turbo.h> #include <phy_crc.h> #include <phy_rate_match.h> -#include <divide.h> +#endif #define DRIVER_NAME baseband_turbo_sw @@ -83,6 +84,7 @@ struct turbo_sw_queue { enum rte_bbdev_op_type type; } __rte_cache_aligned; +#ifdef RTE_BBDEV_SDK_AVX2 static inline char * mbuf_append(struct rte_mbuf *m_head, struct rte_mbuf *m, uint16_t len) { @@ -128,6 +130,7 @@ struct turbo_sw_queue { return result; } +#endif /* Read flag value 0/1 from bitmap */ static inline bool @@ -143,6 +146,7 @@ struct turbo_sw_queue { struct bbdev_private *internals = dev->data->dev_private; static const struct rte_bbdev_op_cap bbdev_capabilities[] = { +#ifdef RTE_BBDEV_SDK_AVX2 { .type = RTE_BBDEV_OP_TURBO_DEC, .cap.turbo_dec = { @@ -172,6 +176,7 @@ struct turbo_sw_queue { .num_buffers_dst = RTE_BBDEV_MAX_CODE_BLOCKS, } }, +#endif RTE_BBDEV_END_OF_CAPABILITIES_LIST() }; @@ -179,7 +184,12 @@ struct turbo_sw_queue { .queue_size = RTE_BBDEV_QUEUE_SIZE_LIMIT, }; +#ifdef RTE_BBDEV_SDK_AVX2 static const enum rte_cpu_flag_t cpu_flag = RTE_CPUFLAG_SSE4_2; + dev_info->cpu_flag_reqs = &cpu_flag; +#else + dev_info->cpu_flag_reqs = NULL; +#endif default_queue_conf.socket = dev->data->socket_id; @@ -191,7 +201,6 @@ struct turbo_sw_queue { dev_info->max_ul_queue_priority = 0; dev_info->default_queue_conf = default_queue_conf; dev_info->capabilities = bbdev_capabilities; - dev_info->cpu_flag_reqs = &cpu_flag; dev_info->min_alignment = 64; rte_bbdev_log_debug("got device info from %u\n", dev->data->dev_id); @@ -410,6 +419,7 @@ struct turbo_sw_queue { .queue_release = q_release }; +#ifdef RTE_BBDEV_SDK_AVX2 /* Checks if the encoder input buffer is correct. * Returns 0 if it's valid, -1 otherwise. */ @@ -464,6 +474,7 @@ struct turbo_sw_queue { return 0; } +#endif static inline void process_enc_cb(struct turbo_sw_queue *q, struct rte_bbdev_enc_op *op, @@ -472,6 +483,7 @@ struct turbo_sw_queue { struct rte_mbuf *m_out, uint16_t in_offset, uint16_t out_offset, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int16_t k_idx; uint16_t m; @@ -724,6 +736,22 @@ struct turbo_sw_queue { } *tmp_out = 0; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(r); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(ncb); + RTE_SET_USED(e); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void @@ -835,6 +863,7 @@ struct turbo_sw_queue { NULL); } +#ifdef RTE_BBDEV_SDK_AVX2 static inline void move_padding_bytes(const uint8_t *in, uint8_t *out, uint16_t k, uint16_t ncb) @@ -847,6 +876,7 @@ struct turbo_sw_queue { rte_memcpy(&out[nd + kpi + 64], &in[kpi], d); rte_memcpy(&out[(nd - 1) + 2 * (kpi + 64)], &in[2 * kpi], d); } +#endif static inline void process_dec_cb(struct turbo_sw_queue *q, struct rte_bbdev_dec_op *op, @@ -856,6 +886,7 @@ struct turbo_sw_queue { uint16_t crc24_overlap, uint16_t in_length, struct rte_bbdev_stats *q_stats) { +#ifdef RTE_BBDEV_SDK_AVX2 int ret; int32_t k_idx; int32_t iter_cnt; @@ -972,6 +1003,22 @@ struct turbo_sw_queue { rte_bbdev_log(ERR, "Turbo Decoder failed"); return; } +#else + RTE_SET_USED(q); + RTE_SET_USED(op); + RTE_SET_USED(c); + RTE_SET_USED(k); + RTE_SET_USED(kw); + RTE_SET_USED(m_in); + RTE_SET_USED(m_out_head); + RTE_SET_USED(m_out); + RTE_SET_USED(in_offset); + RTE_SET_USED(out_offset); + RTE_SET_USED(check_crc_24b); + RTE_SET_USED(crc24_overlap); + RTE_SET_USED(in_length); + RTE_SET_USED(q_stats); +#endif } static inline void diff --git a/mk/rte.app.mk b/mk/rte.app.mk index d0df0b0..2c3d07a 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -219,11 +219,14 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_NULL) += -lrte_pmd_bbdev_null # TURBO SOFTWARE PMD is dependent on the FLEXRAN library _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lrte_pmd_bbdev_turbo_sw +ifeq ($(CONFIG_RTE_BBDEV_SDK_AVX2),y) +# Dependency on the FLEXRAN SDK library if available _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_crc -lcrc _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_turbo -lturbo _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_rate_matching -lrate_matching _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -L$(FLEXRAN_SDK)/lib_common -lcommon _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW) += -lirc -limf -lstdc++ -lipps +endif # CONFIG_RTE_BBDEV_SDK_AVX2 endif # CONFIG_RTE_LIBRTE_BBDEV ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y) -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [dpdk-dev] [PATCH v7 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru @ 2019-06-20 22:42 ` Mokhtar, Amr 0 siblings, 0 replies; 62+ messages in thread From: Mokhtar, Amr @ 2019-06-20 22:42 UTC (permalink / raw) To: Chautru, Nicolas, akhil.goyal, dev Cc: Yigit, Ferruh, thomas, Chalupnik, KamilX, aconole > -----Original Message----- > From: Chautru, Nicolas > Sent: Wednesday 19 June 2019 18:49 > To: akhil.goyal@nxp.com; dev@dpdk.org > Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; thomas@monjalon.net; > Mokhtar, Amr <amr.mokhtar@intel.com>; Chalupnik, KamilX > <kamilx.chalupnik@intel.com>; aconole@redhat.com; Chautru, Nicolas > <nicolas.chautru@intel.com> > Subject: [PATCH v7 1/3] baseband/turbo_sw: option to build turbosw PMD > without SDK > > Adding compile flag to allow to build the turbo_sw PMD > without dependency to have the SDK libraries installed. > > Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> > --- Acked-by: Amr Mokhtar <amr.mokhtar@intel.com> ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v7 2/3] docs/guides: updating turbo_sw building steps 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru @ 2019-06-19 17:48 ` Nicolas Chautru 2019-07-07 8:59 ` Thomas Monjalon 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2019-06-20 17:33 ` [dpdk-dev] [PATCH v7 0/3] BBDEV turbo_sw PMD compilation fix Ferruh Yigit 3 siblings, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-19 17:48 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, aconole, Nicolas Chautru The documentation is clarified to point to steps on building the SDK libraries which are now publicly available: https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> Acked-by: Kamil Chalupnik <kamilx.chalupnik@intel.com> --- doc/guides/bbdevs/turbo_sw.rst | 81 +++++++++++++++++++++++------------------- 1 file changed, 44 insertions(+), 37 deletions(-) diff --git a/doc/guides/bbdevs/turbo_sw.rst b/doc/guides/bbdevs/turbo_sw.rst index 29f7ec9..455fa1d 100644 --- a/doc/guides/bbdevs/turbo_sw.rst +++ b/doc/guides/bbdevs/turbo_sw.rst @@ -4,23 +4,39 @@ SW Turbo Poll Mode Driver ========================= -The SW Turbo PMD (**baseband_turbo_sw**) provides a poll mode bbdev driver that utilizes -Intel optimized libraries for LTE Layer 1 workloads acceleration. This PMD -supports the functions: Turbo FEC, Rate Matching and CRC functions. +The SW Turbo PMD (**baseband_turbo_sw**) provides a software only poll mode bbdev +driver that can optionally utilize Intel optimized libraries for LTE Layer 1 +workloads acceleration. + +Note that the driver can also be built without any dependency with reduced +functionality for maintenance purpose. + +To enable linking to the SDK libraries see detailed installation section below. +One flag can be enabled depending on whether the target machine can support +AVX2 instructions sets and the related SDK libraries for vectorized +signal processing functions are installed : +- CONFIG_RTE_BBDEV_SDK_AVX2 + +By default this flag is disabled. For AVX2 machine and SDK +library installed then this flag can be enabled. + +This PMD supports the functions: FEC, Rate Matching and CRC functions detailed +in the Features section. Features -------- -SW Turbo PMD has support for the following capabilities: +SW Turbo PMD can support for the following capabilities when the SDK libraries +are used: -For the encode operation: +For the LTE encode operation: * ``RTE_BBDEV_TURBO_CRC_24A_ATTACH`` * ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` * ``RTE_BBDEV_TURBO_RATE_MATCH`` * ``RTE_BBDEV_TURBO_RV_INDEX_BYPASS`` -For the decode operation: +For the LTE decode operation: * ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` * ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` @@ -41,14 +57,10 @@ Installation FlexRAN SDK Download ~~~~~~~~~~~~~~~~~~~~ -To build DPDK with the *baseband_turbo_sw* PMD the user is required to download -the export controlled ``FlexRAN SDK`` Libraries. An account at `Intel Resource -Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_ -needs to be registered. +As an option it is possible to link this driver with FleXRAN SDK libraries +which can enable real time signal processing using AVX instructions. -Once registered, the user needs to log in, and look for -*Intel FlexRAN Software Release Package -18-09* to download or directly through -this `link <https://cdrdv2.intel.com/v1/dl/getContent/605167>`_. +These libraries are available through this link `link <https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules>`_. After download is complete, the user needs to unpack and compile on their system before building DPDK. @@ -57,24 +69,24 @@ The following table maps DPDK versions with past FlexRAN SDK releases: .. _table_flexran_releases: -.. table:: DPDK and FlexRAN SDK releases compliance +.. table:: DPDK and FlexRAN FEC SDK releases compliance ===================== ============================ - DPDK version FlexRAN SDK release + DPDK version FlexRAN FEC SDK release ===================== ============================ - 18.02 1.3.0 - 18.05 1.4.0 - 18.08 1.6.0 - 19.02 18.09 + 19.08 19.04 ===================== ============================ FlexRAN SDK Installation ~~~~~~~~~~~~~~~~~~~~~~~~ +Note that the installation of these libraries is optional. + The following are pre-requisites for building FlexRAN SDK Libraries: - (a) An AVX2 supporting machine - (b) CentOS Linux release 7.2.1511 (Core) operating system - (c) Intel ICC 18.0.1 20171018 compiler installed + (a) An AVX2 or AVX512 supporting machine + (b) CentOS Linux release 7.2.1511 (Core) operating system is advised + (c) Intel ICC 18.0.1 20171018 compiler or more recent and related libraries + ICC is available with a free community license `link <https://software.intel.com/en-us/system-studio/choose-download#technical>`_. The following instructions should be followed in this exact order: @@ -84,25 +96,18 @@ The following instructions should be followed in this exact order: source <path-to-icc-compiler-install-folder>/linux/bin/compilervars.sh intel64 -platform linux -#. Extract the ``605167-flexran-18-09-tar.gz`` package: - - .. code-block:: console - - mkdir FlexRAN-18.09 - tar xvzf 605167-flexran-18-09-tar.gz -C FlexRAN-18.09/ - #. Run the SDK extractor script and accept the license: .. code-block:: console - cd <path-to-workspace>/FlexRAN-18.09/ - ./SDK-18.09.sh + cd <path-to-workspace> + ./FlexRAN-FEC-SDK-19-04.sh #. Generate makefiles based on system configuration: .. code-block:: console - cd <path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ + cd <path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/ ./create-makefiles-linux.sh #. A build folder is generated in this form ``build-<ISA>-<CC>``, enter that @@ -129,12 +134,14 @@ Example: .. code-block:: console - export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/build-avx2-icc/install - export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ - + export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/install + export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/ -* Set ``CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y`` in DPDK common configuration - file ``config/common_base``. +* Set ``CONFIG_RTE_BBDEV_SDK_AVX2=y`` + in DPDK common configuration file ``config/common_base`` to be able to use + the SDK libraries as mentioned above. + If no flag are set the PMD driver will still build but its capabilities + will be limited accordingly. To use the PMD in an application, user must: -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/3] docs/guides: updating turbo_sw building steps 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru @ 2019-07-07 8:59 ` Thomas Monjalon 0 siblings, 0 replies; 62+ messages in thread From: Thomas Monjalon @ 2019-07-07 8:59 UTC (permalink / raw) To: Nicolas Chautru Cc: dev, akhil.goyal, ferruh.yigit, amr.mokhtar, kamilx.chalupnik, aconole 19/06/2019 19:48, Nicolas Chautru: > The documentation is clarified to point to steps on building the > SDK libraries which are now publicly available: > https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules > > Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> > Acked-by: Kamil Chalupnik <kamilx.chalupnik@intel.com> > --- > -*Intel FlexRAN Software Release Package -18-09* to download or directly through > -this `link <https://cdrdv2.intel.com/v1/dl/getContent/605167>`_. > +These libraries are available through this link `link <https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules>`_. [...] > + ICC is available with a free community license `link <https://software.intel.com/en-us/system-studio/choose-download#technical>`_. These links are wrongly formatted. I will fix. ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v7 3/3] baseband/turbo_sw: meson build support for PMD driver 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru @ 2019-06-19 17:48 ` Nicolas Chautru 2019-06-20 17:33 ` [dpdk-dev] [PATCH v7 0/3] BBDEV turbo_sw PMD compilation fix Ferruh Yigit 3 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-19 17:48 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, aconole, Nicolas Chautru Turbo_sw PMD driver now building with meson/ninja with or without SDK libraries. Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> Acked-by: Kamil Chalupnik <kamilx.chalupnik@intel.com> --- drivers/baseband/meson.build | 2 +- drivers/baseband/turbo_sw/meson.build | 30 ++++++++++++++++++++++++++++++ meson_options.txt | 2 ++ 3 files changed, 33 insertions(+), 1 deletion(-) create mode 100644 drivers/baseband/turbo_sw/meson.build diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build index 52489df..40a87d2 100644 --- a/drivers/baseband/meson.build +++ b/drivers/baseband/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <bluca@debian.org> -drivers = ['null'] +drivers = ['null', 'turbo_sw'] config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' driver_name_fmt = 'rte_pmd_@0@' diff --git a/drivers/baseband/turbo_sw/meson.build b/drivers/baseband/turbo_sw/meson.build new file mode 100644 index 0000000..9b4fe34 --- /dev/null +++ b/drivers/baseband/turbo_sw/meson.build @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +path = get_option('flexran_sdk') + +if dpdk_conf.has('RTE_BBDEV_SDK_AVX2') + lib = cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: false) + if not lib.found() + build = false + else + ext_deps += cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: true) + ext_deps += cc.find_library('libcrc', dirs: [path + '/lib_crc'], required: true) + ext_deps += cc.find_library('librate_matching', dirs: [path + '/lib_rate_matching'], required: true) + ext_deps += cc.find_library('libcommon', dirs: [path + '/lib_common'], required: true) + ext_deps += cc.find_library('libstdc++', required: true) + ext_deps += cc.find_library('libirc', required: true) + ext_deps += cc.find_library('libimf', required: true) + ext_deps += cc.find_library('libipps', required: true) + ext_deps += cc.find_library('libsvml', required: true) + includes += include_directories(path + '/lib_turbo') + includes += include_directories(path + '/lib_crc') + includes += include_directories(path + '/lib_rate_matching') + includes += include_directories(path + '/lib_common') + endif +endif + +deps += ['bbdev', 'bus_vdev', 'ring'] +name = 'bbdev_turbo_sw' +allow_experimental_apis = true +sources = files('bbdev_turbo_software.c') diff --git a/meson_options.txt b/meson_options.txt index 16d9f92..92d3e97 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -10,6 +10,8 @@ option('enable_kmods', type: 'boolean', value: true, description: 'build kernel modules') option('examples', type: 'string', value: '', description: 'Comma-separated list of examples to build by default') +option('flexran_sdk', type: 'string', value: '', + description: 'Path to FlexRAN SDK optional Libraries for BBDEV device') option('ibverbs_link', type: 'combo', choices : ['shared', 'dlopen'], value: 'shared', description: 'Linkage method (shared/dlopen) for Mellanox PMDs with ibverbs dependencies.') option('include_subdir_arch', type: 'string', value: '', -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [dpdk-dev] [PATCH v7 0/3] BBDEV turbo_sw PMD compilation fix 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru ` (2 preceding siblings ...) 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru @ 2019-06-20 17:33 ` Ferruh Yigit 2019-06-25 12:41 ` Akhil Goyal 3 siblings, 1 reply; 62+ messages in thread From: Ferruh Yigit @ 2019-06-20 17:33 UTC (permalink / raw) To: Nicolas Chautru, akhil.goyal, dev Cc: thomas, amr.mokhtar, kamilx.chalupnik, aconole On 6/19/2019 6:48 PM, Nicolas Chautru wrote: > Update v7: Remove architecture specific reference to RTE_CPUFLAG_SSE4_2 reported on travis-ci > Update v6: Cosmetic change to commit message to include previous Acked-by. > Update v5: Cosmetic change to remove trailing space and to commit message. Rebased to latest. > Update v4: Missed one file for meson build path and minor change to prevent warning for some configurations due to unused symbols. > Update v3: Cosmetic changes in documentation commit to be more 4G/AVX2 specific. > Update v2: Splitting into 3 patches as recommended (ignore previous v2 which had a typo) > > Based on discussion with maintainer, pushing first a patch to help maintenance of the baseband_turbo_sw which had been lacking. > The documentation is clarified to point to steps on building the SDK libraries which are now publicly available. > (Cosmetic changes to the webpage containing these SDK will happen in parallel based on feedback from maintainer). > A compile flag is added to be able to build the turbo_sw PMD when the SDK libraries for AVX2 are installed or not. > In both cases this can be compiled with gcc RTE_TARGET. > Missing meson build support is also added. > > Note that additional BBDEV changes pushed in previous v1 > https://patches.dpdk.org/project/dpdk/list/?series=4657 > will be added in a separate v2 patchset which will depend on this very patchset. > > > Nicolas Chautru (3): > baseband/turbo_sw: option to build turbosw PMD without SDK > docs/guides: updating turbo_sw building steps > baseband/turbo_sw: meson build support for PMD driver > For series, Acked-by: Ferruh Yigit <ferruh.yigit@intel.com> ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [dpdk-dev] [PATCH v7 0/3] BBDEV turbo_sw PMD compilation fix 2019-06-20 17:33 ` [dpdk-dev] [PATCH v7 0/3] BBDEV turbo_sw PMD compilation fix Ferruh Yigit @ 2019-06-25 12:41 ` Akhil Goyal 0 siblings, 0 replies; 62+ messages in thread From: Akhil Goyal @ 2019-06-25 12:41 UTC (permalink / raw) To: Ferruh Yigit, Nicolas Chautru, dev Cc: thomas, amr.mokhtar, kamilx.chalupnik, aconole > > On 6/19/2019 6:48 PM, Nicolas Chautru wrote: > > Update v7: Remove architecture specific reference to RTE_CPUFLAG_SSE4_2 > reported on travis-ci > > Update v6: Cosmetic change to commit message to include previous Acked-by. > > Update v5: Cosmetic change to remove trailing space and to commit message. > Rebased to latest. > > Update v4: Missed one file for meson build path and minor change to prevent > warning for some configurations due to unused symbols. > > Update v3: Cosmetic changes in documentation commit to be more 4G/AVX2 > specific. > > Update v2: Splitting into 3 patches as recommended (ignore previous v2 which > had a typo) > > > > Based on discussion with maintainer, pushing first a patch to help maintenance > of the baseband_turbo_sw which had been lacking. > > The documentation is clarified to point to steps on building the SDK libraries > which are now publicly available. > > (Cosmetic changes to the webpage containing these SDK will happen in > parallel based on feedback from maintainer). > > A compile flag is added to be able to build the turbo_sw PMD when the SDK > libraries for AVX2 are installed or not. > > In both cases this can be compiled with gcc RTE_TARGET. > > Missing meson build support is also added. > > > > Note that additional BBDEV changes pushed in previous v1 > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatches. > dpdk.org%2Fproject%2Fdpdk%2Flist%2F%3Fseries%3D4657&data=02%7C0 > 1%7Cakhil.goyal%40nxp.com%7Cedc9d3ef619248e7b86b08d6f5a56bdd%7C686 > ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636966488159758077&s > data=KEjtdAELhz61dVb3V7SQ8WPoEw5eC59VD9UYOVgiif0%3D&reserved= > 0 > > will be added in a separate v2 patchset which will depend on this very patchset. > > > > > > Nicolas Chautru (3): > > baseband/turbo_sw: option to build turbosw PMD without SDK > > docs/guides: updating turbo_sw building steps > > baseband/turbo_sw: meson build support for PMD driver > > > > For series, > Acked-by: Ferruh Yigit <ferruh.yigit@intel.com> Series Applied to dpdk-next-crypto Thanks. ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v6 2/3] docs/guides: updating turbo_sw building steps 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru @ 2019-06-19 17:11 ` Nicolas Chautru 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-19 17:11 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, Nicolas Chautru The documentation is clarified to point to steps on building the SDK libraries which are now publicly available: https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules Acked-by: Kamil Chalupnik <kamilx.chalupnik@intel.com> Acked-by: Nicolas Chautru <nicolas.chautru@intel.com> Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- doc/guides/bbdevs/turbo_sw.rst | 81 +++++++++++++++++++++++------------------- 1 file changed, 44 insertions(+), 37 deletions(-) diff --git a/doc/guides/bbdevs/turbo_sw.rst b/doc/guides/bbdevs/turbo_sw.rst index 29f7ec9..455fa1d 100644 --- a/doc/guides/bbdevs/turbo_sw.rst +++ b/doc/guides/bbdevs/turbo_sw.rst @@ -4,23 +4,39 @@ SW Turbo Poll Mode Driver ========================= -The SW Turbo PMD (**baseband_turbo_sw**) provides a poll mode bbdev driver that utilizes -Intel optimized libraries for LTE Layer 1 workloads acceleration. This PMD -supports the functions: Turbo FEC, Rate Matching and CRC functions. +The SW Turbo PMD (**baseband_turbo_sw**) provides a software only poll mode bbdev +driver that can optionally utilize Intel optimized libraries for LTE Layer 1 +workloads acceleration. + +Note that the driver can also be built without any dependency with reduced +functionality for maintenance purpose. + +To enable linking to the SDK libraries see detailed installation section below. +One flag can be enabled depending on whether the target machine can support +AVX2 instructions sets and the related SDK libraries for vectorized +signal processing functions are installed : +- CONFIG_RTE_BBDEV_SDK_AVX2 + +By default this flag is disabled. For AVX2 machine and SDK +library installed then this flag can be enabled. + +This PMD supports the functions: FEC, Rate Matching and CRC functions detailed +in the Features section. Features -------- -SW Turbo PMD has support for the following capabilities: +SW Turbo PMD can support for the following capabilities when the SDK libraries +are used: -For the encode operation: +For the LTE encode operation: * ``RTE_BBDEV_TURBO_CRC_24A_ATTACH`` * ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` * ``RTE_BBDEV_TURBO_RATE_MATCH`` * ``RTE_BBDEV_TURBO_RV_INDEX_BYPASS`` -For the decode operation: +For the LTE decode operation: * ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` * ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` @@ -41,14 +57,10 @@ Installation FlexRAN SDK Download ~~~~~~~~~~~~~~~~~~~~ -To build DPDK with the *baseband_turbo_sw* PMD the user is required to download -the export controlled ``FlexRAN SDK`` Libraries. An account at `Intel Resource -Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_ -needs to be registered. +As an option it is possible to link this driver with FleXRAN SDK libraries +which can enable real time signal processing using AVX instructions. -Once registered, the user needs to log in, and look for -*Intel FlexRAN Software Release Package -18-09* to download or directly through -this `link <https://cdrdv2.intel.com/v1/dl/getContent/605167>`_. +These libraries are available through this link `link <https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules>`_. After download is complete, the user needs to unpack and compile on their system before building DPDK. @@ -57,24 +69,24 @@ The following table maps DPDK versions with past FlexRAN SDK releases: .. _table_flexran_releases: -.. table:: DPDK and FlexRAN SDK releases compliance +.. table:: DPDK and FlexRAN FEC SDK releases compliance ===================== ============================ - DPDK version FlexRAN SDK release + DPDK version FlexRAN FEC SDK release ===================== ============================ - 18.02 1.3.0 - 18.05 1.4.0 - 18.08 1.6.0 - 19.02 18.09 + 19.08 19.04 ===================== ============================ FlexRAN SDK Installation ~~~~~~~~~~~~~~~~~~~~~~~~ +Note that the installation of these libraries is optional. + The following are pre-requisites for building FlexRAN SDK Libraries: - (a) An AVX2 supporting machine - (b) CentOS Linux release 7.2.1511 (Core) operating system - (c) Intel ICC 18.0.1 20171018 compiler installed + (a) An AVX2 or AVX512 supporting machine + (b) CentOS Linux release 7.2.1511 (Core) operating system is advised + (c) Intel ICC 18.0.1 20171018 compiler or more recent and related libraries + ICC is available with a free community license `link <https://software.intel.com/en-us/system-studio/choose-download#technical>`_. The following instructions should be followed in this exact order: @@ -84,25 +96,18 @@ The following instructions should be followed in this exact order: source <path-to-icc-compiler-install-folder>/linux/bin/compilervars.sh intel64 -platform linux -#. Extract the ``605167-flexran-18-09-tar.gz`` package: - - .. code-block:: console - - mkdir FlexRAN-18.09 - tar xvzf 605167-flexran-18-09-tar.gz -C FlexRAN-18.09/ - #. Run the SDK extractor script and accept the license: .. code-block:: console - cd <path-to-workspace>/FlexRAN-18.09/ - ./SDK-18.09.sh + cd <path-to-workspace> + ./FlexRAN-FEC-SDK-19-04.sh #. Generate makefiles based on system configuration: .. code-block:: console - cd <path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ + cd <path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/ ./create-makefiles-linux.sh #. A build folder is generated in this form ``build-<ISA>-<CC>``, enter that @@ -129,12 +134,14 @@ Example: .. code-block:: console - export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/build-avx2-icc/install - export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ - + export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/install + export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/ -* Set ``CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y`` in DPDK common configuration - file ``config/common_base``. +* Set ``CONFIG_RTE_BBDEV_SDK_AVX2=y`` + in DPDK common configuration file ``config/common_base`` to be able to use + the SDK libraries as mentioned above. + If no flag are set the PMD driver will still build but its capabilities + will be limited accordingly. To use the PMD in an application, user must: -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v6 3/3] baseband/turbo_sw: meson build support for PMD driver 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru @ 2019-06-19 17:11 ` Nicolas Chautru 2019-06-20 0:05 ` Aaron Conole 2 siblings, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-19 17:11 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, Nicolas Chautru Turbo_sw PMD driver now building with meson/ninja with or without SDK libraries. Acked-by: Kamil Chalupnik <kamilx.chalupnik@intel.com> Acked-by: Nicolas Chautru <nicolas.chautru@intel.com> Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- drivers/baseband/meson.build | 2 +- drivers/baseband/turbo_sw/meson.build | 30 ++++++++++++++++++++++++++++++ meson_options.txt | 2 ++ 3 files changed, 33 insertions(+), 1 deletion(-) create mode 100644 drivers/baseband/turbo_sw/meson.build diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build index 52489df..40a87d2 100644 --- a/drivers/baseband/meson.build +++ b/drivers/baseband/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <bluca@debian.org> -drivers = ['null'] +drivers = ['null', 'turbo_sw'] config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' driver_name_fmt = 'rte_pmd_@0@' diff --git a/drivers/baseband/turbo_sw/meson.build b/drivers/baseband/turbo_sw/meson.build new file mode 100644 index 0000000..9b4fe34 --- /dev/null +++ b/drivers/baseband/turbo_sw/meson.build @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +path = get_option('flexran_sdk') + +if dpdk_conf.has('RTE_BBDEV_SDK_AVX2') + lib = cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: false) + if not lib.found() + build = false + else + ext_deps += cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: true) + ext_deps += cc.find_library('libcrc', dirs: [path + '/lib_crc'], required: true) + ext_deps += cc.find_library('librate_matching', dirs: [path + '/lib_rate_matching'], required: true) + ext_deps += cc.find_library('libcommon', dirs: [path + '/lib_common'], required: true) + ext_deps += cc.find_library('libstdc++', required: true) + ext_deps += cc.find_library('libirc', required: true) + ext_deps += cc.find_library('libimf', required: true) + ext_deps += cc.find_library('libipps', required: true) + ext_deps += cc.find_library('libsvml', required: true) + includes += include_directories(path + '/lib_turbo') + includes += include_directories(path + '/lib_crc') + includes += include_directories(path + '/lib_rate_matching') + includes += include_directories(path + '/lib_common') + endif +endif + +deps += ['bbdev', 'bus_vdev', 'ring'] +name = 'bbdev_turbo_sw' +allow_experimental_apis = true +sources = files('bbdev_turbo_software.c') diff --git a/meson_options.txt b/meson_options.txt index 16d9f92..92d3e97 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -10,6 +10,8 @@ option('enable_kmods', type: 'boolean', value: true, description: 'build kernel modules') option('examples', type: 'string', value: '', description: 'Comma-separated list of examples to build by default') +option('flexran_sdk', type: 'string', value: '', + description: 'Path to FlexRAN SDK optional Libraries for BBDEV device') option('ibverbs_link', type: 'combo', choices : ['shared', 'dlopen'], value: 'shared', description: 'Linkage method (shared/dlopen) for Mellanox PMDs with ibverbs dependencies.') option('include_subdir_arch', type: 'string', value: '', -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [dpdk-dev] [PATCH v6 3/3] baseband/turbo_sw: meson build support for PMD driver 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru @ 2019-06-20 0:05 ` Aaron Conole 2019-06-20 0:34 ` Chautru, Nicolas 0 siblings, 1 reply; 62+ messages in thread From: Aaron Conole @ 2019-06-20 0:05 UTC (permalink / raw) To: Nicolas Chautru Cc: akhil.goyal, dev, ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik Nicolas Chautru <nicolas.chautru@intel.com> writes: > Turbo_sw PMD driver now building with meson/ninja > with or without SDK libraries. > > Acked-by: Kamil Chalupnik <kamilx.chalupnik@intel.com> > Acked-by: Nicolas Chautru <nicolas.chautru@intel.com> It seems strange to ack your own patch. > Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> > --- Something is wrong with this series: https://travis-ci.com/ovsrobot/dpdk/builds/116170401 We cannot enable this on AARCH64 builds - it seems that it wants to take advantage of facilities that are x86 specific (like SSE 4.2) > drivers/baseband/meson.build | 2 +- > drivers/baseband/turbo_sw/meson.build | 30 ++++++++++++++++++++++++++++++ > meson_options.txt | 2 ++ > 3 files changed, 33 insertions(+), 1 deletion(-) > create mode 100644 drivers/baseband/turbo_sw/meson.build > > diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build > index 52489df..40a87d2 100644 > --- a/drivers/baseband/meson.build > +++ b/drivers/baseband/meson.build > @@ -1,7 +1,7 @@ > # SPDX-License-Identifier: BSD-3-Clause > # Copyright(c) 2018 Luca Boccassi <bluca@debian.org> > > -drivers = ['null'] > +drivers = ['null', 'turbo_sw'] > > config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' > driver_name_fmt = 'rte_pmd_@0@' > diff --git a/drivers/baseband/turbo_sw/meson.build b/drivers/baseband/turbo_sw/meson.build > new file mode 100644 > index 0000000..9b4fe34 > --- /dev/null > +++ b/drivers/baseband/turbo_sw/meson.build > @@ -0,0 +1,30 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright(c) 2019 Intel Corporation > + > +path = get_option('flexran_sdk') > + > +if dpdk_conf.has('RTE_BBDEV_SDK_AVX2') > + lib = cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: false) > + if not lib.found() > + build = false > + else > + ext_deps += cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: true) > + ext_deps += cc.find_library('libcrc', dirs: [path + '/lib_crc'], required: true) > + ext_deps += cc.find_library('librate_matching', dirs: [path + '/lib_rate_matching'], required: true) > + ext_deps += cc.find_library('libcommon', dirs: [path + '/lib_common'], required: true) > + ext_deps += cc.find_library('libstdc++', required: true) > + ext_deps += cc.find_library('libirc', required: true) > + ext_deps += cc.find_library('libimf', required: true) > + ext_deps += cc.find_library('libipps', required: true) > + ext_deps += cc.find_library('libsvml', required: true) > + includes += include_directories(path + '/lib_turbo') > + includes += include_directories(path + '/lib_crc') > + includes += include_directories(path + '/lib_rate_matching') > + includes += include_directories(path + '/lib_common') > + endif > +endif > + > +deps += ['bbdev', 'bus_vdev', 'ring'] > +name = 'bbdev_turbo_sw' > +allow_experimental_apis = true > +sources = files('bbdev_turbo_software.c') > diff --git a/meson_options.txt b/meson_options.txt > index 16d9f92..92d3e97 100644 > --- a/meson_options.txt > +++ b/meson_options.txt > @@ -10,6 +10,8 @@ option('enable_kmods', type: 'boolean', value: true, > description: 'build kernel modules') > option('examples', type: 'string', value: '', > description: 'Comma-separated list of examples to build by default') > +option('flexran_sdk', type: 'string', value: '', > + description: 'Path to FlexRAN SDK optional Libraries for BBDEV device') > option('ibverbs_link', type: 'combo', choices : ['shared', 'dlopen'], value: 'shared', > description: 'Linkage method (shared/dlopen) for Mellanox PMDs with ibverbs dependencies.') > option('include_subdir_arch', type: 'string', value: '', ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [dpdk-dev] [PATCH v6 3/3] baseband/turbo_sw: meson build support for PMD driver 2019-06-20 0:05 ` Aaron Conole @ 2019-06-20 0:34 ` Chautru, Nicolas 0 siblings, 0 replies; 62+ messages in thread From: Chautru, Nicolas @ 2019-06-20 0:34 UTC (permalink / raw) To: Aaron Conole Cc: akhil.goyal, dev, Yigit, Ferruh, thomas, Mokhtar, Amr, Chalupnik, KamilX > >-----Original Message----- >From: Aaron Conole [mailto:aconole@redhat.com] >Sent: Wednesday, June 19, 2019 5:06 PM >To: Chautru, Nicolas <nicolas.chautru@intel.com> >Cc: akhil.goyal@nxp.com; dev@dpdk.org; Yigit, Ferruh <ferruh.yigit@intel.com>; thomas@monjalon.net; Mokhtar, Amr <amr.mokhtar@intel.com>; Chalupnik, KamilX <kamilx.chalupnik@intel.com> >Subject: Re: [dpdk-dev] [PATCH v6 3/3] baseband/turbo_sw: meson build support for PMD driver > >Nicolas Chautru <nicolas.chautru@intel.com> writes: > >> Turbo_sw PMD driver now building with meson/ninja with or without SDK >> libraries. >> >> Acked-by: Kamil Chalupnik <kamilx.chalupnik@intel.com> >> Acked-by: Nicolas Chautru <nicolas.chautru@intel.com> > >It seems strange to ack your own patch. I will remove. > >> Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> >> --- > >Something is wrong with this series: > > https://travis-ci.com/ovsrobot/dpdk/builds/116170401 > >We cannot enable this on AARCH64 builds - it seems that it wants to take advantage of facilities that are x86 specific (like SSE 4.2) > Thanks for the feedback. This application can link optionally with AVX/SSE libraries but by default should compile on any platform. This flag can be kept to NULL by default. I will push now as new patchset. Thanks ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v5 2/3] docs/guides: updating turbo_sw building steps 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru @ 2019-06-13 16:51 ` Nicolas Chautru 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-13 16:51 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, Nicolas Chautru The documentation is clarified to point to steps on building the SDK libraries which are now publicly available: https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- doc/guides/bbdevs/turbo_sw.rst | 81 +++++++++++++++++++++++------------------- 1 file changed, 44 insertions(+), 37 deletions(-) diff --git a/doc/guides/bbdevs/turbo_sw.rst b/doc/guides/bbdevs/turbo_sw.rst index 29f7ec9..455fa1d 100644 --- a/doc/guides/bbdevs/turbo_sw.rst +++ b/doc/guides/bbdevs/turbo_sw.rst @@ -4,23 +4,39 @@ SW Turbo Poll Mode Driver ========================= -The SW Turbo PMD (**baseband_turbo_sw**) provides a poll mode bbdev driver that utilizes -Intel optimized libraries for LTE Layer 1 workloads acceleration. This PMD -supports the functions: Turbo FEC, Rate Matching and CRC functions. +The SW Turbo PMD (**baseband_turbo_sw**) provides a software only poll mode bbdev +driver that can optionally utilize Intel optimized libraries for LTE Layer 1 +workloads acceleration. + +Note that the driver can also be built without any dependency with reduced +functionality for maintenance purpose. + +To enable linking to the SDK libraries see detailed installation section below. +One flag can be enabled depending on whether the target machine can support +AVX2 instructions sets and the related SDK libraries for vectorized +signal processing functions are installed : +- CONFIG_RTE_BBDEV_SDK_AVX2 + +By default this flag is disabled. For AVX2 machine and SDK +library installed then this flag can be enabled. + +This PMD supports the functions: FEC, Rate Matching and CRC functions detailed +in the Features section. Features -------- -SW Turbo PMD has support for the following capabilities: +SW Turbo PMD can support for the following capabilities when the SDK libraries +are used: -For the encode operation: +For the LTE encode operation: * ``RTE_BBDEV_TURBO_CRC_24A_ATTACH`` * ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` * ``RTE_BBDEV_TURBO_RATE_MATCH`` * ``RTE_BBDEV_TURBO_RV_INDEX_BYPASS`` -For the decode operation: +For the LTE decode operation: * ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` * ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` @@ -41,14 +57,10 @@ Installation FlexRAN SDK Download ~~~~~~~~~~~~~~~~~~~~ -To build DPDK with the *baseband_turbo_sw* PMD the user is required to download -the export controlled ``FlexRAN SDK`` Libraries. An account at `Intel Resource -Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_ -needs to be registered. +As an option it is possible to link this driver with FleXRAN SDK libraries +which can enable real time signal processing using AVX instructions. -Once registered, the user needs to log in, and look for -*Intel FlexRAN Software Release Package -18-09* to download or directly through -this `link <https://cdrdv2.intel.com/v1/dl/getContent/605167>`_. +These libraries are available through this link `link <https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules>`_. After download is complete, the user needs to unpack and compile on their system before building DPDK. @@ -57,24 +69,24 @@ The following table maps DPDK versions with past FlexRAN SDK releases: .. _table_flexran_releases: -.. table:: DPDK and FlexRAN SDK releases compliance +.. table:: DPDK and FlexRAN FEC SDK releases compliance ===================== ============================ - DPDK version FlexRAN SDK release + DPDK version FlexRAN FEC SDK release ===================== ============================ - 18.02 1.3.0 - 18.05 1.4.0 - 18.08 1.6.0 - 19.02 18.09 + 19.08 19.04 ===================== ============================ FlexRAN SDK Installation ~~~~~~~~~~~~~~~~~~~~~~~~ +Note that the installation of these libraries is optional. + The following are pre-requisites for building FlexRAN SDK Libraries: - (a) An AVX2 supporting machine - (b) CentOS Linux release 7.2.1511 (Core) operating system - (c) Intel ICC 18.0.1 20171018 compiler installed + (a) An AVX2 or AVX512 supporting machine + (b) CentOS Linux release 7.2.1511 (Core) operating system is advised + (c) Intel ICC 18.0.1 20171018 compiler or more recent and related libraries + ICC is available with a free community license `link <https://software.intel.com/en-us/system-studio/choose-download#technical>`_. The following instructions should be followed in this exact order: @@ -84,25 +96,18 @@ The following instructions should be followed in this exact order: source <path-to-icc-compiler-install-folder>/linux/bin/compilervars.sh intel64 -platform linux -#. Extract the ``605167-flexran-18-09-tar.gz`` package: - - .. code-block:: console - - mkdir FlexRAN-18.09 - tar xvzf 605167-flexran-18-09-tar.gz -C FlexRAN-18.09/ - #. Run the SDK extractor script and accept the license: .. code-block:: console - cd <path-to-workspace>/FlexRAN-18.09/ - ./SDK-18.09.sh + cd <path-to-workspace> + ./FlexRAN-FEC-SDK-19-04.sh #. Generate makefiles based on system configuration: .. code-block:: console - cd <path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ + cd <path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/ ./create-makefiles-linux.sh #. A build folder is generated in this form ``build-<ISA>-<CC>``, enter that @@ -129,12 +134,14 @@ Example: .. code-block:: console - export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/build-avx2-icc/install - export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ - + export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/install + export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/ -* Set ``CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y`` in DPDK common configuration - file ``config/common_base``. +* Set ``CONFIG_RTE_BBDEV_SDK_AVX2=y`` + in DPDK common configuration file ``config/common_base`` to be able to use + the SDK libraries as mentioned above. + If no flag are set the PMD driver will still build but its capabilities + will be limited accordingly. To use the PMD in an application, user must: -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v5 3/3] baseband/turbo_sw: meson build support for PMD driver 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru @ 2019-06-13 16:51 ` Nicolas Chautru 2 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-13 16:51 UTC (permalink / raw) To: akhil.goyal, dev Cc: ferruh.yigit, thomas, amr.mokhtar, kamilx.chalupnik, Nicolas Chautru Turbo_sw PMD driver now building with meson/ninja with or without SDK libraries. Acked-by: Kamil Chalupnik <kamilx.chalupnik@intel.com> Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- drivers/baseband/meson.build | 2 +- drivers/baseband/turbo_sw/meson.build | 30 ++++++++++++++++++++++++++++++ meson_options.txt | 2 ++ 3 files changed, 33 insertions(+), 1 deletion(-) create mode 100644 drivers/baseband/turbo_sw/meson.build diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build index 52489df..40a87d2 100644 --- a/drivers/baseband/meson.build +++ b/drivers/baseband/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <bluca@debian.org> -drivers = ['null'] +drivers = ['null', 'turbo_sw'] config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' driver_name_fmt = 'rte_pmd_@0@' diff --git a/drivers/baseband/turbo_sw/meson.build b/drivers/baseband/turbo_sw/meson.build new file mode 100644 index 0000000..9b4fe34 --- /dev/null +++ b/drivers/baseband/turbo_sw/meson.build @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +path = get_option('flexran_sdk') + +if dpdk_conf.has('RTE_BBDEV_SDK_AVX2') + lib = cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: false) + if not lib.found() + build = false + else + ext_deps += cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: true) + ext_deps += cc.find_library('libcrc', dirs: [path + '/lib_crc'], required: true) + ext_deps += cc.find_library('librate_matching', dirs: [path + '/lib_rate_matching'], required: true) + ext_deps += cc.find_library('libcommon', dirs: [path + '/lib_common'], required: true) + ext_deps += cc.find_library('libstdc++', required: true) + ext_deps += cc.find_library('libirc', required: true) + ext_deps += cc.find_library('libimf', required: true) + ext_deps += cc.find_library('libipps', required: true) + ext_deps += cc.find_library('libsvml', required: true) + includes += include_directories(path + '/lib_turbo') + includes += include_directories(path + '/lib_crc') + includes += include_directories(path + '/lib_rate_matching') + includes += include_directories(path + '/lib_common') + endif +endif + +deps += ['bbdev', 'bus_vdev', 'ring'] +name = 'bbdev_turbo_sw' +allow_experimental_apis = true +sources = files('bbdev_turbo_software.c') diff --git a/meson_options.txt b/meson_options.txt index 16d9f92..92d3e97 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -10,6 +10,8 @@ option('enable_kmods', type: 'boolean', value: true, description: 'build kernel modules') option('examples', type: 'string', value: '', description: 'Comma-separated list of examples to build by default') +option('flexran_sdk', type: 'string', value: '', + description: 'Path to FlexRAN SDK optional Libraries for BBDEV device') option('ibverbs_link', type: 'combo', choices : ['shared', 'dlopen'], value: 'shared', description: 'Linkage method (shared/dlopen) for Mellanox PMDs with ibverbs dependencies.') option('include_subdir_arch', type: 'string', value: '', -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v4 2/3] docs/guides: updating turbo_sw building steps 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru @ 2019-06-07 23:54 ` Nicolas Chautru 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-07 23:54 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru The documentation is clarified to point to steps on building the SDK libraries which are now publicly available: https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- doc/guides/bbdevs/turbo_sw.rst | 81 +++++++++++++++++++++++------------------- 1 file changed, 44 insertions(+), 37 deletions(-) diff --git a/doc/guides/bbdevs/turbo_sw.rst b/doc/guides/bbdevs/turbo_sw.rst index 29f7ec9..3fe03b2 100644 --- a/doc/guides/bbdevs/turbo_sw.rst +++ b/doc/guides/bbdevs/turbo_sw.rst @@ -4,23 +4,39 @@ SW Turbo Poll Mode Driver ========================= -The SW Turbo PMD (**baseband_turbo_sw**) provides a poll mode bbdev driver that utilizes -Intel optimized libraries for LTE Layer 1 workloads acceleration. This PMD -supports the functions: Turbo FEC, Rate Matching and CRC functions. +The SW Turbo PMD (**baseband_turbo_sw**) provides a software only poll mode bbdev +driver that can optionally utilize Intel optimized libraries for LTE Layer 1 +workloads acceleration. + +Note that the driver can also be built without any dependency with reduced +functionality for maintenance purpose. + +To enable linking to the SDK libraries see detailed installation section below. +One flag can be enabled depending on whether the target machine can support +AVX2 instructions sets and the related SDK libraries for vectorized +signal processing functions are installed : +- CONFIG_RTE_BBDEV_SDK_AVX2 + +By default this flag is disabled. For AVX2 machine and SDK +library installed then this flag can be enabled. + +This PMD supports the functions: FEC, Rate Matching and CRC functions detailed +in the Features section. Features -------- -SW Turbo PMD has support for the following capabilities: +SW Turbo PMD can support for the following capabilities when the SDK libraries +are used: -For the encode operation: +For the LTE encode operation: * ``RTE_BBDEV_TURBO_CRC_24A_ATTACH`` * ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` * ``RTE_BBDEV_TURBO_RATE_MATCH`` * ``RTE_BBDEV_TURBO_RV_INDEX_BYPASS`` -For the decode operation: +For the LTE decode operation: * ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` * ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` @@ -41,14 +57,10 @@ Installation FlexRAN SDK Download ~~~~~~~~~~~~~~~~~~~~ -To build DPDK with the *baseband_turbo_sw* PMD the user is required to download -the export controlled ``FlexRAN SDK`` Libraries. An account at `Intel Resource -Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_ -needs to be registered. +As an option it is possible to link this driver with FleXRAN SDK libraries +which can enable real time signal processing using AVX instructions. -Once registered, the user needs to log in, and look for -*Intel FlexRAN Software Release Package -18-09* to download or directly through -this `link <https://cdrdv2.intel.com/v1/dl/getContent/605167>`_. +These libraries are available through this link `link <https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules>`_. After download is complete, the user needs to unpack and compile on their system before building DPDK. @@ -57,24 +69,24 @@ The following table maps DPDK versions with past FlexRAN SDK releases: .. _table_flexran_releases: -.. table:: DPDK and FlexRAN SDK releases compliance +.. table:: DPDK and FlexRAN FEC SDK releases compliance ===================== ============================ - DPDK version FlexRAN SDK release + DPDK version FlexRAN FEC SDK release ===================== ============================ - 18.02 1.3.0 - 18.05 1.4.0 - 18.08 1.6.0 - 19.02 18.09 + 19.08 19.04 ===================== ============================ FlexRAN SDK Installation ~~~~~~~~~~~~~~~~~~~~~~~~ +Note that the installation of these libraries is optional. + The following are pre-requisites for building FlexRAN SDK Libraries: - (a) An AVX2 supporting machine - (b) CentOS Linux release 7.2.1511 (Core) operating system - (c) Intel ICC 18.0.1 20171018 compiler installed + (a) An AVX2 or AVX512 supporting machine + (b) CentOS Linux release 7.2.1511 (Core) operating system is advised + (c) Intel ICC 18.0.1 20171018 compiler or more recent and related libraries + ICC is available with a free community license `link <https://software.intel.com/en-us/system-studio/choose-download#technical>`_. The following instructions should be followed in this exact order: @@ -84,25 +96,18 @@ The following instructions should be followed in this exact order: source <path-to-icc-compiler-install-folder>/linux/bin/compilervars.sh intel64 -platform linux -#. Extract the ``605167-flexran-18-09-tar.gz`` package: - - .. code-block:: console - - mkdir FlexRAN-18.09 - tar xvzf 605167-flexran-18-09-tar.gz -C FlexRAN-18.09/ - #. Run the SDK extractor script and accept the license: .. code-block:: console - cd <path-to-workspace>/FlexRAN-18.09/ - ./SDK-18.09.sh + cd <path-to-workspace> + ./FlexRAN-FEC-SDK-19-04.sh #. Generate makefiles based on system configuration: .. code-block:: console - cd <path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ + cd <path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/ ./create-makefiles-linux.sh #. A build folder is generated in this form ``build-<ISA>-<CC>``, enter that @@ -129,12 +134,14 @@ Example: .. code-block:: console - export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/build-avx2-icc/install - export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ - + export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/install + export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/ -* Set ``CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y`` in DPDK common configuration - file ``config/common_base``. +* Set ``CONFIG_RTE_BBDEV_SDK_AVX2=y`` + in DPDK common configuration file ``config/common_base`` to be able to use + the SDK libraries as mentioned above. + If no flag are set the PMD driver will still build but its capabilities + will be limited accordingly. To use the PMD in an application, user must: -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v4 3/3] baseband/turbo_sw: meson build support for PMD driver 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru @ 2019-06-07 23:54 ` Nicolas Chautru 2 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-07 23:54 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- drivers/baseband/meson.build | 2 +- drivers/baseband/turbo_sw/meson.build | 30 ++++++++++++++++++++++++++++++ meson_options.txt | 2 ++ 3 files changed, 33 insertions(+), 1 deletion(-) create mode 100644 drivers/baseband/turbo_sw/meson.build diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build index 52489df..40a87d2 100644 --- a/drivers/baseband/meson.build +++ b/drivers/baseband/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <bluca@debian.org> -drivers = ['null'] +drivers = ['null', 'turbo_sw'] config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' driver_name_fmt = 'rte_pmd_@0@' diff --git a/drivers/baseband/turbo_sw/meson.build b/drivers/baseband/turbo_sw/meson.build new file mode 100644 index 0000000..9b4fe34 --- /dev/null +++ b/drivers/baseband/turbo_sw/meson.build @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +path = get_option('flexran_sdk') + +if dpdk_conf.has('RTE_BBDEV_SDK_AVX2') + lib = cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: false) + if not lib.found() + build = false + else + ext_deps += cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: true) + ext_deps += cc.find_library('libcrc', dirs: [path + '/lib_crc'], required: true) + ext_deps += cc.find_library('librate_matching', dirs: [path + '/lib_rate_matching'], required: true) + ext_deps += cc.find_library('libcommon', dirs: [path + '/lib_common'], required: true) + ext_deps += cc.find_library('libstdc++', required: true) + ext_deps += cc.find_library('libirc', required: true) + ext_deps += cc.find_library('libimf', required: true) + ext_deps += cc.find_library('libipps', required: true) + ext_deps += cc.find_library('libsvml', required: true) + includes += include_directories(path + '/lib_turbo') + includes += include_directories(path + '/lib_crc') + includes += include_directories(path + '/lib_rate_matching') + includes += include_directories(path + '/lib_common') + endif +endif + +deps += ['bbdev', 'bus_vdev', 'ring'] +name = 'bbdev_turbo_sw' +allow_experimental_apis = true +sources = files('bbdev_turbo_software.c') diff --git a/meson_options.txt b/meson_options.txt index 16d9f92..92d3e97 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -10,6 +10,8 @@ option('enable_kmods', type: 'boolean', value: true, description: 'build kernel modules') option('examples', type: 'string', value: '', description: 'Comma-separated list of examples to build by default') +option('flexran_sdk', type: 'string', value: '', + description: 'Path to FlexRAN SDK optional Libraries for BBDEV device') option('ibverbs_link', type: 'combo', choices : ['shared', 'dlopen'], value: 'shared', description: 'Linkage method (shared/dlopen) for Mellanox PMDs with ibverbs dependencies.') option('include_subdir_arch', type: 'string', value: '', -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v3 2/3] docs/guides: updating turbo_sw building steps 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru @ 2019-06-06 10:17 ` Nicolas Chautru 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-06 10:17 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru The documentation is clarified to point to steps on building the SDK libraries which are now publicly available: https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- doc/guides/bbdevs/turbo_sw.rst | 81 +++++++++++++++++++++++------------------- 1 file changed, 44 insertions(+), 37 deletions(-) diff --git a/doc/guides/bbdevs/turbo_sw.rst b/doc/guides/bbdevs/turbo_sw.rst index 29f7ec9..3fe03b2 100644 --- a/doc/guides/bbdevs/turbo_sw.rst +++ b/doc/guides/bbdevs/turbo_sw.rst @@ -4,23 +4,39 @@ SW Turbo Poll Mode Driver ========================= -The SW Turbo PMD (**baseband_turbo_sw**) provides a poll mode bbdev driver that utilizes -Intel optimized libraries for LTE Layer 1 workloads acceleration. This PMD -supports the functions: Turbo FEC, Rate Matching and CRC functions. +The SW Turbo PMD (**baseband_turbo_sw**) provides a software only poll mode bbdev +driver that can optionally utilize Intel optimized libraries for LTE Layer 1 +workloads acceleration. + +Note that the driver can also be built without any dependency with reduced +functionality for maintenance purpose. + +To enable linking to the SDK libraries see detailed installation section below. +One flag can be enabled depending on whether the target machine can support +AVX2 instructions sets and the related SDK libraries for vectorized +signal processing functions are installed : +- CONFIG_RTE_BBDEV_SDK_AVX2 + +By default this flag is disabled. For AVX2 machine and SDK +library installed then this flag can be enabled. + +This PMD supports the functions: FEC, Rate Matching and CRC functions detailed +in the Features section. Features -------- -SW Turbo PMD has support for the following capabilities: +SW Turbo PMD can support for the following capabilities when the SDK libraries +are used: -For the encode operation: +For the LTE encode operation: * ``RTE_BBDEV_TURBO_CRC_24A_ATTACH`` * ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` * ``RTE_BBDEV_TURBO_RATE_MATCH`` * ``RTE_BBDEV_TURBO_RV_INDEX_BYPASS`` -For the decode operation: +For the LTE decode operation: * ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` * ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` @@ -41,14 +57,10 @@ Installation FlexRAN SDK Download ~~~~~~~~~~~~~~~~~~~~ -To build DPDK with the *baseband_turbo_sw* PMD the user is required to download -the export controlled ``FlexRAN SDK`` Libraries. An account at `Intel Resource -Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_ -needs to be registered. +As an option it is possible to link this driver with FleXRAN SDK libraries +which can enable real time signal processing using AVX instructions. -Once registered, the user needs to log in, and look for -*Intel FlexRAN Software Release Package -18-09* to download or directly through -this `link <https://cdrdv2.intel.com/v1/dl/getContent/605167>`_. +These libraries are available through this link `link <https://software.intel.com/en-us/articles/flexran-lte-and-5g-nr-fec-software-development-kit-modules>`_. After download is complete, the user needs to unpack and compile on their system before building DPDK. @@ -57,24 +69,24 @@ The following table maps DPDK versions with past FlexRAN SDK releases: .. _table_flexran_releases: -.. table:: DPDK and FlexRAN SDK releases compliance +.. table:: DPDK and FlexRAN FEC SDK releases compliance ===================== ============================ - DPDK version FlexRAN SDK release + DPDK version FlexRAN FEC SDK release ===================== ============================ - 18.02 1.3.0 - 18.05 1.4.0 - 18.08 1.6.0 - 19.02 18.09 + 19.08 19.04 ===================== ============================ FlexRAN SDK Installation ~~~~~~~~~~~~~~~~~~~~~~~~ +Note that the installation of these libraries is optional. + The following are pre-requisites for building FlexRAN SDK Libraries: - (a) An AVX2 supporting machine - (b) CentOS Linux release 7.2.1511 (Core) operating system - (c) Intel ICC 18.0.1 20171018 compiler installed + (a) An AVX2 or AVX512 supporting machine + (b) CentOS Linux release 7.2.1511 (Core) operating system is advised + (c) Intel ICC 18.0.1 20171018 compiler or more recent and related libraries + ICC is available with a free community license `link <https://software.intel.com/en-us/system-studio/choose-download#technical>`_. The following instructions should be followed in this exact order: @@ -84,25 +96,18 @@ The following instructions should be followed in this exact order: source <path-to-icc-compiler-install-folder>/linux/bin/compilervars.sh intel64 -platform linux -#. Extract the ``605167-flexran-18-09-tar.gz`` package: - - .. code-block:: console - - mkdir FlexRAN-18.09 - tar xvzf 605167-flexran-18-09-tar.gz -C FlexRAN-18.09/ - #. Run the SDK extractor script and accept the license: .. code-block:: console - cd <path-to-workspace>/FlexRAN-18.09/ - ./SDK-18.09.sh + cd <path-to-workspace> + ./FlexRAN-FEC-SDK-19-04.sh #. Generate makefiles based on system configuration: .. code-block:: console - cd <path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ + cd <path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/ ./create-makefiles-linux.sh #. A build folder is generated in this form ``build-<ISA>-<CC>``, enter that @@ -129,12 +134,14 @@ Example: .. code-block:: console - export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/build-avx2-icc/install - export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-18.09/SDK-18.09/sdk/ - + export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/install + export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-FEC-SDK-19-04/sdk/build-avx2-icc/ -* Set ``CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y`` in DPDK common configuration - file ``config/common_base``. +* Set ``CONFIG_RTE_BBDEV_SDK_AVX2=y`` + in DPDK common configuration file ``config/common_base`` to be able to use + the SDK libraries as mentioned above. + If no flag are set the PMD driver will still build but its capabilities + will be limited accordingly. To use the PMD in an application, user must: -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v3 3/3] baseband/turbo_sw: meson build support for PMD driver 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru @ 2019-06-06 10:17 ` Nicolas Chautru 2 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-06 10:17 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- drivers/baseband/meson.build | 2 +- drivers/baseband/turbo_sw/meson.build | 30 ++++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletion(-) create mode 100644 drivers/baseband/turbo_sw/meson.build diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build index 52489df..40a87d2 100644 --- a/drivers/baseband/meson.build +++ b/drivers/baseband/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <bluca@debian.org> -drivers = ['null'] +drivers = ['null', 'turbo_sw'] config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' driver_name_fmt = 'rte_pmd_@0@' diff --git a/drivers/baseband/turbo_sw/meson.build b/drivers/baseband/turbo_sw/meson.build new file mode 100644 index 0000000..9b4fe34 --- /dev/null +++ b/drivers/baseband/turbo_sw/meson.build @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +path = get_option('flexran_sdk') + +if dpdk_conf.has('RTE_BBDEV_SDK_AVX2') + lib = cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: false) + if not lib.found() + build = false + else + ext_deps += cc.find_library('libturbo', dirs: [path + '/lib_turbo'], required: true) + ext_deps += cc.find_library('libcrc', dirs: [path + '/lib_crc'], required: true) + ext_deps += cc.find_library('librate_matching', dirs: [path + '/lib_rate_matching'], required: true) + ext_deps += cc.find_library('libcommon', dirs: [path + '/lib_common'], required: true) + ext_deps += cc.find_library('libstdc++', required: true) + ext_deps += cc.find_library('libirc', required: true) + ext_deps += cc.find_library('libimf', required: true) + ext_deps += cc.find_library('libipps', required: true) + ext_deps += cc.find_library('libsvml', required: true) + includes += include_directories(path + '/lib_turbo') + includes += include_directories(path + '/lib_crc') + includes += include_directories(path + '/lib_rate_matching') + includes += include_directories(path + '/lib_common') + endif +endif + +deps += ['bbdev', 'bus_vdev', 'ring'] +name = 'bbdev_turbo_sw' +allow_experimental_apis = true +sources = files('bbdev_turbo_software.c') -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v2 2/3] doc/guides: documentation for the FPGA BBDEV PMD 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 0/3] baseband/fpga_lte_fec: adding driver for FEC on FPGA Nicolas Chautru 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 1/3] " Nicolas Chautru @ 2019-06-05 21:20 ` Nicolas Chautru 2019-06-05 21:21 ` [dpdk-dev] [PATCH v2 3/3] baseband/fpga_lte_fec: meson support Nicolas Chautru 2 siblings, 0 replies; 62+ messages in thread From: Nicolas Chautru @ 2019-06-05 21:20 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Adding documentation related to the new PMD driver in previous commit Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- doc/guides/bbdevs/fpga_lte_fec.rst | 318 +++++++++++++++++++++++++++++++++++++ doc/guides/bbdevs/index.rst | 1 + 2 files changed, 319 insertions(+) create mode 100644 doc/guides/bbdevs/fpga_lte_fec.rst diff --git a/doc/guides/bbdevs/fpga_lte_fec.rst b/doc/guides/bbdevs/fpga_lte_fec.rst new file mode 100644 index 0000000..71b058c --- /dev/null +++ b/doc/guides/bbdevs/fpga_lte_fec.rst @@ -0,0 +1,318 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation + +Intel(R) FPGA LTE FEC Poll Mode Driver +====================================== + +The BBDEV FPGA LTE FEC poll mode driver (PMD) supports an FPGA implementation of a VRAN +Turbo Encode / Decode LTE wireless acceleration function, using Intel's PCI-e and FPGA +based Vista Creek device. + +Features +-------- + +FPGA LTE FEC PMD supports the following features: + +- Turbo Encode in the DL with total throughput of 4.5 Gbits/s +- Turbo Decode in the UL with total throughput of 1.5 Gbits/s assuming 8 decoder iterations +- 8 VFs per PF (physical device) +- Maximum of 32 UL queues per VF +- Maximum of 32 DL queues per VF +- PCIe Gen-3 x8 Interface +- MSI-X +- SR-IOV + + +FPGA LTE FEC PMD supports the following BBDEV capabilities: + +* For the turbo encode operation: + - ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` : set to attach CRC24B to CB(s) + - ``RTE_BBDEV_TURBO_RATE_MATCH`` : if set then do not do Rate Match bypass + - ``RTE_BBDEV_TURBO_ENC_INTERRUPTS`` : set for encoder dequeue interrupts + + +* For the turbo decode operation: + - ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` : check CRC24B from CB(s) + - ``RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE`` : perform subblock de-interleave + - ``RTE_BBDEV_TURBO_DEC_INTERRUPTS`` : set for decoder dequeue interrupts + - ``RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN`` : set if negative LLR encoder i/p is supported + - ``RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP`` : keep CRC24B bits appended while decoding + + +Limitations +----------- + +FPGA LTE FEC does not support the following: + +- Scatter-Gather function + + +Installation +-------------- + +Section 3 of the DPDK manual provides instuctions on installing and compiling DPDK. The +default set of bbdev compile flags may be found in config/common_base, where for example +the flag to build the FPGA LTE FEC device, ``CONFIG_RTE_LIBRTE_PMD_FPGA_LTE_FEC``, is already +set. It is assumed DPDK has been compiled using for instance: + +.. code-block:: console + + make install T=x86_64-native-linuxapp-gcc + + +DPDK requires hugepages to be configured as detailed in section 2 of the DPDK manual. +The bbdev test application has been tested with a configuration 40 x 1GB hugepages. The +hugepage configuration of a server may be examined using: + +.. code-block:: console + + grep Huge* /proc/meminfo + + +Initialization +-------------- + +When the device first powers up, its PCI Physical Functions (PF) can be listed through this command: + +.. code-block:: console + + sudo lspci -vd1172:5052 + +The physical and virtual functions are compatible with Linux UIO drivers: +``vfio`` and ``igb_uio``. However, in order to work the FPGA LTE FEC device firstly needs +to be bound to one of these linux drivers through DPDK. + + +Bind PF UIO driver(s) +~~~~~~~~~~~~~~~~~~~~~ + +Install the DPDK igb_uio driver, bind it with the PF PCI device ID and use +``lspci`` to confirm the PF device is under use by ``igb_uio`` DPDK UIO driver. + +The igb_uio driver may be bound to the PF PCI device using one of three methods: + + +1. PCI functions (physical or virtual, depending on the use case) can be bound to +the UIO driver by repeating this command for every function. + +.. code-block:: console + + cd <dpdk-top-level-directory> + insmod ./build/kmod/igb_uio.ko + echo "1172 5052" > /sys/bus/pci/drivers/igb_uio/new_id + lspci -vd1172: + + +2. Another way to bind PF with DPDK UIO driver is by using the ``dpdk-devbind.py`` tool + +.. code-block:: console + + cd <dpdk-top-level-directory> + ./usertools/dpdk-devbind.py -b igb_uio 0000:06:00.0 + +where the PCI device ID (example: 0000:06:00.0) is obtained using lspci -vd1172: + + +3. A third way to bind is to use ``dpdk-setup.sh`` tool + +.. code-block:: console + + cd <dpdk-top-level-directory> + ./usertools/dpdk-setup.sh + + select 'Bind Ethernet/Crypto/Baseband device to IGB UIO module' + or + select 'Bind Ethernet/Crypto/Baseband device to VFIO module' depending on driver required + enter PCI device ID + select 'Display current Ethernet/Crypto/Baseband device settings' to confirm binding + + +In the same way the FPGA LTE FEC PF can be bound with vfio, but vfio driver does not +support SR-IOV configuration right out of the box, so it will need to be patched. + + +Enable Virtual Functions +~~~~~~~~~~~~~~~~~~~~~~~~ + +Now, it should be visible in the printouts that PCI PF is under igb_uio control +"``Kernel driver in use: igb_uio``" + +To show the number of available VFs on the device, read ``sriov_totalvfs`` file.. + +.. code-block:: console + + cat /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_totalvfs + + where 0000\:<b>\:<d>.<f> is the PCI device ID + + +To enable VFs via igb_uio, echo the number of virtual functions intended to +enable to ``max_vfs`` file.. + +.. code-block:: console + + echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/max_vfs + + +Afterwards, all VFs must be bound to appropriate UIO drivers as required, same +way it was done with the physical function previously. + +Enabling SR-IOV via vfio driver is pretty much the same, except that the file +name is different: + +.. code-block:: console + + echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_numvfs + + +Configure the VFs through PF +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The PCI virtual functions must be configured before working or getting assigned +to VMs/Containers. The configuration involves allocating the number of hardware +queues, priorities, load balance, bandwidth and other settings necessary for the +device to perform FEC functions. + +This configuration needs to be executed at least once after reboot or PCI FLR and can +be achieved by using the function ``fpga_lte_fec_configure()``, which sets up the +parameters defined in ``fpga_lte_fec_conf`` structure: + +.. code-block:: c + + struct fpga_lte_fec_conf { + bool pf_mode_en; + uint8_t vf_ul_queues_number[FPGA_LTE_FEC_NUM_VFS]; + uint8_t vf_dl_queues_number[FPGA_LTE_FEC_NUM_VFS]; + uint8_t ul_bandwidth; + uint8_t dl_bandwidth; + uint8_t ul_load_balance; + uint8_t dl_load_balance; + uint16_t flr_time_out; + }; + +- ``pf_mode_en``: identifies whether only PF is to be used, or the VFs. PF and + VFs are mutually exclusive and cannot run simultaneously. + Set to 1 for PF mode enabled. + If PF mode is enabled all queues available in the device are assigned + exclusively to PF and 0 queues given to VFs. + +- ``vf_*l_queues_number``: defines the hardware queue mapping for every VF. + +- ``*l_bandwidth``: in case of congestion on PCIe interface. The device + allocates different bandwidth to UL and DL. The weight is configured by this + setting. The unit of weight is 3 code blocks. For example, if the code block + cbps (code block per second) ratio between UL and DL is 12:1, then the + configuration value should be set to 36:3. The schedule algorithm is based + on code block regardless the length of each block. + +- ``*l_load_balance``: hardware queues are load-balanced in a round-robin + fashion. Queues get filled first-in first-out until they reach a pre-defined + watermark level, if exceeded, they won't get assigned new code blocks.. + This watermark is defined by this setting. + + If all hardware queues exceeds the watermark, no code blocks will be + streamed in from UL/DL code block FIFO. + +- ``flr_time_out``: specifies how many 16.384us to be FLR time out. The + time_out = flr_time_out x 16.384us. For instance, if you want to set 10ms for + the FLR time out then set this setting to 0x262=610. + + +An example configuration code calling the function ``fpga_lte_fec_configure()`` is shown +below: + +.. code-block:: c + + struct fpga_lte_fec_conf conf; + unsigned int i; + + memset(&conf, 0, sizeof(struct fpga_lte_fec_conf)); + conf.pf_mode_en = 1; + + for (i = 0; i < FPGA_LTE_FEC_NUM_VFS; ++i) { + conf.vf_ul_queues_number[i] = 4; + conf.vf_dl_queues_number[i] = 4; + } + conf.ul_bandwidth = 12; + conf.dl_bandwidth = 5; + conf.dl_load_balance = 64; + conf.ul_load_balance = 64; + + /* setup FPGA PF */ + ret = fpga_lte_fec_configure(info->dev_name, &conf); + TEST_ASSERT_SUCCESS(ret, + "Failed to configure 4G FPGA PF for bbdev %s", + info->dev_name); + + +Test Application +---------------- + +BBDEV provides a test application, ``test-bbdev.py`` and range of test data for testing +the functionality of FPGA LTE FEC turbo encode and turbo decode, depending on the device's +capabilities. The test application is located under app->test-bbdev folder and has the +following options: + +.. code-block:: console + + "-p", "--testapp-path": specifies path to the bbdev test app. + "-e", "--eal-params" : EAL arguments which are passed to the test app. + "-t", "--timeout" : Timeout in seconds (default=300). + "-c", "--test-cases" : Defines test cases to run. Run all if not specified. + "-v", "--test-vector" : Test vector path (default=dpdk_path+/app/test-bbdev/test_vectors/bbdev_null.data). + "-n", "--num-ops" : Number of operations to process on device (default=32). + "-b", "--burst-size" : Operations enqueue/dequeue burst size (default=32). + "-l", "--num-lcores" : Number of lcores to run (default=16). + "-i", "--init-device" : Initialise PF device with default values. + + +To execute the test application tool using simple turbo decode or turbo encode data, +type one of the following: + +.. code-block:: console + + ./test-bbdev.py -c validation -n 64 -b 8 -v ./turbo_dec_default.data + ./test-bbdev.py -c validation -n 64 -b 8 -v ./turbo_enc_default.data + + +The test application ``test-bbdev.py``, supports the ability to configure the PF device with +a default set of values, if the "-i" or "- -init-device" option is included. The default values +are defined in test_bbdev_perf.c as: + +- VF_UL_QUEUE_VALUE 4 +- VF_DL_QUEUE_VALUE 4 +- UL_BANDWIDTH 3 +- DL_BANDWIDTH 3 +- UL_LOAD_BALANCE 128 +- DL_LOAD_BALANCE 128 +- FLR_TIMEOUT 610 + + +Test Vectors +~~~~~~~~~~~~ + +In addition to the simple turbo decoder and turbo encoder tests, bbdev also provides +a range of additional tests under the test_vectors folder, which may be useful. The results +of these tests will depend on the FPGA LTE FEC capabilities: + +* turbo decoder tests: + - ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_high_snr.data`` + - ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_low_snr.data`` + - ``turbo_dec_c1_k6144_r0_e34560_negllr.data`` + - ``turbo_dec_c1_k6144_r0_e34560_sbd_negllr.data`` + - ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr_crc24b.data`` + - ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr.data`` + + +* turbo encoder tests: + - ``turbo_enc_c1_k40_r0_e1190_rm.data`` + - ``turbo_enc_c1_k40_r0_e1194_rm.data`` + - ``turbo_enc_c1_k40_r0_e1196_rm.data`` + - ``turbo_enc_c1_k40_r0_e272_rm.data`` + - ``turbo_enc_c1_k6144_r0_e18444.data`` + - ``turbo_enc_c1_k6144_r0_e32256_crc24b_rm.data`` + - ``turbo_enc_c2_k5952_r0_e17868_crc24b.data`` + - ``turbo_enc_c3_k4800_r2_e14412_crc24b.data`` + - ``turbo_enc_c4_k4800_r2_e14412_crc24b.data`` + + diff --git a/doc/guides/bbdevs/index.rst b/doc/guides/bbdevs/index.rst index 93276ed..005b95e 100644 --- a/doc/guides/bbdevs/index.rst +++ b/doc/guides/bbdevs/index.rst @@ -10,3 +10,4 @@ Baseband Device Drivers null turbo_sw + fpga_lte_fec -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* [dpdk-dev] [PATCH v2 3/3] baseband/fpga_lte_fec: meson support 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 0/3] baseband/fpga_lte_fec: adding driver for FEC on FPGA Nicolas Chautru 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 1/3] " Nicolas Chautru 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 2/3] doc/guides: documentation for the FPGA BBDEV PMD Nicolas Chautru @ 2019-06-05 21:21 ` Nicolas Chautru 2019-06-06 8:25 ` Bruce Richardson 2 siblings, 1 reply; 62+ messages in thread From: Nicolas Chautru @ 2019-06-05 21:21 UTC (permalink / raw) To: thomas, akhil.goyal, dev; +Cc: ferruh.yigit, amr.mokhtar, Nicolas Chautru Adding support for meson build environment for the new BBDEV PMD driver. Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> --- drivers/baseband/fpga_lte_fec/meson.build | 7 +++++++ drivers/baseband/meson.build | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-) create mode 100644 drivers/baseband/fpga_lte_fec/meson.build diff --git a/drivers/baseband/fpga_lte_fec/meson.build b/drivers/baseband/fpga_lte_fec/meson.build new file mode 100644 index 0000000..bf44e6b --- /dev/null +++ b/drivers/baseband/fpga_lte_fec/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +deps += ['bbdev', 'bus_vdev', 'ring', 'pci', 'bus_pci'] +name = 'bbdev_fpga_lte_fec' +allow_experimental_apis = true +sources = files('fpga_lte_fec.c') diff --git a/drivers/baseband/meson.build b/drivers/baseband/meson.build index 52489df..a17d549 100644 --- a/drivers/baseband/meson.build +++ b/drivers/baseband/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <bluca@debian.org> -drivers = ['null'] +drivers = ['null', 'fpga_lte_fec'] config_flag_fmt = 'RTE_LIBRTE_@0@_PMD' driver_name_fmt = 'rte_pmd_@0@' -- 1.8.3.1 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [dpdk-dev] [PATCH v2 3/3] baseband/fpga_lte_fec: meson support 2019-06-05 21:21 ` [dpdk-dev] [PATCH v2 3/3] baseband/fpga_lte_fec: meson support Nicolas Chautru @ 2019-06-06 8:25 ` Bruce Richardson 2019-06-06 10:16 ` Ferruh Yigit 0 siblings, 1 reply; 62+ messages in thread From: Bruce Richardson @ 2019-06-06 8:25 UTC (permalink / raw) To: Nicolas Chautru; +Cc: thomas, akhil.goyal, dev, ferruh.yigit, amr.mokhtar On Wed, Jun 05, 2019 at 02:21:00PM -0700, Nicolas Chautru wrote: > Adding support for meson build environment > for the new BBDEV PMD driver. > > Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> > --- > drivers/baseband/fpga_lte_fec/meson.build | 7 +++++++ > drivers/baseband/meson.build | 2 +- > 2 files changed, 8 insertions(+), 1 deletion(-) > create mode 100644 drivers/baseband/fpga_lte_fec/meson.build > The meson changes should go in the same patch as the makefile ones. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [dpdk-dev] [PATCH v2 3/3] baseband/fpga_lte_fec: meson support 2019-06-06 8:25 ` Bruce Richardson @ 2019-06-06 10:16 ` Ferruh Yigit 2019-06-06 16:39 ` Chautru, Nicolas 0 siblings, 1 reply; 62+ messages in thread From: Ferruh Yigit @ 2019-06-06 10:16 UTC (permalink / raw) To: Bruce Richardson, Nicolas Chautru; +Cc: thomas, akhil.goyal, dev, amr.mokhtar On 6/6/2019 9:25 AM, Bruce Richardson wrote: > On Wed, Jun 05, 2019 at 02:21:00PM -0700, Nicolas Chautru wrote: >> Adding support for meson build environment >> for the new BBDEV PMD driver. >> >> Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> >> --- >> drivers/baseband/fpga_lte_fec/meson.build | 7 +++++++ >> drivers/baseband/meson.build | 2 +- >> 2 files changed, 8 insertions(+), 1 deletion(-) >> create mode 100644 drivers/baseband/fpga_lte_fec/meson.build >> > The meson changes should go in the same patch as the makefile ones. > +1, also better to have doc in same patch, so all can merged into single patch. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [dpdk-dev] [PATCH v2 3/3] baseband/fpga_lte_fec: meson support 2019-06-06 10:16 ` Ferruh Yigit @ 2019-06-06 16:39 ` Chautru, Nicolas 0 siblings, 0 replies; 62+ messages in thread From: Chautru, Nicolas @ 2019-06-06 16:39 UTC (permalink / raw) To: Yigit, Ferruh, Richardson, Bruce; +Cc: thomas, akhil.goyal, dev, Mokhtar, Amr >-----Original Message----- >From: Yigit, Ferruh >Sent: Thursday, June 6, 2019 3:16 AM > >On 6/6/2019 9:25 AM, Bruce Richardson wrote: >> On Wed, Jun 05, 2019 at 02:21:00PM -0700, Nicolas Chautru wrote: >>> Adding support for meson build environment for the new BBDEV PMD >>> driver. >>> >>> Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com> >>> --- >>> drivers/baseband/fpga_lte_fec/meson.build | 7 +++++++ >>> drivers/baseband/meson.build | 2 +- >>> 2 files changed, 8 insertions(+), 1 deletion(-) create mode 100644 >>> drivers/baseband/fpga_lte_fec/meson.build >>> >> The meson changes should go in the same patch as the makefile ones. >> > >+1, also better to have doc in same patch, so all can merged into single patch. > No problem. Pushing in parallel as a new version. ^ permalink raw reply [flat|nested] 62+ messages in thread
end of thread, other threads:[~2019-07-07 8:59 UTC | newest] Thread overview: 62+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-06-05 17:38 [dpdk-dev] [PATCH] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-05 17:38 ` [dpdk-dev] [PATCH] baseband/turbo_sw: Option to build turbosw PMD without SDK Nicolas Chautru 2019-06-05 17:47 ` Thomas Monjalon 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 1/3] version: 19.05-rc4 Nicolas Chautru 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 2/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru 2019-06-05 20:06 ` [dpdk-dev] [PATCH v2 3/3] docs/guides: updating turbo_sw building steps Nicolas Chautru 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru 2019-06-06 10:34 ` Ferruh Yigit 2019-06-06 17:03 ` Chautru, Nicolas 2019-06-05 20:10 ` [dpdk-dev] [PATCH v2 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 0/3] baseband/fpga_lte_fec: adding driver for FEC on FPGA Nicolas Chautru 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 1/3] " Nicolas Chautru 2019-06-06 9:04 ` [dpdk-dev] [PATCH v3] " Nicolas Chautru 2019-06-06 9:04 ` Nicolas Chautru 2019-06-08 0:17 ` [dpdk-dev] [PATCH v4] " Nicolas Chautru 2019-06-08 0:17 ` Nicolas Chautru 2019-06-10 17:01 ` Nicolas Chautru 2019-06-10 17:01 ` Nicolas Chautru [not found] ` <EEA9FF629BF25B47BD67ADE995041EE2496A888B@IRSMSX103.ger.corp.intel.com> 2019-06-13 17:05 ` Chautru, Nicolas 2019-06-13 17:28 ` [dpdk-dev] [PATCH v5] " Nicolas Chautru 2019-06-13 17:28 ` Nicolas Chautru 2019-06-14 16:12 ` [dpdk-dev] [PATCH v6] " Nicolas Chautru 2019-06-14 16:12 ` [dpdk-dev] [PATCH v5] " Nicolas Chautru 2019-06-14 16:17 ` [dpdk-dev] [PATCH v6] " Nicolas Chautru 2019-06-14 16:17 ` Nicolas Chautru 2019-06-19 15:20 ` Chautru, Nicolas [not found] ` <EEA9FF629BF25B47BD67ADE995041EE2496B2975@IRSMSX103.ger.corp.intel.com> 2019-06-20 13:35 ` Ferruh Yigit 2019-06-25 12:37 ` Akhil Goyal 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 0/3] BBDEV turbo_sw PMD compilation fix Nicolas Chautru 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 1/3] baseband/turbo_sw: option to build turbosw PMD without SDK Nicolas Chautru 2019-06-20 22:42 ` Mokhtar, Amr 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru 2019-07-07 8:59 ` Thomas Monjalon 2019-06-19 17:48 ` [dpdk-dev] [PATCH v7 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2019-06-20 17:33 ` [dpdk-dev] [PATCH v7 0/3] BBDEV turbo_sw PMD compilation fix Ferruh Yigit 2019-06-25 12:41 ` Akhil Goyal 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru 2019-06-19 17:11 ` [dpdk-dev] [PATCH v6 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2019-06-20 0:05 ` Aaron Conole 2019-06-20 0:34 ` Chautru, Nicolas 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru 2019-06-13 16:51 ` [dpdk-dev] [PATCH v5 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru 2019-06-07 23:54 ` [dpdk-dev] [PATCH v4 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 2/3] docs/guides: updating turbo_sw building steps Nicolas Chautru 2019-06-06 10:17 ` [dpdk-dev] [PATCH v3 3/3] baseband/turbo_sw: meson build support for PMD driver Nicolas Chautru 2019-06-05 21:20 ` [dpdk-dev] [PATCH v2 2/3] doc/guides: documentation for the FPGA BBDEV PMD Nicolas Chautru 2019-06-05 21:21 ` [dpdk-dev] [PATCH v2 3/3] baseband/fpga_lte_fec: meson support Nicolas Chautru 2019-06-06 8:25 ` Bruce Richardson 2019-06-06 10:16 ` Ferruh Yigit 2019-06-06 16:39 ` Chautru, Nicolas
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).